types not working in multiple grok patterns

[hari9973]

Types are not working when i used the following config

<source>
  @type tail
  tag test1
  path /log
  pos_file /log.pos
  path_key tailed_path
  <parse>
   @type grok
   <grok>
     types type1,type2
     pattern1
   </grok>
   <grok>
     types type1,type2
     pattern2
 </parse>
</source>

I am getting the error while exposing the metrics
failed to instrument a metric. error_class=TypeError error=#<TypeError: String can't be coerced into Intege

[okkez]

The grok section does not handle types parameter.

<parse>
  @type grok
  <grok>
    pattern %{INT:user_id:integer} paid %{NUMBER:paid_amount:float}
  </grok>
</parse>

[hari9973]

still getting the error this is my parse block

<parse>
   @type grok
   <grok>
     pattern (?:%{SYSLOGTIMESTAMP:syslog_timestamp:string}|%{TIMESTAMP_ISO8601:timestamp8601:string}) %{IPORHOST:syslog_server:string} %{SYSLOGPROG}: %{IP:client_ip:string}:%{INT:client_port:integer} \[%{HAPROXYDATE:accept_date:string}\] %{NOTSPACE:frontend_name:string} %{NOTSPACE:backend_name:string}/%{NOTSPACE:server_name:string} %{INT:time_request:integer}/%{INT:time_queue:integer}/%{INT:time_backend_connect:integer}/%{INT:time_backend_response:integer}/%{NOTSPACE:time_duration:string} %{INT:http_status_code:integer} %{NOTSPACE:bytes_read:string} %{DATA:captured_request_cookie:strig} %{DATA:captured_response_cookie:string} %{NOTSPACE:termination_state:string} %{INT:actconn:integer}/%{INT:feconn:integer}/%{INT:beconn:integer}/%{INT:srvconn:integer}/%{NOTSPACE:retries:integer} %{INT:srv_queue:integer}/%{INT:backend_queue:integer} (\{%{HAPROXYCAPTUREDREQUESTHEADERS}\})?( )?(\{%{HAPROXYCAPTUREDRESPONSEHEADERS}\})?( )?"(<BADREQ>|(%{WORD:http_verb:string} (%{URIPROTO:http_proto:string}://)?(?:%{USER:http_user:string}(?::[^@]*)?@)?(?:%{URIHOST:http_host:string})?(?:%{URIPATHPARAM:http_request:string})?( HTTP/%{NUMBER:http_version:float})?))?"
   </grok>
   <grok>
     pattern %{IP:client_ip:string}:%{INT:client_port:integer} \[%{HAPROXYDATE:accept_date:string}\] %{NOTSPACE:frontend_name:string} %{NOTSPACE:backend_name:string}/%{NOTSPACE:server_name:string} %{INT:time_request:integer}/%{INT:time_queue:integer}/%{INT:time_backend_connect:integer} %{INT:http_status_code:integer} %{NOTSPACE:termination_state:string} %{INT:actconn:integer}/%{INT:feconn:integer}/%{INT:beconn:integer}/%{INT:srvconn:integer}/%{INT:retries:integer} %{INT:srv_queue:integer}/%{INT:backend_queue:integer}
   </grok>
 </parse>

and i was trying to get gauge metrics of actconn,feconn

[okkez]

Show me your error message, please.
BTW, typo in your configuration:

%{DATA:captured_request_cookie:strig}
%{DATA:captured_request_cookie:string}

[hari9973]

I fixed the typo and this is my error message
2019-02-05 08:48:18 +0530 [warn]: #0 prometheus: failed to instrument a metric. error_class=ArgumentError error=#<ArgumentError: value must be a number> tag="HAProxy" name="HAProxy_demo"

And this is my metrics block

<metric>
    name HAProxy_demo
    type gauge
    desc demo gauge metrics
    key srv_queue
    <labels>
      source ${tailed_path}
    </labels>
  </metric>

No metrics showing up in /metrics url

HAProxy_response_upstream_transmit_time_milliseconds_count{source="/home/hari/Desktop/test/HAProxy.txt",method="",status="246"} 1
# TYPE HAProxy_demo gauge
# HELP HAProxy_demo demo gauge metrics
# TYPE server_log_level_count counter

[okkez]

2019-02-05 08:48:18 +0530 [warn]: #0 prometheus: failed to instrument a metric. error_class=ArgumentError error=#<ArgumentError: value must be a number> tag="HAProxy" name="HAProxy_demo"

This log is from fluent-plugin-prometheus, not from this plugin.

[hari9973]

this is my full config with one grok pattern and without prometheus filters and just outputed the records to stdout

with one grok pattern
config:-

<source>
  @type tail
  tag HAProxy
  path /home/hari/Desktop/test/HAProxy.txt
  pos_file /home/hari/Desktop/test/fluent_HAProxy.pos
  path_key tailed_path
  <parse>
   @type grok
   types actconn:integer
   grok_pattern (?:%{SYSLOGTIMESTAMP:syslog_timestamp:string}|%{TIMESTAMP_ISO8601:timestamp8601:string}) %{IPORHOST:syslog_server:string} %{SYSLOGPROG:temp:string}: %{IP:client_ip:string}:%{INT:client_port:integer} \[%{HAPROXYDATE:accept_date:string}\] %{NOTSPACE:frontend_name:string} %{NOTSPACE:backend_name:string}/%{NOTSPACE:server_name:string} %{INT:time_request:integer}/%{INT:time_queue:integer}/%{INT:time_backend_connect:integer}/%{INT:time_backend_response:integer}/%{NOTSPACE:time_duration:string} %{INT:http_status_code:integer} %{NOTSPACE:bytes_read:string} %{DATA:captured_request_cookie:string} %{DATA:captured_response_cookie:string} %{NOTSPACE:termination_state:string} %{INT:actconn:integer}/%{INT:feconn:integer}/%{INT:beconn:integer}/%{INT:srvconn:integer}/%{NOTSPACE:retries:integer} %{INT:srv_queue:integer}/%{INT:backend_queue:integer} (\{%{HAPROXYCAPTUREDREQUESTHEADERS:temp1:string}\})?( )?(\{%{HAPROXYCAPTUREDRESPONSEHEADERS:temp2:string}\})?( )?"(<BADREQ>|(%{WORD:http_verb:string} (%{URIPROTO:http_proto:string}://)?(?:%{USER:http_user:string}(?::[^@]*)?@)?(?:%{URIHOST:http_host:string})?(?:%{URIPATHPARAM:http_request:string})?( HTTP/%{NUMBER:http_version:float})?))?"
 </parse>
</source>

<match HAProxy>
  @type copy
  <store>
    @type stdout
  </store>
</match>

output:-

2019-02-06 17:05:07.758048944 +0530 HAProxy: {"syslog_timestamp":"Jan 31 06:32:15","syslog_server":"s04","temp":"haproxy[16603]","program":"haproxy","pid":"16603","client_ip":"42.107.156.44","client_port":"34452","accept_date":"31/Jan/2019:06:32:15.110","haproxy_monthday":"31","haproxy_month":"Jan","haproxy_year":"2019","haproxy_time":"06:32:15","haproxy_hour":"06","haproxy_minute":"32","haproxy_second":"15","haproxy_milliseconds":"110","frontend_name":"example.com","backend_name":"example.com","server_name":"g2","time_request":"103","time_queue":"0","time_backend_connect":"0","time_backend_response":"1","time_duration":"+104","http_status_code":"200","bytes_read":"+188","captured_request_cookie":"-","captured_response_cookie":"-","termination_state":"----","actconn":135,"feconn":"135","beconn":"1","srvconn":"0","retries":"0","srv_queue":"0","backend_queue":"0","http_verb":"POST","http_request":"/logs/log.php","http_version":"1.1","tailed_path":"/home/hari/Desktop/test/HAProxy.txt"}

As you can see the actconn value is an integer
and other observation is that if i did not specify types parameter even though types are in grok pattern it still gives string values

with multi grok pattern
config:-

<source>
  @type tail
  tag HAProxy
  path /home/hari/Desktop/test/HAProxy.txt
  pos_file /home/hari/Desktop/test/fluent_HAProxy.pos
  path_key tailed_path
  <parse>
   @type grok
   <grok>
     pattern (?:%{SYSLOGTIMESTAMP:syslog_timestamp:string}|%{TIMESTAMP_ISO8601:timestamp8601:string}) %{IPORHOST:syslog_server:string} %{SYSLOGPROG:temp:string}: %{IP:client_ip:string}:%{INT:client_port:integer} \[%{HAPROXYDATE:accept_date:string}\] %{NOTSPACE:frontend_name:string} %{NOTSPACE:backend_name:string}/%{NOTSPACE:server_name:string} %{INT:time_request:integer}/%{INT:time_queue:integer}/%{INT:time_backend_connect:integer}/%{INT:time_backend_response:integer}/%{NOTSPACE:time_duration:string} %{INT:http_status_code:integer} %{NOTSPACE:bytes_read:string} %{DATA:captured_request_cookie:string} %{DATA:captured_response_cookie:string} %{NOTSPACE:termination_state:string} %{INT:actconn:integer}/%{INT:feconn:integer}/%{INT:beconn:integer}/%{INT:srvconn:integer}/%{NOTSPACE:retries:integer} %{INT:srv_queue:integer}/%{INT:backend_queue:integer} (\{%{HAPROXYCAPTUREDREQUESTHEADERS:temp1:string}\})?( )?(\{%{HAPROXYCAPTUREDRESPONSEHEADERS:temp2:string}\})?( )?"(<BADREQ>|(%{WORD:http_verb:string} (%{URIPROTO:http_proto:string}://)?(?:%{USER:http_user:string}(?::[^@]*)?@)?(?:%{URIHOST:http_host:string})?(?:%{URIPATHPARAM:http_request:string})?( HTTP/%{NUMBER:http_version:float})?))?"
   </grok>
   <grok>
     pattern %{IP:client_ip:string}:%{INT:client_port:integer} \[%{HAPROXYDATE:accept_date:string}\] %{NOTSPACE:frontend_name:string} %{NOTSPACE:backend_name:string}/%{NOTSPACE:server_name:string} %{INT:time_request:integer}/%{INT:time_queue:integer}/%{INT:time_backend_connect:integer} %{INT:http_status_code:integer} %{NOTSPACE:termination_state:string} %{INT:actconn:integer}/%{INT:feconn:integer}/%{INT:beconn:integer}/%{INT:srvconn:integer}/%{INT:retries:integer} %{INT:srv_queue:integer}/%{INT:backend_queue:integer}
   </grok>
 </parse>
</source>

<match HAProxy>
  @type copy
  <store>
    @type stdout
  </store>
</match>

output:-

2019-02-06 17:15:44.280216860 +0530 HAProxy: {"syslog_timestamp":"Jan 31 06:32:15","syslog_server":"s04","temp":"haproxy[16603]","program":"haproxy","pid":"16603","client_ip":"76.202.85.81","client_port":"60117","accept_date":"31/Jan/2019:06:32:15.574","haproxy_monthday":"31","haproxy_month":"Jan","haproxy_year":"2019","haproxy_time":"06:32:15","haproxy_hour":"06","haproxy_minute":"32","haproxy_second":"15","haproxy_milliseconds":"574","frontend_name":"indianrummynetwork-com","backend_name":"games-indianrummynetwork-com-backend","server_name":"games01","time_request":"0","time_queue":"0","time_backend_connect":"0","time_backend_response":"11","time_duration":"+11","http_status_code":"200","bytes_read":"+188","captured_request_cookie:strig":"-","captured_response_cookie":"-","termination_state":"----","actconn":"135","feconn":"135","beconn":"1","srvconn":"0","retries":"0","srv_queue":"0","backend_queue":"0","http_verb":"POST","http_request":"/messaging/players_online.php","http_version":"1.1","tailed_path":"/home/hari/Desktop/test/HAProxy.txt"}

The values are still strings

[okkez]

Could you show me raw logs from HAProxy?

[hari9973]

sample logs for two patterns

Feb 1 16:0:00 s4 haproxy[16608]: 183.8.83.115:29962 [01/Feb/2019:16:05:00.000] ssl-example1-com~ ssl-example2-com/http-01 26/1/65 239 -- 16/16/16/3/0 0/0
Jan 31 06:32:15 s4 haproxy[16603]: 86.15.164.91:53475 [31/Jan/2019:06:32:14.897] example-com gam-example2-com-back/gam03 190/0/1/13/+204 200 +188 - - ---- 135/135/1/0/0 0/0 "POST /messaging/online.php HTTP/1.1"

[okkez]

@hari9973 Could you try #64 ?