Support haproxy logs without captured headers

[jsoriano]

haproxy logs don't contain captured headers by default, but current grok patterns expect them, make them optional and add test cases for them.

Some examples:

Without captured headers:

Dec 10 12:01:46 voyager haproxy[19312]: 127.0.0.1:35982 [10/Dec/2018:12:01:46.395] http-webservices http-webservices/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"

With request captured headers:

Dec 10 15:46:49 voyager haproxy[29785]: 127.0.0.1:43738 [10/Dec/2018:15:46:49.497] http-webservices http-webservices/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 1/1/0/0/0 0/0 {localhost:8888||} "GET /foo HTTP/1.1"

With request and response captured headers:

Dec 10 15:48:56 voyager haproxy[7873]: 127.0.0.1:44542 [10/Dec/2018:15:48:56.017] http-webservices http-webservices/<NOSRV> 0/-1/-1/-1/0 503 213 - - SC-- 1/1/0/0/0 0/0 {localhost:8888||} {|} "GET /foo HTTP/1.1"

Headers capture is configured in haproxy config with options like:

  capture request header Host len 15
  capture request header X-Forwarded-For len 15
  capture request header Referer len 15
  capture response header Content-length len 9
  capture response header Location len 15

Notice that when only one type of headers is being captured it is not possible to know on default log formats if they are request or response headers. Maybe we should ignore them by default.

[elasticmachine]

Pinging @elastic/infrastructure

[sentient]

Would it not just be simpler to document in the haproxy module the log-format you are expecting
And give us the log_format string that we can put in haproxy.cfg ?

[jsoriano]

@sentient actually it is also in our plans to document supported formats (see elastic/integrations#3250)

The idea is to support the default haproxy formats (no log options, option httplog and option tcplog).

The issue addressed here is that the log format used by option httplog includes an optional part that depends on other settings.

If you are having this issue, an option to workaround it is to add captured headers, even if they don't exist, something like:

  capture request header Host len 15
  capture response header Content-length len 9