Some log lines are absent from ALB access log

[nhlushak]

According to syntax of ALB access logs, there are 5 fields of log line that are missing if using this plugin. The most important one are "actions taken" and "error reason", that are useful during troubleshooting. Is it possible to rewrite regex used in plugin to get these fields?

[shinsaka]

@NikitaGl Thank you for request.
I changed REGEX, and released version 1.2.0. 😃

[nhlushak]

@shinsaka Just tested updated version, thanks a lot, it works great!
Despite all fields are now present, is it possible to allow preserving entire log line "as is"? Like this:

"@message": "http 2019-10-30T15:39:10.664331Z app/xxx-lb/xxxxxxx x.x.x.x:58463 172.62.5.25:80 0.000 0.901 0.000 500 500 239 212 \"GET http://app.domain.com:80/ HTTP/1.1\" \"PostmanRuntime/7.15.2\" - - arn:aws:elasticloadbalancing:us-west-2:xxxxxxxxxxxx:targetgroup/xxxxxxxxxxxx/xxxxxxxxxx \"Root=....\" \"-\" \"-\" 5 2019-10-30T15:39:09.763000Z \"forward\" \"-\" \"-\" \"172.x.x.x:80\" \"200\""

[shinsaka]

@NikitaGl
I added include_all_message setting on config.
When set true then output all_message.
Try version 1.3.1.
Thank you!

[nhlushak]

Hi @shinsaka, thanks a lot for implementing my requests.
I've tested v1.3.1, but it fails to process alb logs from s3, no matter if include_all_message option is set or not.
Here is an error message:
fluentd_logger_1 | 2019-11-08 16:12:59 +0000 [warn]: #0 error occurred: wrong number of arguments (given 2, expected 1)

[shinsaka]

@NikitaGl sorry, v1.3.1 does not work on ruby version 2.5.
Fixed to work on ruby version 2.5.
Try version v1.3.2 🙇

[nhlushak]

Yup, seems to be working now.
I'm still not understand the reason for optionN fields in output record:

    "option1": null,
    "option2": null,
    "option3": "- \"Root=****\" \"x.x.x.x\" \"arn:aws:acm:*****:******:certificate/****-****-****-****-*****\" 0 2019-11-07T19:52:22.680000Z \"fixed-response\" \"-\" \"-\" \"-\" \"-\"",

As you see, sometimes it catches values, that corresponds to other keys. In my tests it always happens to logs of requests that did not passed through ALB and been answered with fixed response.
Here's and full example of such log. I made it with include_all_message disabled, but it makes no difference

{
  "_index": "xxxxxxxxxx-2019.11.11",
  "_type": "_doc",
  "_id": "xxxxxxxxxxxxxxxxxxxxxx",
  "_version": 1,
  "_score": null,
  "_source": {
    "account_id": "xxxxxxxxxxxxx",
    "region": "xxx-xxxxx-xx",
    "logfile_date": "2019/11/11",
    "logfile_elb_name": "xxx.xxxxxxxxxxxx-lb.1xxxxxxxxxxxx",
    "elb_ip_address": "xx.xx.xx.xx",
    "logfile_hash": "xxxxxxxx",
    "elb_timestamp": "20191111T0530Z",
    "key": "xxxxxxxxx/AWSLogs/xxxxxxxxxxx/elasticloadbalancing/xx-xxxx-xx/2019/11/11/xxxxxxxxxx_elasticloadbalancing_xx-xxxxxxx-xx_app.xxxxxx-lb.xxxxxxxx_20191111T0530Z_xx.xx.xx.xx_xxxxx.log.gz",
    "prefix": "xxxxxx",
    "elb_timestamp_unixtime": 1573450200,
    "s3_last_modified_unixtime": 1573450208,
    "time": "2019-11-11T05:28:32.708475+0000",
    "elb": "app/xxxxxx-lb/xxxxxxxxxxxxxxx",
    "client": "xx.xx.xx.xx",
    "client_port": "xxx",
    "backend": "-",
    "backend_port": null,
    "request_processing_time": -1,
    "backend_processing_time": -1,
    "response_processing_time": -1,
    "elb_status_code": "404",
    "backend_status_code": "-",
    "received_bytes": 202,
    "sent_bytes": 178,
    "request_method": "GET",
    "request_uri": "https://xx.xx.xx.xx:443/",
    "request_protocol": "HTTP/1.1",
    "user_agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36",
    "ssl_cipher": "ECDHE-RSA-AES128-GCM-SHA256",
    "ssl_protocol": "TLSv1.2",
    "type": "https",
    "target_group_arn": null,
    "trace_id": null,
    "domain_name": null,
    "chosen_cert_arn": null,
    "matched_rule_priority": null,
    "request_creation_time": null,
    "actions_executed": null,
    "redirect_url": null,
    "error_reason": null,
    "option1": null,
    "option2": null,
    "option3": "- \"Root=1-xxxxx-xxxxxxxxxxxxxxx\" \"-\" \"arn:aws:acm:xx-xx-x:xxxxxxxxxxxx:certificate/xxxxxx-xxxx-xxxx-xxxx-xxxxxxx\" 0 2019-11-11T05:28:32.708000Z \"fixed-response\" \"-\" \"-\" \"-\" \"-\"",
    "@timestamp": "2019-11-11T05:28:32.000000000+00:00"
  },
  "fields": {
    "logfile_date": [
      "2019-11-11T00:00:00.000Z"
    ],
    "@timestamp": [
      "2019-11-11T05:28:32.000Z"
    ],
    "time": [
      "2019-11-11T05:28:32.708Z"
    ]
  },
  "sort": [
    1573450112000
  ]
}

[nhlushak]

Consider issue to be resolved, thanks.