Update Filebeat Traefik module to handle mixed Common and Combined Log Format #8768

sayden · 2018-10-26T15:23:58Z

This is a continuation of PRs #6488 and #6136

Previous PR's were misleading because there's no difference between Traefik 1.5 and 1.7 log formats but as you can see in this issue Traefik could miss some quote which would make our parser to fail. So those PR's were to address that issue.

Traefik uses by default since 1.4 version (at least) the Combined Log Format https://httpd.apache.org/docs/trunk/logs.html#combined also used in Apache. In 1.7 it still uses this format but you can also setup Common Log Format or JSON. Maybe we should open an issue to support JSON in the future.

Solves #8015 and #6111

frontend_name field was added so Traefik frontend_name is missing #8015 is solved.
The Grok pattern have been improved to handle the case of Traefik module for Filebeat is not working as expected #6111 and the log lines described there have been added to the test cases successfully so the issue is also solved.
Added user_identifier field which was hardcoded to '-' before
Added duration field which was missing too
Added request_count field
Numeric fields added as string have been converted to long like response_code

sayden · 2018-10-26T19:57:41Z

GeoIP plugin error, as usual

ruflin · 2018-10-29T07:56:03Z

The geoip problem should be solve for quite some time now. Make sure to run the most recent master version locally and run the test command with GENERATE=1.

For the IP address: I would prefer to keep the "original" IP addresses instead of defaulting to 1.2.3.4 in this PR.

ruflin · 2018-10-29T07:57:17Z

Can you add a CHANGELOG entry?

I also added the needs_backport label. I'm thinking we should backport this to 6.x and potentially event 6.5. @jsoriano WDYT?

filebeat/module/traefik/access/ingest/pipeline.json

filebeat/module/traefik/access/test/test.log-expected.json

jsoriano · 2018-10-29T11:55:43Z

@ruflin it'd be great if it could make it to 6.5, yes

sayden · 2018-10-29T21:38:26Z

@ruflin about the 1.7 format that you mention here. Do you think it should be added to this PR already or we open a new one once this is merged?

ruflin · 2018-10-30T06:32:37Z

@sayden Let's keep this focused on 1.5 for now and follow up on 1.7 afterwards. Let me know if you need help with the generated data.

jsoriano · 2018-11-27T10:23:05Z

This seems to need an update.

sayden · 2018-11-27T12:34:48Z

Apart from the fields.go file, I'm carefully looking at the rebase as there were some updated in the expected JSON, I think coming from the ECS migration

ruflin · 2018-11-27T13:37:24Z

This will soon conflict with this PR. @webmat for awareness: #9005 I would suggest to get this one in and then ECS to make sure we cover all the fields but please coordinate together.

webmat · 2018-11-27T14:32:01Z

The race is on, to merge first! ;-)

Kidding, I don't mind if this gets in first. Looks like this PR needs a rebase, and has test failures, though.

I can keep my Traefik PR for last to give this time to get in, but note that I'd like to merge mine today.

@sayden did you expect to be finishing this up today?

sayden · 2018-11-27T17:26:11Z

@webmat If you want and you are close to finish just go on and I'll resolve conflicts later. I prefer to be careful with small details and take my time 👍

webmat · 2018-11-27T18:09:36Z

Perfect.

Note I may not get to my own Traefik PR today, as I need to work on something else before this.

ruflin · 2018-11-28T12:38:04Z

filebeat/module/traefik/access/ingest/pipeline.json

            }
        },
        {
-            "rename": {
+            "remove": {


What is the reason we remove this one now?

After spending some minutes looking at it without understanding very well why either, I think that in a first look it made me think that it was a typo, because the field is set in the previous rule. Then, in a second look (maybe I couple of days later) I probably saw the "target_field": "traefik.access.user_agent.original" and I also thought it was some kind of typo that shouldn't be there and I delete it.

I guess that the approach here would be to not leave it again as a rename to preserve the original user_agent extracted from the Grok pattern, right?

Yes, please keep this field around

ruflin · 2018-11-28T12:39:36Z

filebeat/module/traefik/access/ingest/pipeline.json

-                ],
-                "ignore_missing": true
+                    "(?:%{NUMBER:traefik.access.body_sent.bytes}|-)( \"%{DATA:traefik.access.referrer}\")?( \"%{DATA:traefik.access.agent}\")?(?:%{NUMBER:traefik.access.request_count}|-)?( \"%{DATA:traefik.access.frontend_name}\")?( \"%{DATA:traefik.access.backend_url}\")?",
+                    "%{IPORHOST:traefik.access.remote_ip} - %{DATA:traefik.access.user_name} \\[%{HTTPDATE:traefik.access.time}\\] \"%{WORD:traefik.access.method} %{DATA:traefik.access.url} HTTP/%{NUMBER:traefik.access.http_version}\" (?:%{NUMBER:traefik.access.response_code}|-) (?:%{NUMBER:traefik.access.body_sent.bytes}|-)( \"%{DATA:traefik.access.referrer}\")?( \"%{DATA:traefik.access.agent}\")?(?: %{NUMBER:traefik.access.request_count}| -)?( \"%{DATA:traefik.access.frontend_name}\"| -)?( \"%{DATA:traefik.access.backend_url}\"| -)?( %{NUMBER:traefik.access.response_time}ms)?",


I see we add 2 new grok patterns: How are they different? Below we only add 1 new log line to test. Would be good to have at least 1 for each new pattern.

Yes, I also think so. I'll add a new input test line.

sayden · 2018-11-30T11:58:55Z

After digging deeply into this issue I discovered that there's nothing like Traefik logs format but that they have always used CLF - Common Log Format or Combined Log Format.

The problem is that Traefik had a bug prior 1.5 where if the request_referrer field was empty it didn't match the requirements of CLF because quotes were missing. Fix is merged here.

Current version of Traefik (1.7) uses Combined Log Format as default output format.

I suggest to leave the pattern with Combined Log Format to "force" Traefik users to update. It doesn't make sense that we have a modified Grok pattern just to cover some bug in a different product that we will have track to eventually delete it. Option 2 is to try to cover both: combined and common log format although it seems that Common is not used in Traefik right now.

WDYT @ruflin?

…ollow up PR

…d in dissect and in the ingest node again (instead of only the ingest node)

ruflin · 2018-12-05T19:02:03Z

filebeat/test.json

@@ -0,0 +1,3171 @@
+{


@sayden Is this file a leftover?

ruflin

Waiting on CI to go green.

…g Format (elastic#8768) * Added support for Common Log Format and Combined Log Format in Traefik which is the default format until now. * Added more log lines, including one in Common Log Format * Added user_identifier field which was hardcoded to '-' before * Added log test inputs where the user_name and user_identifier were set and not set. * Added duration field which was missing * Numeric fields added as string have been converted to long * Added request count field * Added two log lines more that covers issue elastic#6111 * Make bytes_sent field of type long. Most Grok expressions are optional now * Added CHANGELOG entry (cherry picked from commit 0e9054c)

ruflin · 2018-12-07T11:39:16Z

@sayden I suggest we follow up this PR with moving the dissect processing to the ingest pipeline. It should still be dissect as it's more efficient but like this we have everything in one place if we want to do reprocessing for example. We can do this change for 7.0 but not backport it.

…xed Common and Combined Log Format (#9419) * Update Filebeat Traefik module to handle mixed Common and Combined Log Format (#8768) * Added support for Common Log Format and Combined Log Format in Traefik which is the default format until now. * Added more log lines, including one in Common Log Format * Added user_identifier field which was hardcoded to '-' before * Added log test inputs where the user_name and user_identifier were set and not set. * Added duration field which was missing * Numeric fields added as string have been converted to long * Added request count field * Added two log lines more that covers issue #6111 * Make bytes_sent field of type long. Most Grok expressions are optional now * Added CHANGELOG entry (cherry picked from commit 0e9054c) * Updated fields.go file * Upload pre-ECS generated JSON expected file which differs from master

…g Format (elastic#8768) * Added support for Common Log Format and Combined Log Format in Traefik which is the default format until now. * Added more log lines, including one in Common Log Format * Added user_identifier field which was hardcoded to '-' before * Added log test inputs where the user_name and user_identifier were set and not set. * Added duration field which was missing * Numeric fields added as string have been converted to long * Added request count field * Added two log lines more that covers issue elastic#6111 * Make bytes_sent field of type long. Most Grok expressions are optional now * Added CHANGELOG entry (cherry picked from commit 0e9054c) fields.go file is updated and pre-ECS generated JSON expected file which differs from master is used # Conflicts: # filebeat/include/fields.go # filebeat/module/traefik/access/test/test.log-expected.json

…xed Common and Combined Log Format (#9439) * Update Filebeat Traefik module to handle mixed Common and Combined Log Format (#8768) * Added support for Common Log Format and Combined Log Format in Traefik which is the default format until now. * Added more log lines, including one in Common Log Format * Added user_identifier field which was hardcoded to '-' before * Added log test inputs where the user_name and user_identifier were set and not set. * Added duration field which was missing * Numeric fields added as string have been converted to long * Added request count field * Added two log lines more that covers issue #6111 * Make bytes_sent field of type long. Most Grok expressions are optional now * Added CHANGELOG entry (cherry picked from commit 0e9054c) fields.go file is updated and pre-ECS generated JSON expected file which differs from master is used # Conflicts: # filebeat/include/fields.go # filebeat/module/traefik/access/test/test.log-expected.json

…ndle mixed Common and Combined Log Format (elastic#9439) * Update Filebeat Traefik module to handle mixed Common and Combined Log Format (elastic#8768) * Added support for Common Log Format and Combined Log Format in Traefik which is the default format until now. * Added more log lines, including one in Common Log Format * Added user_identifier field which was hardcoded to '-' before * Added log test inputs where the user_name and user_identifier were set and not set. * Added duration field which was missing * Numeric fields added as string have been converted to long * Added request count field * Added two log lines more that covers issue elastic#6111 * Make bytes_sent field of type long. Most Grok expressions are optional now * Added CHANGELOG entry (cherry picked from commit 5db2afb) fields.go file is updated and pre-ECS generated JSON expected file which differs from master is used # Conflicts: # filebeat/include/fields.go # filebeat/module/traefik/access/test/test.log-expected.json

sayden added in progress Pull request is currently in progress. module Filebeat Filebeat enhancement labels Oct 26, 2018

ruflin added the needs_backport PR is waiting to be backported to other branches. label Oct 29, 2018

jsoriano reviewed Oct 29, 2018

View reviewed changes

filebeat/module/traefik/access/ingest/pipeline.json Outdated Show resolved Hide resolved

filebeat/module/traefik/access/test/test.log-expected.json Outdated Show resolved Hide resolved

ruflin mentioned this pull request Oct 29, 2018

fix grok expression for traefik #6136

Closed

ruflin added the Team:Integrations Label for the Integrations team label Nov 23, 2018

ruflin added Team:Integrations Label for the Integrations team and removed Team:Integrations Label for the Integrations team labels Nov 27, 2018

sayden force-pushed the traefik-additional-logs-parsing branch from 461872a to 100baa6 Compare November 27, 2018 15:12

ruflin reviewed Nov 28, 2018

View reviewed changes

sayden force-pushed the traefik-additional-logs-parsing branch from a68ab30 to 17361bb Compare November 30, 2018 08:52

This was referenced Dec 3, 2018

Traefik module: frontend_name and backend_url fields are classified as String only. #6467

Closed

Traefik frontend_name is missing #8015

Closed

sayden added 8 commits December 5, 2018 17:01

Remove keyword type for frontend_name and backend_url. Will do in a f…

9ed423d

…ollow up PR

Added CHANGELOG entry

fb0367a

Added two log lines more that covers issue 6111

fc9e192

Added CHANGELOG entry

febb025

Added dot at the end of the CHANGELOG line

a0c247c

Leave old log lines untouched

566eb0a

Leave the original pipeline incoming field name so the input is parse…

9bcc9c7

…d in dissect and in the ingest node again (instead of only the ingest node)

Make bytes_sent field of type long. Most expressions are optional now

6737664

sayden force-pushed the traefik-additional-logs-parsing branch from ed9f4ea to 6737664 Compare December 5, 2018 16:09

ruflin reviewed Dec 5, 2018

View reviewed changes

Delete leftover file

092d629

ruflin approved these changes Dec 6, 2018

View reviewed changes

sayden merged commit 0e9054c into elastic:master Dec 6, 2018

sayden mentioned this pull request Dec 6, 2018

Cherry-pick #8768 to 6.x: Update Filebeat Traefik module to handle mixed Common and Combined Log Format #9419

Merged

sayden added v6.6.0 and removed needs_backport PR is waiting to be backported to other branches. labels Dec 6, 2018

ruflin added review and removed in progress Pull request is currently in progress. labels Dec 7, 2018

sayden mentioned this pull request Dec 7, 2018

Move dissect tokenizing to Ingest pipeline in Traefik Filebeat Module #9434

Closed

This was referenced Dec 7, 2018

Add traefik 1.5 formatted logs and update grok rule to handle them #6488

Closed

Traefik module for Filebeat is not working as expected #6111

Closed

sayden mentioned this pull request Dec 7, 2018

Cherry-pick #8768 to 6.5: Update Filebeat Traefik module to handle mixed Common and Combined Log Format #9439

Merged

sayden added the v6.5.3 label Dec 7, 2018

sayden deleted the traefik-additional-logs-parsing branch July 30, 2022 05:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Filebeat Traefik module to handle mixed Common and Combined Log Format #8768

Update Filebeat Traefik module to handle mixed Common and Combined Log Format #8768

sayden commented Oct 26, 2018 •

edited

sayden commented Oct 26, 2018

ruflin commented Oct 29, 2018

ruflin commented Oct 29, 2018

jsoriano commented Oct 29, 2018

sayden commented Oct 29, 2018

ruflin commented Oct 30, 2018

jsoriano commented Nov 27, 2018

sayden commented Nov 27, 2018

ruflin commented Nov 27, 2018

webmat commented Nov 27, 2018

sayden commented Nov 27, 2018

webmat commented Nov 27, 2018

ruflin Nov 28, 2018

sayden Nov 28, 2018

webmat Nov 28, 2018

ruflin Nov 28, 2018

sayden Nov 28, 2018

sayden commented Nov 30, 2018

ruflin Dec 5, 2018

sayden Dec 6, 2018

ruflin left a comment

ruflin commented Dec 7, 2018

Update Filebeat Traefik module to handle mixed Common and Combined Log Format #8768

Update Filebeat Traefik module to handle mixed Common and Combined Log Format #8768

Conversation

sayden commented Oct 26, 2018 • edited

sayden commented Oct 26, 2018

ruflin commented Oct 29, 2018

ruflin commented Oct 29, 2018

jsoriano commented Oct 29, 2018

sayden commented Oct 29, 2018

ruflin commented Oct 30, 2018

jsoriano commented Nov 27, 2018

sayden commented Nov 27, 2018

ruflin commented Nov 27, 2018

webmat commented Nov 27, 2018

sayden commented Nov 27, 2018

webmat commented Nov 27, 2018

ruflin Nov 28, 2018

Choose a reason for hiding this comment

sayden Nov 28, 2018

Choose a reason for hiding this comment

webmat Nov 28, 2018

Choose a reason for hiding this comment

ruflin Nov 28, 2018

Choose a reason for hiding this comment

sayden Nov 28, 2018

Choose a reason for hiding this comment

sayden commented Nov 30, 2018

ruflin Dec 5, 2018

Choose a reason for hiding this comment

sayden Dec 6, 2018

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

ruflin commented Dec 7, 2018

sayden commented Oct 26, 2018 •

edited