Fix filebeat system module timezone parsing #13308

pragkent · 2019-08-22T00:54:06Z

elasticmachine · 2019-08-22T00:54:09Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

elasticmachine · 2019-08-22T00:54:09Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

jsoriano · 2019-08-22T16:59:35Z

ok to test

jsoriano

@pragkent thanks for reporting the issue and opening this PR to fix it. I think that you are actually right and we are incorrectly handling timezones, at least since #12253 (7.3.0).

I wonder if other modules are affected too after #12311.

Could you also add a changelog entry mentioning the fix?

filebeat/module/system/auth/ingest/pipeline.json

pragkent · 2019-08-23T00:02:56Z

Sure, I'll handle this later

pragkent · 2019-08-23T00:15:02Z

And maybe I can try to fix this issue in other modules by the way.

elasticcla · 2019-08-23T04:41:53Z

Hi @pragkent, we have found your signature in our records, but it seems like you have signed with a different e-mail than the one used in your Git commit. Can you please add both of these e-mails into your Github profile (they can be hidden), so we can match your e-mails to your Github profile?

jsoriano · 2019-08-23T11:15:57Z

filebeat/module/elasticsearch/deprecation/ingest/pipeline.json

@@ -43,7 +43,8 @@
        {
            "date": {
                "if": "ctx.event.timezone != null",
-                "field": "@timestamp",


@pragkent Thanks for looking at these modules too, but let's focus first on the system module. For the elasticsearch one I think we will also need to change the formats, because it is actually like ISO8601, but without timezone.

@jsoriano yes, you are correct. we need to define a new format, or the timezone is wrong. I'll fix it later.

I grabbed some elasticsearch 7.3 logs from our server, it seems that ISO8601 format was used (timezone included). So, use ISO8601 format for parsing is actually right.

And logs from elasticsearch 6.x does not have timezone part, as you said.

So I added a format for parsing elasticsearch 6.x logs.

Could you please double-check that?

Part of the logs is listed as follow:

{"type": "deprecation", "timestamp": "2019-08-20T05:08:27,756+0000", "level": "WARN", "component": "o.e.d.c.s.Settings", "cluster.name": "hulk", "node.name": "hulk-es-22x9pqqc4q", "message": "[discovery.zen.hosts_provider] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." } {"type": "server", "timestamp": "2019-08-22T20:52:56,131+0000", "level": "INFO", "component": "o.e.m.j.JvmGcMonitorService", "cluster.name": "hulk", "node.name": "hulk-es-22x9pqqc4q", "cluster.uuid": "3pYlYTCPT4iLMqHmeq7Yew", "node.id": "UFfAvAUkTIuOlUbMgT9x0A", "message": "[gc][229011] overhead, spent [293ms] collecting in the last [1s]" }

Looks good, but I think we could remove the ISO8601 format.

Or do something like what I proposed for ISO8601 in the system module.

Hmm, I think we should not remove ISO8601 for elasticsearch module. For example, if we have elasticsearch 7.3, and have event.timezone set, then the parsing will failed without this ISO8601 format.

I am trying and this seems correct in 7.3 for JSON logs, but not for plain text logs. We may need to move date parsing to the specific pipeline (there are three pipelines, a common one, and another two for the specific needings of plain text and json formats).
Also I think that the format for timestamps without timezone should be yyyy-MM-dd'T'HH:mm:ss,SSS, as they contain milliseconds.

You are right. I just changed the format to yyyy-MM-dd'T'HH:mm:ss,SSS. When the format is yyyy-MM-dd'T'HH:mm:ss, the date processor will always choose ISO8601 format for parsing, because the time in log files contain milliseconds part.

Is there any other reasons which caused plain text logs parsing failure that I missed out?

jsoriano

Changes in Kafka and nginx modules LGTM, let's not add more modules to this PR.
I have added a couple of comments for elasticsearch and system modules.

jsoriano · 2019-08-25T11:04:22Z

filebeat/module/elasticsearch/deprecation/ingest/pipeline.json

@@ -43,7 +43,8 @@
        {
            "date": {
                "if": "ctx.event.timezone != null",
-                "field": "@timestamp",


Looks good, but I think we could remove the ISO8601 format.

filebeat/module/system/syslog/ingest/pipeline.json

pragkent · 2019-08-26T07:28:37Z

Changes in Kafka and nginx modules LGTM, let's not add more modules to this PR.
I have added a couple of comments for elasticsearch and system modules.

I just reverted changes to nginx and kafka modules.

jsoriano

It seems to be working for the system module, but I am not so sure about the elasticsearch one. What do you think about focusing in this PR on the system module and fix the other modules in other PRs?

filebeat/module/system/syslog/ingest/pipeline.json

pragkent · 2019-08-27T01:26:23Z

It seems to be working for the system module, but I am not so sure about the elasticsearch one. What do you think about focusing in this PR on the system module and fix the other modules in other PRs?

OK, I'll submit other prs for other modules

pragkent · 2019-08-27T01:29:58Z

I've updated the commits to only include changes about system module.

pragkent · 2019-08-28T09:43:33Z

@jsoriano Just created a few PRs:

#13367 elasticsearch
#13368 kafka
#13369 nginx

Continuation of #13308

Continuation of elastic#13308 (cherry picked from commit fed669d)

Continuation of #13308 (cherry picked from commit fed669d)

Continuation of #13308 (cherry picked from commit fed669d) Co-authored-by: Kent Wang <pragkent@gmail.com>

ztyzbb · 2019-09-30T05:33:19Z

Found a similar date processor usage in logstash module.
filebeat/module/logstash/log/ingest/pipeline-plain.json
Does it need a fix too?

jsoriano · 2019-10-02T10:29:56Z

@ztyzbb yes, I can confirm that this issue also happens in logstash module when parsing plain text files. I will open a PR to fix this.

Use the same date format that we are using now for Elasticsearch timestamps without timezone, that is the same format used for timestamps in Logstash plain text logs. Similar to #13308, related to #13877.

) Continuation of elastic#13308 (cherry picked from commit a433d5e)

… parsing (elastic#13358) Fix timezone handling in system module when non-UTC timezones are used. Fix elastic#13306 (cherry picked from commit af371de) Co-authored-by: Kent Wang <pragkent@gmail.com>

) Continuation of elastic#13308 (cherry picked from commit a433d5e) Co-authored-by: Kent Wang <pragkent@gmail.com>

pragkent requested a review from a team as a code owner August 22, 2019 00:54

jsoriano added bug Filebeat Filebeat module needs_backport PR is waiting to be backported to other branches. v7.3.2 v7.4.0 labels Aug 22, 2019

jsoriano reviewed Aug 22, 2019

View reviewed changes

filebeat/module/system/auth/ingest/pipeline.json Show resolved Hide resolved

jsoriano assigned jsoriano and pragkent Aug 22, 2019

pragkent requested a review from a team as a code owner August 23, 2019 04:41

jsoriano mentioned this pull request Aug 23, 2019

[Filebeat] Add timezone information in apache error fileset #13304

Merged

jsoriano reviewed Aug 23, 2019

View reviewed changes

jsoriano reviewed Aug 25, 2019

View reviewed changes

andresrc added Team:Integrations Label for the Integrations team [zube]: In Review labels Aug 26, 2019

jsoriano reviewed Aug 26, 2019

View reviewed changes

filebeat/module/system/syslog/ingest/pipeline.json Show resolved Hide resolved

Fix system module timezone parsing

1e67390

jsoriano added 2 commits August 27, 2019 13:07

Fix changelog

04a3238

Merge branch 'master' into fix-filebeat-timezone

8436f42

jsoriano approved these changes Aug 27, 2019

View reviewed changes

pragkent mentioned this pull request Aug 28, 2019

Fix filebeat elasticsearch module ingest timezone #13367

Merged

pragkent deleted the fix-filebeat-timezone branch August 28, 2019 09:26

This was referenced Aug 28, 2019

Fix filebeat kafka module ingest timezone #13368

Merged

Fix filebeat nginx module ingest timezone #13369

Merged

jsoriano pushed a commit that referenced this pull request Aug 29, 2019

Fix filebeat kafka module ingest timezone (#13368)

fed669d

Continuation of #13308

jsoriano mentioned this pull request Aug 29, 2019

Cherry-pick #13368 to 7.4: Fix filebeat kafka module ingest timezone #13398

Merged

jsoriano pushed a commit to jsoriano/beats that referenced this pull request Aug 29, 2019

Fix filebeat kafka module ingest timezone (elastic#13368)

02eb748

Continuation of elastic#13308 (cherry picked from commit fed669d)

jsoriano mentioned this pull request Aug 29, 2019

Cherry-pick #13368 to 7.3: Fix filebeat kafka module ingest timezone #13399

Merged

jsoriano pushed a commit to jsoriano/beats that referenced this pull request Aug 29, 2019

Fix filebeat kafka module ingest timezone (elastic#13368)

e758fb2

Continuation of elastic#13308 (cherry picked from commit fed669d)

jsoriano added a commit that referenced this pull request Aug 29, 2019

Fix filebeat kafka module ingest timezone (#13368) (#13398)

31677eb

Continuation of #13308 (cherry picked from commit fed669d)

jsoriano added a commit that referenced this pull request Aug 29, 2019

Fix filebeat kafka module ingest timezone (#13368) (#13399)

5ca1a15

Continuation of #13308 (cherry picked from commit fed669d) Co-authored-by: Kent Wang <pragkent@gmail.com>

andresrc removed the [zube]: Done label Sep 3, 2019

This was referenced Oct 2, 2019

Some filebeat modules incorrectly parse timestamps #13877

Closed

Fix timezone parsing in Rabbitmq logs #13879

Merged

Fix timezone parsing of logstash module ingest pipelines #13890

Merged

Fix timezone parsing of Cisco module ingest pipelines #13893

Merged

jsoriano mentioned this pull request Oct 4, 2019

Fix timezone parsing in iptables, mssql and panw modules #13926

Merged

This was referenced Oct 16, 2019

Fix filebeat logstash module ingest timezone #14085

Closed

Fix filebeat logstash module ingest timezone #14088

Closed

leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023

Fix filebeat kafka module ingest timezone (elastic#13368) (elastic#13398

41a912b

) Continuation of elastic#13308 (cherry picked from commit a433d5e)

leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023

Fix filebeat kafka module ingest timezone (elastic#13368) (elastic#13399

ddcab15

) Continuation of elastic#13308 (cherry picked from commit a433d5e) Co-authored-by: Kent Wang <pragkent@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix filebeat system module timezone parsing #13308

Fix filebeat system module timezone parsing #13308

pragkent commented Aug 22, 2019 •

edited by andresrc

elasticmachine commented Aug 22, 2019

elasticmachine commented Aug 22, 2019

jsoriano commented Aug 22, 2019

jsoriano left a comment

pragkent commented Aug 23, 2019

pragkent commented Aug 23, 2019

elasticcla commented Aug 23, 2019

jsoriano Aug 23, 2019

pragkent Aug 23, 2019

pragkent Aug 24, 2019

jsoriano Aug 25, 2019

jsoriano Aug 25, 2019

pragkent Aug 26, 2019

jsoriano Aug 26, 2019

pragkent Aug 27, 2019

jsoriano left a comment

jsoriano Aug 25, 2019

pragkent commented Aug 26, 2019

jsoriano left a comment

pragkent commented Aug 27, 2019

pragkent commented Aug 27, 2019

pragkent commented Aug 28, 2019

ztyzbb commented Sep 30, 2019

jsoriano commented Oct 2, 2019

Fix filebeat system module timezone parsing #13308

Fix filebeat system module timezone parsing #13308

Conversation

pragkent commented Aug 22, 2019 • edited by andresrc

elasticmachine commented Aug 22, 2019

elasticmachine commented Aug 22, 2019

jsoriano commented Aug 22, 2019

jsoriano left a comment

Choose a reason for hiding this comment

pragkent commented Aug 23, 2019

pragkent commented Aug 23, 2019

elasticcla commented Aug 23, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pragkent commented Aug 26, 2019

jsoriano left a comment

Choose a reason for hiding this comment

pragkent commented Aug 27, 2019

pragkent commented Aug 27, 2019

pragkent commented Aug 28, 2019

ztyzbb commented Sep 30, 2019

jsoriano commented Oct 2, 2019

pragkent commented Aug 22, 2019 •

edited by andresrc