Remove event.timezone from events from some json logs #13918
Conversation
Filebeat modules for Elasticsearch and Logstash support two different log formats, the JSON one contains timezones, so it doesn't need the `event.timezone` added by `add_locale` for date parsing. Also having this added `event.timezone` can be inconsistent in some cases as it may be different to the timezone of the logs parsed. Don't add this field when the log message is in JSON format.
jsoriano
added
review
Filebeat
|
Pinging @elastic/stack-monitoring (Stack monitoring) |
| @@ -6,4 +6,5 @@ paths: | |||
| exclude_files: [".gz$"] | |||
|
|
|||
| processors: | |||
| - add_locale: ~ | |||
| # Locale for timezone is only needed in non-json logs | |||
| - add_locale.when.not.regexp.message: "^{" | |||
There was a problem hiding this comment.
I thought we discussed (on Zoom) about leaving this as-is but modifying the JSON processing ingest pipelines to remove event.timezone? That way we wouldn't be duplicating the regexp check in multiple places.
There was a problem hiding this comment.
True, but while doing the change I thought that we would be removing event.timezone even if a user adds it for any other reason, what would be counter-intuitive and probably hard to debug (though probably this is a quite corner case).
If we avoid executing add_locale when not needed as is done here, we still keep the possibility of using add_locale or manually adding this field if a user wants it.
Even if we have the regexp in two places, any change there will make tests with example logs fail, and I don't expect to change these patterns so much.
I'd slightly prefer to don't run add_locale if not needed, but I don't have a strong opinion, both options look fine to me, so as you prefer.
There was a problem hiding this comment.
Yeah, I prefer the option of not having to unnecessarily run this processor if we don't need to either. I just wish there was a way not to duplicate the same check in multiple places, but I don't see a way around that given that some processing happens on the beats side and some on the ES side.
Let's go with what you have here.
|
Thanks @ycombinator for your feedback! |
Filebeat modules for Elasticsearch and Logstash support two different
log formats, the JSON one contains timezones, so it doesn't need the
event.timezoneadded byadd_localefor date parsing. Also havingthis added
event.timezonecan be inconsistent in some cases as itmay be different to the timezone of the logs parsed. Don't add this
field when the log message is in JSON format.
More context about this in #13874 (review)
Part of #13877