Some filebeat module reports wrong year in timestamp

[kaiyan-sheng]

When timestamp in log entries has no year, such as 30 May 12:23:52.442, current year will be added to the timestamp. This problem originally happened in the filebeat unit tests #9849 and was fixed by removing timestamp in unit test output. Now this is happening also in users environments when year is missing in log entries. For example: elastic/kibana#28259.

[weltenwort]

Just to clarify: The missing year seems to be guessed by Elasticsearch at the indexing time because filebeat just passes through the ambiguous timestamp string. So in order to fix that on the filebeat side it would have to add the year to the timestamp before passing it on to Elasticsearch.

But then again, how would it know that it's a current and not a historic log from 10 years ago that is being read. 🤔

[kaiyan-sheng]

Yeah good point. Let's say today is 01-10-2019, so if timestamp has MM-DD to be before 01-10, then gives a timestamp year as 2019. If MM-DD is larger/after 01-10, then gives a timestamp year as 2018. But as you said, it can also be logs 10 years ago instead of last year.

[jsoriano]

An strategy could be to use the time of the file to obtain the year when the file was written. This may need some special handling for files written during new year, but it should work for normal logging use cases.

For historical logs whose original date is not preserved in the file we may need some extra configs, or we could document that users can modify the time of the files to the year they expect.

[ruflin]

What about a log file that was created 2018-12-31 23:00 and contains events from 2018 and 2019? If we have a config or detect the file creation or change data, all events would either contain 2018 or 2019 as year.

[jsoriano]

Yes, this is what I meant with needing some special handling for files written during new year 🙂 But I cannot think on a good solution 😕

We could use the year of the creation time and then increase it if on the same file we are going back to the past (to a previous month), but creation time is not usually set 🙁 this would require manual config.

Doing it the other way around (using the modification time) would require to modify events from log lines written before the change of year, or to read the file twice, with a first pass only to check if it contains a change of year.

[weltenwort]

How about showing a warning to the user about ambiguous timestamps as a first step?

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.