Join GitHub today
Conversion To Unix Timestamps Is 24 Hours Behind #830
All of our servers (Amazon Linux 2015.03 running rsyslog
We use the following GELF template to format log messages and ship them to our Graylog server:
But all 'timestamp' components of the message are 24 hours in the past. As an experiment, I tried changing the dateformat to 'rfc3339' and all of the timestamps began being reported correctly. I then changed it back to unixtimestamp and it went back to being 24 hours in the past.
A quick example is below for some log messages:
When 'dateformat' is set to 'unixtimestamp':
When 'dateformat' is set to 'rfc3339':
This information was pulled from running a tcpdump on one of our application servers and capturing the logging traffic as it left the server in order to eliminate Graylog as the sou
I can't think of any reason why this could have suddenly happened (and Graylog shows that it happened to all servers at exactly the same time - midnight UTC on 29/02/2016). We made no change to any of our systems or applications from 48 hours before this occurred and have been in a change freeze since.
Thanks for your help!
I strongly think this is a bug in rsyslog: we use some custom code to compute the Unix timestamp. I think the reason was that we missed some features in the usual api because of the different time zones we need to support concurrently. My guess is that this code has a problem with this years leap year. I will investigate today and report back.
I think I have found the culprit. We are using lookup tables for year second values, but it was an oversight on how to handle month within the leap year itself. In other words: it'll auto-heal on 2017-01-01... I also noticed that we seem to be consistently one second too low.
Now working on a simple fix to the current algo. When done, I'll see if I can refactor this to mktime(), not sure what was the orginal cause why we didn't use it from day one.
Unfortunately, there is solid reason not to use mktime(). From the (new) code comment:
So we need to stay with the "home grown" code, at least the testbench now has good tests up until 2100 (which is when rsyslog needs at least some new lookup tables, so it is it's current expiration date ;)).