New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experiments with Time::Moment and normalizing logfiles to UTC (#11) #13

Merged
merged 2 commits into from Mar 12, 2016

Conversation

Projects
None yet
2 participants
@StarLightPL
Contributor

StarLightPL commented Mar 2, 2016

Hello,

This change tries to implement issue #11 in a couple of ways.

First, it introduces NormalizeToUTC parser variable which makes it try to output iso 8601 compliant datetimes. It has two modes of operation - slow one using DateTime for DST calculation when your syslog doesn't have iso 8601 datetimes (i.e. the timezone offset is missing), and an ultra fast one using only Time::Moment - when your syslog is compliant.

Sadly, the slow normalize mode is slower than the previous iteration (see benchmarks), this is mainly to heavy magic going on in DateTime to calculate correct DST values needed to calculate UTC offset.

Second, this patch introduces functions for "setting syslog timezone", making the parser treat all datetimes without explicit timezones in the timestamp as belonging to the set zone. It allows you to set zones other than local and normalize them if for example you have files from different TZs.

If you don't need normalizing, and only care about literal date values, you can use IgnoreTimeZones mode - which is notably faster than the default (~300%).

You can also turn off date parsing altogether.

Last but not least tests also received a bit of love.

StarLightPL added some commits Feb 27, 2016

date parsing improvements
+ introduction of Time::Moment for calculations
+ possibility to set_syslog_timezone / get_syslog_timezone which affects
  the timestamps without explicit timezone designation
+ new mode NormalizeToUTC: makes sure we output ISO 8601 compliant dates
+ New mode IgnoreTimeZones: ignore the timezone altogether - useful when
  you just want the literal dates parsed
+ special mode set by use_utc_syslog - parsing ISO 8601 compliant
  syslogs using only Time::Moment. Blazingly fast at that.
+ new mode DateParsing - allows you to disable date parsing
+ patched date_iso8601 regex to match valid ZULU (UTC-0) dates

Tests:
+ Set mocked time in test suite, should fix the weird leaping dates if
timestamp
  seems to be in the future.
+ added tests for NormalizeToUTC datetime calculation
+ sort keys in testing loops ensures the order of tests is always the
same
  instead of hashkeys-random order

reyjrar added a commit that referenced this pull request Mar 12, 2016

Merge pull request #13 from StarLightPL/master
Experiments with Time::Moment and normalizing logfiles to UTC (#11)

@reyjrar reyjrar merged commit eedb698 into reyjrar:master Mar 12, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment