New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Load logpath only after findtime is configured #2173

Merged
merged 2 commits into from Jul 6, 2018

Conversation

Projects
None yet
4 participants
@mattsta
Copy link
Contributor

mattsta commented Jul 5, 2018

Commit Message

When new log paths are configured, their start offset is immediately determined
by a filter searching for (now - findTime).
But, since findTime is configured after the log is loaded and
searched, logs are only searched back by the default 10 minute findTime,
regardless of user configuration of jail settings.

So, findTime must be configured before logpath or else the default findtime
is used, which ignores any findtime time defined by the user.

This fixes new reads on startup for actual log files. The systemd filter
always performed as expected due to being setup after the jail's
findtime config submission.

Story Time!

It took me probably 10 hours to track down this problem. Spent a long time thinking regexes were just broken for half my jails because lookback-on-load worked reliably on some jails, but not others. Turns out the reliable jails were all systemd and loading from files was kinda broken. Then took a while to figure out how configs actually get loaded.

Additional Fixes Maybe

The newer config options logtimezone and logencoding are also loaded after the logfile is initially processed. If those are used to read the log, they should also be configured before logpath gets sent to the transmitter, otherwise default class values will always be used instead of configured values.

and, one minor change that may have helped cut hours off these debuggles: surfacing the initial lookback results in INFO mode instead of deep debug mode (here and here) would be nice for users to always know (since it's the start point for actually searching for bad actions on startup).

@coveralls

This comment has been minimized.

Copy link

coveralls commented Jul 5, 2018

Coverage Status

Coverage remained the same at 97.404% when pulling 1eb93e2 on mattsta:fix/findtime-backsearch-on-file-load into 857d695 on fail2ban:0.10.

@codecov-io

This comment has been minimized.

Copy link

codecov-io commented Jul 5, 2018

Codecov Report

Merging #2173 into 0.10 will not change coverage.
The diff coverage is 100%.

Impacted file tree graph

@@           Coverage Diff           @@
##             0.10    #2173   +/-   ##
=======================================
  Coverage   95.57%   95.57%           
=======================================
  Files          76       76           
  Lines       13190    13190           
  Branches     2081     2082    +1     
=======================================
  Hits        12607    12607           
  Misses        315      315           
  Partials      268      268
Impacted Files Coverage Δ
fail2ban/client/jailreader.py 93.91% <ø> (ø) ⬆️
fail2ban/server/filter.py 95.75% <100%> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 857d695...1eb93e2. Read the comment docs.

mattsta and others added some commits Jul 5, 2018

Load logpath only after findtime is configured
When new log paths are configured, their start offset is immediately determined
by a filter searching for (now - findTime).
But, since findTime is configured *after* the log is loaded and
searched, logs are only searched back by the default 10 minute findTime,
regardless of user configuration of jail settings.

So, findTime must be configured before logpath or else the default findtime
is used, which ignores any findtime time defined by the user.

This fixes new reads on startup for actual log files. The systemd filter
always performed as expected due to being setup after the jail's
findtime config submission.
filter.py: repair start-time of initial seek to time (regardless the …
…position of `findtime` option in config);

jailreader.py: additionally relocate the option `logpath` after all log-related data (backend, date-pattern, etc) that may be needed by the first usage (gh-2173).
Thanks to Matt Stancliff (mattsta)

@sebres sebres changed the base branch from 0.11 to 0.10 Jul 6, 2018

@sebres sebres force-pushed the mattsta:fix/findtime-backsearch-on-file-load branch from 2f33813 to 1eb93e2 Jul 6, 2018

@sebres

This comment has been minimized.

Copy link
Member

sebres commented Jul 6, 2018

I've rebased this to 0.10 (because affected also) and made a small amendment to cover all possible constellations.
Will merge hereafter if all test-cases passed.
Thank you for your contribution!

@sebres sebres merged commit 1eb93e2 into fail2ban:0.10 Jul 6, 2018

3 checks passed

codecov/patch 100% of diff hit (target 95.57%)
Details
codecov/project 95.57% (+0%) compared to 857d695
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

sebres added a commit that referenced this pull request Jul 6, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment