Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

0.10 filter captures to actions #1698

Merged
merged 23 commits into from
Feb 27, 2017

Conversation

sebres
Copy link
Contributor

@sebres sebres commented Feb 20, 2017

  • Introduced new filter option prefregex for pre-filtering using single regular expression;
    Closes fail2ban failing on lines with many (10.000+) characters #1173
  • Many times faster and fewer CPU-hungry because of parsing with maxlines=1, so without line buffering (scrolling of the buffer-window).
    Combination of tags <F-MLFID> and <F-NOFAIL> can be used now to process multi-line logs using single-line expressions:
    • tag <F-MLFID>: used to identify resp. store failure info for groups of log-lines with the same identifier (e. g. combined failure-info for the same conn-id by <F-MLFID>(?:conn-id)</F-MLFID>, see sshd.conf for example)
    • tag <F-NOFAIL>: used as mark for no-failure (helper to accumulate common failure-info, e. g. from lines that contain IP-address);
  • Several filters optimized with pre-filtering using new option prefregex, and multiline filter using <F-MLFID> + <F-NOFAIL> combination;
  • [filter.d/pam-generic.conf]: grave fix injection on user name to host fixed;
  • Exposes filter group captures in actions (non-recursive interpolation of tags <F-...>);
    Closes Expose filter regex group captures in actions #1110
  • Some filters extended with user name (can be used in Regex to reset counter for IP #1243 to distinguish IP and user, resp. to remove after success login the user-related failures only);
  • Safer, more stable and faster replaceTag interpolation (switched from cycle over all tags to re.sub with callable)
  • substituteRecursiveTags optimization + moved in helpers facilities (because currently used commonly in server and in client)
  • section "Init" no more needed (can be removed later from all action/filter definitions), currently excepting conditional (?family=...), because only expected parameters and section [init] supplied to the server from actionreader.
  • provides new tag <ip-rev> for PTR reversed representation of IP address
  • action.d/complain.conf fixed using new tag <ip-rev>
    Closes WIP: Make Abusix lookup compatible with Dash #1685

Closes gh-1625

@coveralls
Copy link

coveralls commented Feb 20, 2017

Coverage Status

Coverage increased (+0.06%) to 96.434% when pulling fe06ffc on sebres:0.10-filter-captures-to-actions into abd8069 on fail2ban:0.10.

@codecov-io
Copy link

codecov-io commented Feb 20, 2017

Codecov Report

Merging #1698 into 0.10 will increase coverage by 0.2%.
The diff coverage is 94.46%.

@@           Coverage Diff           @@
##            0.10   #1698     +/-   ##
=======================================
+ Coverage     94%   94.2%   +0.2%     
=======================================
  Files         77      77             
  Lines      11404   11546    +142     
  Branches    1778    1754     -24     
=======================================
+ Hits       10720   10877    +157     
+ Misses       412     391     -21     
- Partials     272     278      +6
Impacted Files Coverage Δ
fail2ban/tests/fail2banclienttestcase.py 95.14% <ø> (ø)
config/action.d/smtp.py 76.92% <ø> (ø)
fail2ban/client/configreader.py 89.76% <100%> (+0.6%)
fail2ban/tests/actiontestcase.py 98.26% <100%> (+0.39%)
fail2ban/tests/filtertestcase.py 98.99% <100%> (ø)
fail2ban/client/filterreader.py 100% <100%> (ø)
fail2ban/server/server.py 97.4% <100%> (+0.04%)
fail2ban/client/actionreader.py 84.78% <100%> (+0.33%)
fail2ban/helpers.py 95.51% <100%> (+1.65%)
fail2ban/tests/servertestcase.py 92.67% <100%> (+0.05%)
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update abd8069...e4a265c. Read the comment docs.

…le regular expression;

Some filters extended with user name;
[filter.d/pam-generic.conf]: grave fix injection on user name to host fixed;
test-cases in testSampleRegexsFactory can now check the captured groups (using additionally fields in failJSON structure)
@sebres sebres force-pushed the 0.10-filter-captures-to-actions branch from 949b64a to 4ff8d05 Compare February 20, 2017 15:54
…the convert-stream;

Allow using failure-id (`<HOST>`) within `prefregex` (by common prefix for all expressions specified with `failregex`)
@sebres sebres force-pushed the 0.10-filter-captures-to-actions branch from a543ba4 to 8bcaeb9 Compare February 21, 2017 16:07
…ng of needed failure information to process in further lines.

Many times faster and fewer CPU-hungry because of parsing with `maxlines=1`, so without line buffering (scrolling of the buffer-window).
Combination of tags `<F-MLFID>` and `<F-NOFAIL>` can be used now to process multi-line logs using single-line expressions:
- tag `<F-MLFID>`: used to identify resp. store failure info for groups of log-lines with the same identifier (e. g. combined failure-info for the same conn-id by `<F-MLFID>(?:conn-id)</F-MLFID>`, see sshd.conf for example)
- tag `<F-NOFAIL>`: used as mark for no-failure (helper to accumulate common failure-info);
filter.d/sshd.conf: [sshd], [sshd-ddos], [sshd-aggressive] optimized with pre-filtering using new option `prefregex` and new multi-line handling.
@sebres sebres force-pushed the 0.10-filter-captures-to-actions branch from ec626f9 to 4efcc29 Compare February 22, 2017 21:21
@coveralls
Copy link

coveralls commented Feb 22, 2017

Coverage Status

Coverage decreased (-0.07%) to 96.308% when pulling 4efcc29 on sebres:0.10-filter-captures-to-actions into abd8069 on fail2ban:0.10.

…ing, template possibility (used in new ActionInfo objects);

new ActionInfo handling: saves content between actions, without interim copying (save original on demand, recoverable via reset);
test cases extended
@coveralls
Copy link

Coverage Status

Coverage decreased (-0.05%) to 96.332% when pulling d2a3d09 on sebres:0.10-filter-captures-to-actions into abd8069 on fail2ban:0.10.

@coveralls
Copy link

coveralls commented Feb 24, 2017

Coverage Status

Coverage increased (+0.2%) to 96.605% when pulling e4a265c on sebres:0.10-filter-captures-to-actions into abd8069 on fail2ban:0.10.

@sebres
Copy link
Contributor Author

sebres commented Feb 24, 2017

I've ported many features from my private branches, otherwise doing the further development was always needlessly complicated to me.

If no objections follow, I'll merge it next week.

Small info by the way about this branch:

  • ca. 25% faster in banning/unbanning;
  • ca. 50% faster and fewer CPU-hungry in failure search by multi-line filters (using prefregex resp. combination of tags <F-MLFID> and <F-NOFAIL>) and up to 20% without it (old multiline syntax);
  • no more hits missed in failure-recognizing on busy resp. deeply loaded services by multi-line filters (previously if too many log entries, multiline expression does not find failure, because second line exceeds the maxlines value, so out of scanning buffer)

@sebres sebres merged commit 81129f0 into fail2ban:0.10 Feb 27, 2017
This was referenced Mar 24, 2017
@sebres sebres mentioned this pull request Nov 1, 2018
3 tasks
sebres added a commit to sebres/fail2ban that referenced this pull request Jun 12, 2020
…ix `<F-TUPLE_` (that would combine value of `<F-V>` with all value of <F-TUPLE_V?_n?> tags), for examples see new tests in fail2banregextestcase;

closes fail2bangh-2755 (extends fail2ban#1454 and fail2ban#1698).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants