wiki : Developing Regex in Fail2ban

[sebres]

@egberts

Firstly thank you for the contribution to our wiki, it looks like good described and detailed how-to for Developing Regex in Fail2ban.

But... a brief view of the page shows me several inaccuracies or even mistakes:

datepattern

As regards the datepattern, I did not see any info that part of message matched by datepattern will be cut out before the message is supplied to prefregex/failregex (so both should not include it in its REs).
This could be important.

In every filter file, prefregex defaults to '^(?P<content>.+)$'. If you haven’t touch or set the prefregex, move on to the next section.

This is not true. The default prefregex is empty (resulting to None in python), what means that it would not be applied at all.
This is faster as a catch-all that you assumes (^(?P<content>.+)$) and don't do some other stuff prefregex serve.

You can specify prefregex in order to:

• specify 1 common regex to match some common part present in every messages (do avoid unneeded match in every failregex if you have more as one);
• to cut some interesting part of message only (to simplify failregex) enclosed between tags <F-CONTENT>...</F-CONTENT>;
• to gather some failure identifier (e. g. some prefix matched by <F-MLFID>...<F-MLFID/> tag) to identify several messages belonging to same session, where a connect message containing IP followed by failure message(s) that are not contain IP;
  this provides a new multi-line parsing method as replacement for old (slow an ugly) multi-line parsing using buffering window (maxlines > 1);
• to ignore some wrong, too long or even unneeded messages (a.k.a. parasite log traffic) which can be also present in journal, before failregex search would take place.

failregex = query.+<HOST>

Though I understand this is just an example, but such regex's are not good as examples (especially in wiki about "Developing Regex"), because are unsafe, neither contain start- (^) nor end-anchor ($), as well as contain catch-all like .+, especially which is immediately followed by unprecise <HOST> tag which is accepting every word as hostname.
For instance, it would match first word (or here a last letter '0' because unbound and greedy .+ eats everything else) after query as DNS:

fail2ban-regex -o row "$(date) query injection.attempt.example.com from 192.0.2.1 to 192.0.2.100" 'query.+<HOST>'
['0.0.0.0',     1601478982.0,   {'ip6': None, 'dns': '0', 'ip4': None}],

One could provide better example for regex or at least use <ADDR> tag instead of <HOST>, add some boundary after catch-all, and write a comment that this is just an unsafe example.

The explanation following the RE example above is more confusing as helpful, in my opinion.

prefregex = ^ <F-CONTENT>.+</F-CONTENT>$
...
failregex = ^%(_client)s <HOST>#\d{1,5}%(_view)s: %(_query_refused)s

As already said prefregex is totally unneeded in this resulting filter, because it does not serve any need, so you could remove it here (and simply add this mandatory space directly in failregex).
But it would make sense if you'd for example have multiple failregex.

My few cents to the subject.
Please note that was a brief view (as often under time pressure), so maybe I could miss something...

[egberts]

datepattern

As regards the datepattern, I did not see any info that part of message matched by datepattern will be cut out before the message is supplied to prefregex/failregex (so both should not include it in its REs).
This could be important.

I too was bitten by this until I examined the code. If I had to define it, I would say this:

prefregex is a pattern of the entire log file that is inherently all the same within the same log file. Such common pattern found in line-by-line log file are:

• date (pretty much always)
• daemon name (optional)
• subroutine name and/or line number (optional)
• process ID (optional)
• severity level (optional)

So, the prefregex is highly dependent on proper supporting of this combinatorial of the above list of patterns (some always there, and mostly optional) in order to make it work for everyone that uses the application which generates the logs.

Secondary benefit of prefregex is to ensure that failregex is left with the most dynamic part of the line. prefregex takes that most common part (see above list) of the line.

<--- prefregex   -->|<--   failregex  ->
3-Jan-2020 myscript: Dynamic error message part

[egberts]

Since the named daemon has many error messages, having this prefregex paves the way for easier (and simplier) addition of failregex.

And in our DNS community, we got many error conditions related to distributed denial-of-service attacks to define.

Please feel free to expand the article. I merely wanted to get that word out. Corrections are welcome. Meanwhile I'll try and expand a few with your words by the next day.

[sebres]

Thx!

Some few "objections" from my side:

prefregex is ideally for a pattern that is found (after the datepattern portion) in each and every line of the entire log file. ...

Sure it is, but only if you have multiple failregex or have to implement one of pre-filtering cases mentioned in my previous comment.
Otherwise it is enough to specify own __prefix_line pattern (just as config interpolation variable) and add its substitution to failregex:

__prefix_line = ^(?:...)...
failregex = %(__prefix_line)s ... <HOST>$

Otherwise unnecessary providing of prefregex would just cause a performance regression or (by poor RE of some inexperienced developer) could introduce additional vector for some vulnerability (affected by injection or slow on some artificial messages).

Secondary benefit of prefregex is to ensure that failregex is left with the most dynamic (and interesting) part of the regex line. prefregex takes that most common parts (see above list) of the line.

No. It is not secondary but solely primary role of prefregex. Again see above for which purposes prefregex is currently used.

prefregex = ^ <F-CONTENT>.+</F-CONTENT>$

This is still a part of your resulting example... I don't see that as good and necessary prefregex here, because a mandatory space can be used pretty well in failregex directly (so it can only confuse people reading that example).

Still no time to make a whole review... Nevertheless it is good to have such how-to's and we'll be happy to have even more of that.
Thank you again!

[egberts]

I’m going to have to take some timeout and redo the whole steps just using your words. So bear with me while I incorporate what makes for easier prototyping. Prototyping? Mmmm, a better title?

[egberts]

Curious, as it is currently designed, why didn’t the datepattern already included that ending space?

[sebres]

why didn't the datepattern already included that ending space?

Because common datepatterns can be used in logs where it is included in brackets or have double point after them.
It is implemented in two forms, bounded and not bounded on words.

But it is recommended to specify own pattern, and you could indeed add \s+ at end of your own pattern.

[egberts]

Looks like for Bind/named, I’m going to have to make my own date pattern that includes the ending space. Or put the starting space that into the prefregex with my own prefix_line (currently called _client), which is better?

Also got that 2nd filter ready (more micro DDoS and more ISC refusal to fix).

I don’t like + or *, as I prefer range-constraint curly brace set.

[egberts]

And moving the doc file to fail2ban.wiki for easier and longer-term collaboration.