[RFE]: Log file line pre-processing

[wqweto]

Feature request type

Ability to pre-process input log file before hitting filters classes with a simple find and replace regex.

Description

The idea is for instance this log line

Apr  6 20:40:51 localhost haproxy[11570]: 194.61.55.218:7207 [06/Apr/2021:20:40:49.669] rdp-wqw-4392 rdp-wqw-4392/wqw-pc-3389 1/0/1711 1185 CD 3762/393/393/393/0 0/0

to be able to get converted to this line

Apr  6 20:40:51 localhost haproxy[11570]: 194.61.55.0/24:7207 [06/Apr/2021:20:40:49.669] rdp-wqw-4392 rdp-wqw-4392/wqw-pc-3389 1/0/1711 1185 CD 3762/393/393/393/0 0/0

with a simple (\d+\.\d+\.\d+)\.\d+ -> $1.0/24 regex, so that <HOST> in failregex can match 194.61.55.0/24 instead of the original single IP 194.61.55.218 and so to count failures and later ban the whole subnet.

Considered alternatives

Chaining an external tool to do the pre-processing but I'm not sure how to do this.

Any additional information

If this is already possible it is not obvious to me reading the docs and sources.

I'm aware how to tweak iptables actions to add /24 cidr but in this case fail2ban keeps account of individual IPs in the jail so each IP address x.y.z.1 and x.y.z.2 gets a separate fail count instead of combined one which allows evading detection when scanning/DDoS'ing from compromised subnets.

[sebres]

FWIW, just few comments from my side:

1. subnet banning is basically different issue (Support to ban subnets #927)
2. probably I can provide it very simple (without preprocessors at all) if my branch with regex options gets merged:

{^CIDR(24)}^\s*\S+ haproxy\[\d+\]: <ADDR>:\d+ ...

3. another possibility would be to provide new tags like <F-SUBNET-24/> (which would wrap the <ADDR> to subnet with expected width (CIDR)).
4. there are other issues show certain necessity of regex-based preprocessors (e. g. json converter in MongoDB 4.4 logs are in .json format and not recognized  #2932, etc)

[wqweto]

I just did some preliminary tests by adding a quick and dirty

line = re.sub(r"(\d{1,3}\.\d{1,3}\.\d{1,3})\.\d{1,3}", r"\1.0/24", line)

in processLineAndAdd function in filter.py and the results were quick and deadly for them spammers:

# fail2ban-client status haproxy-rdp
Status for the jail: haproxy-rdp
|- Filter
|  |- Currently failed: 4
|  |- Total failed:     316
|  `- File list:        /var/log/haproxy.log
`- Actions
   |- Currently banned: 12
   |- Total banned:     12
   `- Banned IP list:   185.193.88.0 185.156.74.0 45.155.205.0 31.207.46.0 194.61.55.0 45.146.166.0 185.153.199.0 185.153.197.0 45.134.26.0 94.232.40.0 91.241.19.0 185.202.2.0

# iptables-save
# Generated by iptables-save v1.4.7 on Tue Apr  6 22:54:40 2021
...
-A f2b-default -s 45.146.167.0/24 -j DROP
-A f2b-default -s 185.202.2.0/24 -j DROP
-A f2b-default -s 91.241.19.0/24 -j DROP
-A f2b-default -s 94.232.40.0/24 -j DROP
-A f2b-default -s 45.134.26.0/24 -j DROP
-A f2b-default -s 185.153.197.0/24 -j DROP
-A f2b-default -s 185.153.199.0/24 -j DROP
-A f2b-default -s 45.146.166.0/24 -j DROP
-A f2b-default -s 194.61.55.0/24 -j DROP
-A f2b-default -s 31.207.46.0/24 -j DROP
-A f2b-default -s 45.155.205.0/24 -j DROP
-A f2b-default -s 185.156.74.0/24 -j DROP
-A f2b-default -s 185.193.88.0/24 -j DROP
-A f2b-default -j RETURN

[sebres]

Hmm... No idea where your iptables rules coming from (also chain name is f2b-default and not f2b-haproxy-rdp), but I'm pretty sure it was not correct matched in filter (also status shows that) or your action is also manipulated somehow (it were incorrect), because also if the line will be rewritten correctly, the IPs are just addresses without CIDR notation (and going banned so).
In this case failregex must use newest tags <SUBNET> or <ADDR>(?:/<CIDR>)? (implemented in #2560).
If you would do that fail2ban status will show 192.0.2.0/24 instead of 192.0.2.0 and a subnet is really banned (not the IP).

[wqweto]

JFYI, I tweaked my old fail2ban directly in processLineAndAdd in /usr/lib/python2.6/site-packages/fail2ban/server/filter.py (or /usr/lib/python3/dist-packages/fail2ban/server/filter.py) with this

		if self.__ignoreCommand:
			a, b = self.__ignoreCommand.split('/->/', 1)
			logSys.debug('Pre-process regex: ' + a + ' -> ' + b)
			line = re.sub(a, b, line)
			logSys.debug('    replaced line: ' + line)

so now I can use (hijack) the ignorecommand setting to setup the regex substution in the jail config like this

[haproxy-rdp]
. . .
action = iptables-allports[name=haproxy-rdp, subnet=24]
ignorecommand = (\d{1,3}\.\d{1,3}\.\d{1,3})\.\d{1,3}/->/\1.0/24

and finally had to tweak iptables-allports.conf like this

[Definition]
. . .
actionban = <iptables> -I f2b-<name> 1 -s <ip>/<subnet> -j <blocktype>
actionunban = <iptables> -D f2b-<name> -s <ip>/<subnet> -j <blocktype>

[Init]
subnet = 32

to be able to pass a custom subnet to the action from the jail config above which suits the ignorecommand substitution.

Best would have been to add a separate jail config option instead of hijacking (the obscure) ignorecommand but that is not something that looks easy to do (at least for a non-python dev).

Both of the tweaks above easily implement a simple devops requirement (monitor, count access and ban whole subnets) that is very effective in countering attacks and I stand baffled that these options are not built-in provided that fail2ban is not a new project already.

[sebres]

This is not correct approach, just because you're banning 192.0.2.0 in banmanager (so it can also ban every other IP from this subnet), but 192.0.2.0/32 in banning action... what can potentially cause several issues (starting from multiple banning tickets for the same subnet in banmanager queues and ending with different ID in ticket and action).
Let alone in case of changed action you don't need this tweaking rewrite at all (192.0.2.55/24 is basically the same as 192.0.2.0/24, at least for whole range netmask).

How it would be correct I wrote in comment above (use new tags in failregex, so after rewrite it capturing subnet instead of IP).

[wqweto]

This is not correct approach, just because you're banning 192.0.2.0 in banmanager (so it can also ban every other IP from this subnet), but 192.0.2.0/32 in banning action...

Why do you think so? Notice the jail config action = iptables-allports[name=haproxy-rdp, subnet=24] -- this sets subnet parameter for the action to 24.

The subnet = 32 in [Init] section is the default setting just to keep the current iptables-allports action well behaved for other jails.

Let alone in case of changed action you don't need this tweaking rewrite at all (192.0.2.55/24 is basically the same as 192.0.2.0/24, at least for whole range netmask).

The rewrite is not targeted at the iptables action but exactly at the banmanager so that that the whole 192.0.2.x/24 subnet is reduced to a single 192.0.2.0 address in banmanager so that failiures from all IPs from that subnet count towards a single ban limit.

This on one hand prevents getting mutliple bans for 192.0.2.0/24 in iptables and on second counts failiures more correctly when the DDoS comes from consequtive IPs in posession of the atackers.

Edit: I'm really glad to deliver feedback from the fields to the project developers. My case above is a real-life example of what is actually a badly needed feature from fail2ban for actual devops, namely 1. count whole subnets as one entry in banmanager and 2. ban whole subnets in iptables -- as simple as that :-))

[sebres]

Why do you think so?

Because I know how it works :) Your ticket ID is 192.0.2.0 (what is not a subnet), but in action it is banned as 192.0.2.0/24 (or whatever you supply to the action).
This is indirectly a conflict, that can cause an above mentioned issues.

the whole 192.0.2.x/24 subnet is reduced to a single 192.0.2.0 address in banmanager

Nothing is reduced.... you just used <ADDR> or <HOST> tags in failregex, therefore it is banned as single IP.
And I already wrote you which tags you need to use in case of subnet.

so that failiures from all IPs from that subnet count towards a single ban limit.

Again in banmanager it is not a subnet, but single IP only (with all related issues).
You're not the first who trying to "implement" banning of subnets in fail2ban.

[wqweto]

I'm configuring this on Fail2Ban v0.9.6 so are these "tags" you wrote above that I need to use in case of subnet available on this ancient version? Because my current tweaks seem to work although obviously with dire consequences for IP matching (for whitelisting etc.)

You're not the first who trying to "implement" banning of subnets in fail2ban.

Btw, I'm not "implementing" anything here or this would have been a PR not a RFE. I'm just sharing experience with everyone who happen to be in my position and would have to implement similar tweaks for one reason or another.

Whether devs would consider adding log files preprocessing (about 5-10 LOC total) is up to you to decide but I happen to notice it's currently missing and would come handy for solving those large class of problems -- the so called unknown unknowns, the kind of problems you might have never seen in production yet.

[sebres]

I'm configuring this on Fail2Ban v0.9.6 so are these "tags" you wrote above that I need to use in case of subnet available on this ancient version?

Sure as for for 0.9.
Just because you didn't provide your version before, one guess we're speaking about actual versions :)

[wqweto]

10x

I'll have to research these then.

[wqweto]

@sebres Hi!

2. probably I can provide it very simple (without preprocessors at all) if my branch with regex options gets merged:

{^CIDR(24)}^\s*\S+ haproxy\[\d+\]: <ADDR>:\d+ ...

I was wondering if your branch with regex options got merged.

Can {^CIDR(24)} (or something similar) be used with latest fail2ban to ban subnets?

[sebres]

I was wondering if your branch with regex options got merged.

Sorry, but not yet (#3007) - unfortunately I have persistently no time to fulfill that.
And if it gets merged one would still need a small enhancement (aforementioned {^CIDR(24)}) to specify target subnet/CIDR in RE.

Alternatively I could try to rebase #1161 (at least the part with subnetmask (default CIDR of jail)).