Asterisk / UDP / Connection Tracking / Priority of default rules

[noppel]

Fedora 30, 64bit
Fail2Ban v0.10.4, installed with dnf, no customizations

There is a general problem with fail2ban. There are iptables rules set by the linux distributions or iptables by default that are interfering with fail2ban. The internet is full of complaints that banning UDP with fail2ban does not work (especially in conjunction with the asterisk jail). I was able to verify these complaints. However I could not find a solution for that although I was spending a lot of time googleing it.

On a fresh fedora 30 installation with firewalld enabled, iptables -S will show me

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N FORWARD_IN_ZONES
-N FORWARD_IN_ZONES_SOURCE
-N FORWARD_OUT_ZONES
-N FORWARD_OUT_ZONES_SOURCE
-N FORWARD_direct
-N FWDI_external
-N FWDI_external_allow
-N FWDI_external_deny
-N FWDI_external_log
-N FWDI_public
-N FWDI_public_allow
-N FWDI_public_deny
-N FWDI_public_log
-N FWDO_external
-N FWDO_external_allow
-N FWDO_external_deny
-N FWDO_external_log
-N FWDO_public
-N FWDO_public_allow
-N FWDO_public_deny
-N FWDO_public_log
-N INPUT_ZONES
-N INPUT_ZONES_SOURCE
-N INPUT_direct
-N IN_external
-N IN_external_allow
-N IN_external_deny
-N IN_external_log
-N IN_public
-N IN_public_allow
-N IN_public_deny
-N IN_public_log
-N OUTPUT_direct
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -m conntrack --ctstate INVALID -j LOG --log-prefix "STATE_INVALID_DROP: "
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -j LOG --log-prefix "FINAL_REJECT: "
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -m conntrack --ctstate INVALID -j LOG --log-prefix "STATE_INVALID_DROP: "
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j LOG --log-prefix "FINAL_REJECT: "
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -j OUTPUT_direct
-A FORWARD_IN_ZONES -i eth0 -g FWDI_external
-A FORWARD_IN_ZONES -g FWDI_public
-A FORWARD_OUT_ZONES -o eth0 -g FWDO_external
-A FORWARD_OUT_ZONES -g FWDO_public
-A FWDI_external -j FWDI_external_log
-A FWDI_external -j FWDI_external_deny
-A FWDI_external -j FWDI_external_allow
-A FWDI_external -p icmp -j ACCEPT
-A FWDI_public -j FWDI_public_log
-A FWDI_public -j FWDI_public_deny
-A FWDI_public -j FWDI_public_allow
-A FWDI_public -p icmp -j ACCEPT
-A FWDO_external -j FWDO_external_log
-A FWDO_external -j FWDO_external_deny
-A FWDO_external -j FWDO_external_allow
-A FWDO_external_allow -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
-A FWDO_external_allow -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
-A FWDO_public -j FWDO_public_log
-A FWDO_public -j FWDO_public_deny
-A FWDO_public -j FWDO_public_allow
-A INPUT_ZONES -i eth0 -g IN_external
-A INPUT_ZONES -g IN_public
-A IN_external -j IN_external_log
-A IN_external -j IN_external_deny
-A IN_external -j IN_external_allow
-A IN_external -p icmp -j ACCEPT
-A IN_external_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
-A IN_public -j IN_public_log
-A IN_public -j IN_public_deny
-A IN_public -j IN_public_allow
-A IN_public -p icmp -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
-A IN_public_allow -d 224.0.0.251/32 -p udp -m udp --dport 5353 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT

So the rule that is breaking the proper functionality of fail2ban is this one:
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
The reason is that all fail2ban rules are added with the same priority of 0. As they are added later, they will in fact be executed after the ACCEPT rule from above. This will result in confusing error messages (XXX is already banned), at least for UDP where connection tracking seems to work differently than for TCP.

I fixed this problem by putting into /etc/rc.d/rc.local:

firewall-cmd --direct --add-rule ipv4 filter INPUT 1000 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -D INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 1000 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -D INPUT_direct -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

However I am not sure if I should add FORWARD rules as well and wheter I should use --permanent. I am also not sure if fail2ban should fix this or wheter it is an issue of the linux distribution. I am not even sure wheter this problem exists with other distributions. I am also unsure why I have to use a mixture of iptables and firewall-cmd to fix this.

The question now is: Should fail2ban dected this/such rules and change priorities so that block rules are executed first?

[sebres]

Do I correct understand the action used here is some of firewallcmd-*?

@fail2ban/contributors, @fail2ban/maintainers Any idea what we can do here?

So the rule that is breaking the proper functionality of fail2ban is this one:
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

Hmm... why this should be an issue? If I correct understand this rule causes accept for ESTABLISHED connections only. So the new connection for banned IPs were not affected by this rule.
Or do you want to kill the packets from (reset) established connection immediately if it got banned?

On a fresh fedora 30 installation with firewalld enabled, iptables -S will show me ...

Connection tracking per default? o..O
_{Hmm... Good that firewalld as well as the distribution like fedora/(RHEL?) are not really my choice...}

[noppel]

On Fedora standard installation, firewalld is installed. However not all rules are visible with firewall-cmd, some need to be modified with iptables. As I prefer firewalld, I have deleted the breaking iptables rule (you cited the correct one in your question) and I have readded it with a higher priority with firewall-cmd. I think you can not set priorities with iptables but you can with firewall-cmd but I might be mistaken. Anyway, these might be details.
While searching the internet, a lot of people have problems that rejecting an ip trying to connect with UDP will not help at all if this breaking rule is present. So if your asterisk is hammered with REGISTER attempts by udp, fail2ban will not be successful in blocking further attempts and prompt a thousand times "IP XXX already banned". Only if the workaround I posted is applied, fail2ban is successful.
You should search the internet for this problem "asterisk / udp / fail2ban / already banned" and you will find a lot of frustrated people who have no solution for this. It took me quite some time figuring this all out as I am not an expert with iptables.

[noppel]

I think fail2ban will correctly block new TCP connections trying to connect to your asterisk server but for UDP, my impression is that the tcp/ip stack can not track the connection state with certainity as udp seems to lack that information. as a workaround, it will look at source ip, maybe delay and port number in order to decide between NEW or ESTABLISHED. and the password hammering will always be considered as ESTABLISHED and therefore is always allowed and therefore that rule will absolutely need a higher priority nubmer so it is executed after the fail2ban blocking rule.

[noppel]

Sorry, I should have posted my jail.local

[DEFAULT]
bantime = 3600
banaction = firewallcmd-ipset
backend=systemd

[sshd]
enabled = true
maxretry = 5

[asterisk]
enabled = true
maxretry = 5
port = 5060,5061,5062

[sebres]

I think you can not set priorities with iptables

You don't need it... because one can use insert instead of append, so able to insert a rule at any position.

but you can with firewall-cmd

If you don't have some other "bothering" rules as priority 0.
(If I correct understand the documentation you cannot insert something ahead of prio 0 rules).

Thus I will say it is broken per design, so I don't know what we can do here.
I'll suggest to increase all the priorities of your "default" rules (prio 0 rules conflicting with fail2ban) to be able to use fail2ban.

If you have other idea (about how it would be resolvable within firewalld-actions) - welcome with PR with that stuff. I don't think that using conntrack as in your workaround is good at all (keyword performance), as well as don't see how it would be applicable as some regular solution for the actions (but to be honest I'm not so familiar with firewalld).

[noppel]

My workaround is removing the offending rules andn reinserting them at a highder priority. I assume the same problem applies to various linux distributions. Maybe someone should open a bug with those distributions and ask them to have the default rules at a higher priority number so that fail2ban can override them. Otherwise I still think fail2ban should be responsibe for inserting its rules at the correct priority even if that means removing already existing rules and reinserting them at a higher priority. How to best achive this - you are more qualified to figure this out than I am.
Please dont underestimate the troubles people have with UDP blocks not working. Sip credentials hacking is as bad as SSH hacking.

[sebres]

Otherwise I still think fail2ban should be responsibe for inserting its rules at the correct priority even if that means removing already existing rules and reinserting them at a higher priority.
How to best achive this - you are more qualified to figure this out than I am.

As already said, I don't see any "correct" possibility to achieve that... as long as the "default" firewald rule set does not really intend to arrange some enhancements ahead default rules (and how it looks it does not at the moment).
To overwrite (or even to remove) some default prio-0 rules is not an option at all, because such manipulations were more dangerous due to many different distributions, packets or even manual rules of user. I will be the first that would block such enhancement if it does not show proper rewrite mechanism working on all possible sets (and I don't see any at the moment).

If you still want to use firewallcmd-* as your banaction (e. g. why not iptables?)... why you don't ask the developers or maintainers of firewald for moving this default rules to prio-1, even in order to allow an insert of blocking rules before implicit allowing rules?
With other words, why you thinking it should be really our issue?

Please dont underestimate the troubles people have with UDP blocks not working.

We don't do that. The question is pretty simple: why you are thinking fail2ban is responsible for a "disaster" basically introduced by some explicit allowing rules, that are placed as prio-0 per default (before any other rules)?...

I can reopen the issue if it is your intention, but don't think it changes something as long as some default allowing rules are conflicting with the blocking rules, that were properly added by fail2ban on place, where it is only possible (per default).

[noppel]

thanks, lets see what happens
firewalld/firewalld#515

[erig0]

Isn't this a duplicate of #1609 ?

[cepheid666]

Isn't this a duplicate of #1609 ?

#1609 refers to TCP traffic where f2b doesn't terminate existing connections. As far as I can tell, the OP is referring to UDP traffic.

That said, #1609 remains a problem. Sendmail is a particular issue -- it does not automatically terminate connections after failures the way, e.g., sshd does. A spammer can sit and try numerous bad passwords, or bad recipients, or whatever, and even though f2b will ban them through sendmail-reject or some other rule (I've got one for saslauthd failures), it won't actually terminate the connection, and f2b will then complain that " is already banned" when it tries to ban them over and over.

I tried to implement the "tcpcut" script given by a contributor to that thread, but it seems to do nothing. I think 1609 might need to be reopened, as well -- while technically it's a 3rd-party issue, the 3rd-party software is behaving "as intended," so I think the termination of the TCP (or UDP, in the case of this thread) will have to be implemented within f2b. (And, really, it would seem like this fits in f2b's charter... there's not much effectiveness in blocking traffic if existing traffic from offenders isn't also terminated.)

[erig0]

Isn't this a duplicate of #1609 ?

#1609 refers to TCP traffic where f2b doesn't terminate existing connections. As far as I can tell, the OP is referring to UDP traffic.

It's the same cause. Firewalld's default ruleset has an early "accept existing connections". This affects both TCP and UDP. Firewalld has two backends; iptables and nftables. In the iptables backend the generic "accept existing connections" occurs before rules addded via --direct are executed and therefore rules added to block traffic won't actually block pre-existing connections. When the nftables backend was added this was fixed - rules added with --direct are always executed before the generic "accept existing connections". See also my comment in the firewalld report, firewalld/firewalld#515 (comment).

[sebres]

Thus closed:

• either switch the backend of firewalld as suggested by @erig0;
• or switch the banaction of fail2ban to something native (iptables/ipset/etc).

[cepheid666]

• either switch the backend of firewalld as suggested by @erig0;
• or switch the banaction of fail2ban to something native (iptables/ipset/etc).

Just to note, I use iptables, not firewalld, and I have the same problem noted in #1609. (I don't know if I have the same problem as in THIS thread since I'm not working with UDP packets.) I'll open a comment there, although that thread is closed...

[sebres]

I use iptables, not firewalld, and I have the same problem noted in #1609.

Basically answered in #1609 (comment)

As already said it is not directly the same problem - if your firewalld (or rather some prerouting rules created by it) are active, you have quasi a white-listing for established connections.
No matter which firewall or net-filter you use, you should be sure that rules created by fail2ban don't conflicting with your own (or even firewall) rules which take place before the fail2ban rules.

Simplest solution would be to eliminate the conflict, despite I don't understand at all which advantages such prerouting using connection tracking (conntrack, Karl!) may provide.
Or to create a chain F2B before any other rules and set it in your fail2ban configs instead of INPUT chain.
Or even adding still one action dropping or killing active established connection of the banned IP (using something like tcpkill, killcx, ss etc).