New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FirewallD Ban Actions #1474

Open
Geraden07 opened this Issue Jul 1, 2016 · 33 comments

Comments

Projects
None yet
7 participants
@Geraden07

Geraden07 commented Jul 1, 2016

Apologies for not adhering to the bug report format but I'm not creating this issue for a bug.

This is half a question, half a suggestion.

All the supplied fail2ban action files for use with firewalld use 'direct' rule configuration to essentially ban IPs in the same way that iptables would.

However, from their own man page ( https://fedoraproject.org/wiki/FirewallD ) the authors of FirewallD state that Direct options should be used only as a last resort when it's not possible to use for example --add-service=service or --add-rich-rule='rule'.

Considering this, the more appropriate (and significantly simpler) manner in which to ban an IP address in FirewallD is by adding the IP (referred to as the "source") to an appropriate zone, such as "drop" or "block".

The command to do this would be firewall-cmd --permanent --zone=drop --add-source=<ip>. The benefit of this (apart from the simplicity) would be that firewalld would then also be able to display the banned IPs associated with the drop (or block) zone by using the command firewall-cmd --info-zone=drop or even firewall-cmd --list-all-zones.

No setup or teardown actions would be necessary. Essentially this approach delegates the nitty-gritty details of rule writing to firewalld. Instead of telling firewalld HOW you want to ban an IP, you instead tell firewalld THAT you want to ban an IP and let firewalld deal with the HOW.

If the 'block' zone were to be used, the response to banned IPs would be an icmp-host-prohibited message for IPv4 and icmp6-adm-prohibited for IPv6. If the 'drop' zone were used, the packets would be dropped with no response.

I personally am in favor of using the drop behavior, but it's easy enough that both action's could be and should be supplied as options.

Am I missing something as to why this approach is not preferable? Is there a reason that --direct rules are necessary and my suggestion is unsuitable? Or is the current behavior simply a result of how iptables rules were ported over to firewalld for compatibility at first release? In which case I strongly suggest this at the very least as an option that should be included with the fail2ban package-provided action files.

@sebres

This comment has been minimized.

Show comment
Hide comment
@sebres

sebres Aug 9, 2016

Member

Thx.
We will continue reorganizing of firewallcmd-actions (#1447, etc.) for our new 0.10 version, so we can see, how this can be optional implemented there.

Member

sebres commented Aug 9, 2016

Thx.
We will continue reorganizing of firewallcmd-actions (#1447, etc.) for our new 0.10 version, so we can see, how this can be optional implemented there.

@Geraden07

This comment has been minimized.

Show comment
Hide comment
@Geraden07

Geraden07 Jan 14, 2017

So I had some time to play around with this. I believe this works as desired:

EDIT: I have updated this script with improvements discussed further down in this discussion thread.

cat << EOF > /etc/fail2ban/action.d/custom-firewalld.conf
[INCLUDES]
before  =

[Definition]
actionstart =
actionstop =
actioncheck =

actionflush = sed -i '/<source address=/d' /etc/firewalld/zones/drop.xml
actionban = firewall-cmd --change-source=<ip> --zone=drop && firewall-cmd --change-source=<ip> --zone=drop --permanent
actionunban = firewall-cmd --remove-source=<ip> --zone=drop && firewall-cmd --remove-source=<ip> --zone=drop --permanent || echo 0

[Init]

EOF

This "custom-firewalld" action created as shown above will simply add the IP to the drop zone. Testing on my machine seems to show this functioning as desired.

The command to look at the zone through firewalld is
firewall-cmd --list-all --zone=drop
or equivalently
firewall-cmd --info-zone drop

No start or stop actions required. No includes required. etc.

EDIT: this script has been updated as the discussion below has progressed.

Geraden07 commented Jan 14, 2017

So I had some time to play around with this. I believe this works as desired:

EDIT: I have updated this script with improvements discussed further down in this discussion thread.

cat << EOF > /etc/fail2ban/action.d/custom-firewalld.conf
[INCLUDES]
before  =

[Definition]
actionstart =
actionstop =
actioncheck =

actionflush = sed -i '/<source address=/d' /etc/firewalld/zones/drop.xml
actionban = firewall-cmd --change-source=<ip> --zone=drop && firewall-cmd --change-source=<ip> --zone=drop --permanent
actionunban = firewall-cmd --remove-source=<ip> --zone=drop && firewall-cmd --remove-source=<ip> --zone=drop --permanent || echo 0

[Init]

EOF

This "custom-firewalld" action created as shown above will simply add the IP to the drop zone. Testing on my machine seems to show this functioning as desired.

The command to look at the zone through firewalld is
firewall-cmd --list-all --zone=drop
or equivalently
firewall-cmd --info-zone drop

No start or stop actions required. No includes required. etc.

EDIT: this script has been updated as the discussion below has progressed.

@sebres

This comment has been minimized.

Show comment
Hide comment
@sebres

sebres Mar 9, 2017

Member

firewall-cmd --add-source=<ip> --zone=drop && firewall-cmd --add-source=<ip> --zone=drop --permanent

Why you need it twice (once without and once with --permanent)? I mean the second part with --permanent should be enough?

BTW. We can extend action with optional parameters like ban_service = --add-service=<service> and ban_port = --add-port=<port>/<protocol> for targeting the ban for given port resp. service ports.

Member

sebres commented Mar 9, 2017

firewall-cmd --add-source=<ip> --zone=drop && firewall-cmd --add-source=<ip> --zone=drop --permanent

Why you need it twice (once without and once with --permanent)? I mean the second part with --permanent should be enough?

BTW. We can extend action with optional parameters like ban_service = --add-service=<service> and ban_port = --add-port=<port>/<protocol> for targeting the ban for given port resp. service ports.

@Geraden07

This comment has been minimized.

Show comment
Hide comment
@Geraden07

Geraden07 Mar 10, 2017

With --permanent, the effect will not be immediate, and would only occur after the next time the firewall is reloaded.
Without --permanent ensures it is applied to the in-memory configuration of the firewall but this would not persist through a reload.

By using both we ensure it applies immediately and will persist through firewall reloads.

As for using add-port and add-service options, they aren't what we want here. Packets are matched to a zone first by either interface or by source IP. Once they belong to a zone, then that zone's rules apply as to whether the packets should be allowed through or not. Zone's have a "target" that is the default action to take if the packet DOESN'T match any rules.

The target of the Drop zone is DROP, and the target of the Block zone is %%REJECT%%. The DROP target will drop the packet with no response. The %%REJECT%% target will respond with an icmp-host-prohibited message for IPv4 and icmp6-adm-prohibited for IPv6.

Services are really just aliases for popular port/protocol combinations. Adding either a service or a port/protocol would mean the packets would MATCH the rules in the zone, and thus they will NOT be blocked or dropped.

See http://www.firewalld.org/documentation/zone/options.html for the documentation about zone configuration.

Geraden07 commented Mar 10, 2017

With --permanent, the effect will not be immediate, and would only occur after the next time the firewall is reloaded.
Without --permanent ensures it is applied to the in-memory configuration of the firewall but this would not persist through a reload.

By using both we ensure it applies immediately and will persist through firewall reloads.

As for using add-port and add-service options, they aren't what we want here. Packets are matched to a zone first by either interface or by source IP. Once they belong to a zone, then that zone's rules apply as to whether the packets should be allowed through or not. Zone's have a "target" that is the default action to take if the packet DOESN'T match any rules.

The target of the Drop zone is DROP, and the target of the Block zone is %%REJECT%%. The DROP target will drop the packet with no response. The %%REJECT%% target will respond with an icmp-host-prohibited message for IPv4 and icmp6-adm-prohibited for IPv6.

Services are really just aliases for popular port/protocol combinations. Adding either a service or a port/protocol would mean the packets would MATCH the rules in the zone, and thus they will NOT be blocked or dropped.

See http://www.firewalld.org/documentation/zone/options.html for the documentation about zone configuration.

@sebres

This comment has been minimized.

Show comment
Hide comment
@sebres

sebres Mar 10, 2017

Member

Very strange behavior that the --permanent changes are not effective immediately (and no option to do it).
Then I don't see any benefit to use --permanent: so holding the ban "ticket" in the fail2ban and in firewalld config.
For the possible restart of firewalld you'll need anyway a dependency to fail2ban (because without it you can get a undefined behavior as for leaks in the rules). But if you've such dependency, no matter using --permanent or not, because fail2ban will unban it by stop.
The reload of firewalld (opposite to restart) does not restore --permanent entries (do it anyway for the runtime configuration).

So IMHO the usage of --permanent in sense of fail2ban action is totally unnecessary.

As for options add-port etc: You misunderstood me - I meant the common firewalld-configuration file (as single file) where the options corresponding zone=drop, as well as add-port and add-service may be combined as optional, that all controlling the result of actions.
Just an idea to have exact one config file instead of multiple (take a look at our iptables-* - we've a lot of files doing almost the same with minimal differences).

Member

sebres commented Mar 10, 2017

Very strange behavior that the --permanent changes are not effective immediately (and no option to do it).
Then I don't see any benefit to use --permanent: so holding the ban "ticket" in the fail2ban and in firewalld config.
For the possible restart of firewalld you'll need anyway a dependency to fail2ban (because without it you can get a undefined behavior as for leaks in the rules). But if you've such dependency, no matter using --permanent or not, because fail2ban will unban it by stop.
The reload of firewalld (opposite to restart) does not restore --permanent entries (do it anyway for the runtime configuration).

So IMHO the usage of --permanent in sense of fail2ban action is totally unnecessary.

As for options add-port etc: You misunderstood me - I meant the common firewalld-configuration file (as single file) where the options corresponding zone=drop, as well as add-port and add-service may be combined as optional, that all controlling the result of actions.
Just an idea to have exact one config file instead of multiple (take a look at our iptables-* - we've a lot of files doing almost the same with minimal differences).

@Geraden07

This comment has been minimized.

Show comment
Hide comment
@Geraden07

Geraden07 Mar 11, 2017

I don't see it as being strange behaviour, in regard to the --permanent option. The option to have it take effect immediately IS simply not including the --permanent option. This makes a clear distinction between in-memory and disk-persisted configurations. This allows for changes to be made to the in-memory config only as temporary changes, then once they are no longer needed, you can simply reload.

I'm not sure that I follow what you mean about the dependency you mention and rule leakage etc. You say "the reload of firewalld (opposite to restart) does not restore --permanent entries". This is false.

From a firewalld perspective a restart (that is rebooting of the system) and a reload (that is loading firewall config from disk) are essentially the same thing. The --permanent option will mean that a rule will persist through either of these events. However, the --permanent option will NOT cause a rule to apply immediately, since this doesn't affect the current in-memory configuration.

Like I said, if you use both --permanent and the same rule without --permanent as I have written it above, then what you get is a rule that takes effect immediately in memory, and will persist through firewall reloads (and system restarts etc.).

I still don't follow what you mean about the add-port options etc. The options add-port (or add-service) are generally used to allow a specific port and protocol through a zone that defaults to blocking them in some way. Now it would be possible to add a source IP to a zone that defaults to allowing packets through, then use the add-port or add-service options to block traffic from the given IP on a specific port and protocol, but this has the side effect of causing all OTHER traffic from that IP (which might ordinarily be blocked) to now be accepted unless it matches the specific add-port/add-service rules you've added to the zone. Causing traffic to default to ACCEPT is not a good idea. Especially when we are talking about IPs that are suspected of malicious behaviour which is what fail2ban is all about.

Geraden07 commented Mar 11, 2017

I don't see it as being strange behaviour, in regard to the --permanent option. The option to have it take effect immediately IS simply not including the --permanent option. This makes a clear distinction between in-memory and disk-persisted configurations. This allows for changes to be made to the in-memory config only as temporary changes, then once they are no longer needed, you can simply reload.

I'm not sure that I follow what you mean about the dependency you mention and rule leakage etc. You say "the reload of firewalld (opposite to restart) does not restore --permanent entries". This is false.

From a firewalld perspective a restart (that is rebooting of the system) and a reload (that is loading firewall config from disk) are essentially the same thing. The --permanent option will mean that a rule will persist through either of these events. However, the --permanent option will NOT cause a rule to apply immediately, since this doesn't affect the current in-memory configuration.

Like I said, if you use both --permanent and the same rule without --permanent as I have written it above, then what you get is a rule that takes effect immediately in memory, and will persist through firewall reloads (and system restarts etc.).

I still don't follow what you mean about the add-port options etc. The options add-port (or add-service) are generally used to allow a specific port and protocol through a zone that defaults to blocking them in some way. Now it would be possible to add a source IP to a zone that defaults to allowing packets through, then use the add-port or add-service options to block traffic from the given IP on a specific port and protocol, but this has the side effect of causing all OTHER traffic from that IP (which might ordinarily be blocked) to now be accepted unless it matches the specific add-port/add-service rules you've added to the zone. Causing traffic to default to ACCEPT is not a good idea. Especially when we are talking about IPs that are suspected of malicious behaviour which is what fail2ban is all about.

@getgray

This comment has been minimized.

Show comment
Hide comment
@getgray

getgray Jul 6, 2017

@Geraden07 , your method seems to be a better fit for a system that defaults to firewalld. The recommendations to users are to just use the firewalld commands to manage the firewall. As someone who does not have a lot of experience with iptables, I prefer the firewalld syntax I've gotten used to.

Add to that, in other threads on this subject I saw where people were having issues with the direct iptables commands not working correctly for things inserted by fail2ban. That may have been fixed, I'm not sure about that.

Circling back to your original questions I have a couple:

  1. did this continue to work for you properly?
  2. is there any reason to not do it this way? Particularly if I'm OK with banning all ports for the offending IP

(I understand the need for --permanent and without. Not a big deal I think, simply do both to cover the 2 cases).

Finally, as a new fail2ban user I'm fuzzy about these defaults:
banaction =
banaction_allports =
Do I need an action for both of these? I was concerned about not setting them both, maybe it's a non-issue if I set the action explicitly in each of my jails? I can't tell when one or the other is used by default.

Do they both need to be set to custom-firewalld.conf if I want to use this method? (I get that this custom-firewalld action, as written above is for all ports)

If a single port action was required, one could use this syntax I think:
firewall-cmd --permanent --zone=drop --add-rich-rule=' rule family="ipv4" source address="1.2.3.4/32" port protocol="tcp" port="1234" drop'
(change default ports to
port = 0-65535)
Maybe this syntax is a problem, since firewalld needs a dash in a port range? I assume that's not easily changed? AFAIK firewalld takes ports as a range port-port to a single port. No list of individual ports.

In any case, I would sure like to stick to firewalld syntax if possible.

getgray commented Jul 6, 2017

@Geraden07 , your method seems to be a better fit for a system that defaults to firewalld. The recommendations to users are to just use the firewalld commands to manage the firewall. As someone who does not have a lot of experience with iptables, I prefer the firewalld syntax I've gotten used to.

Add to that, in other threads on this subject I saw where people were having issues with the direct iptables commands not working correctly for things inserted by fail2ban. That may have been fixed, I'm not sure about that.

Circling back to your original questions I have a couple:

  1. did this continue to work for you properly?
  2. is there any reason to not do it this way? Particularly if I'm OK with banning all ports for the offending IP

(I understand the need for --permanent and without. Not a big deal I think, simply do both to cover the 2 cases).

Finally, as a new fail2ban user I'm fuzzy about these defaults:
banaction =
banaction_allports =
Do I need an action for both of these? I was concerned about not setting them both, maybe it's a non-issue if I set the action explicitly in each of my jails? I can't tell when one or the other is used by default.

Do they both need to be set to custom-firewalld.conf if I want to use this method? (I get that this custom-firewalld action, as written above is for all ports)

If a single port action was required, one could use this syntax I think:
firewall-cmd --permanent --zone=drop --add-rich-rule=' rule family="ipv4" source address="1.2.3.4/32" port protocol="tcp" port="1234" drop'
(change default ports to
port = 0-65535)
Maybe this syntax is a problem, since firewalld needs a dash in a port range? I assume that's not easily changed? AFAIK firewalld takes ports as a range port-port to a single port. No list of individual ports.

In any case, I would sure like to stick to firewalld syntax if possible.

@Geraden07

This comment has been minimized.

Show comment
Hide comment
@Geraden07

Geraden07 Jul 6, 2017

@getgray Heya, yes this has been and continues to be the way I run it on my home server. To my knowledge there is no reason to avoid this approach, and as you point out, lots of reasons to take this approach.

As for understanding the defaults and everything: in my sshd jail file I have banaction defined. I don't even have a line for banaction_allports so I don't think it's needed.

Basically my setup is as follows:

  1. I made the action file as described several comments above. I named it custom-firewalld.conf
  2. I changed my jail.d files on the banaction line to point to "custom-firewalld" (no .conf needed)
  3. I also added in several new patterns to the filter.d/sshd.conf file because I was noticing my distro/package versions were resulting in log messages that were not matching what came in the filter by default. I recommend you check your filters against your log messages periodically because message formats can change as versions are updated.

As for supporting port specific bans, I agree the way to do so would be via the rich rule approach. However, I am still against port specific bans in general. If an IP is attempting to brute force me, I'm not going to ban him from sending me SSH packets just to have him move on to the next available service I'm running.

If the IP is legitimate and simply tried the wrong password too many times, there's no benefit to still being able to hit other services after being banned from the SSH login. Once you've accidentally been banned, either wait for the timeout or get someone with access from another IP to unban your IP.

Geraden07 commented Jul 6, 2017

@getgray Heya, yes this has been and continues to be the way I run it on my home server. To my knowledge there is no reason to avoid this approach, and as you point out, lots of reasons to take this approach.

As for understanding the defaults and everything: in my sshd jail file I have banaction defined. I don't even have a line for banaction_allports so I don't think it's needed.

Basically my setup is as follows:

  1. I made the action file as described several comments above. I named it custom-firewalld.conf
  2. I changed my jail.d files on the banaction line to point to "custom-firewalld" (no .conf needed)
  3. I also added in several new patterns to the filter.d/sshd.conf file because I was noticing my distro/package versions were resulting in log messages that were not matching what came in the filter by default. I recommend you check your filters against your log messages periodically because message formats can change as versions are updated.

As for supporting port specific bans, I agree the way to do so would be via the rich rule approach. However, I am still against port specific bans in general. If an IP is attempting to brute force me, I'm not going to ban him from sending me SSH packets just to have him move on to the next available service I'm running.

If the IP is legitimate and simply tried the wrong password too many times, there's no benefit to still being able to hit other services after being banned from the SSH login. Once you've accidentally been banned, either wait for the timeout or get someone with access from another IP to unban your IP.

@getgray

This comment has been minimized.

Show comment
Hide comment
@getgray

getgray Jul 7, 2017

Agree. Thanks for confirming it is working. I'm on Centos7 and for now, I went on and used the default per port firewallcmd-ipset rule, it is working. But I'm apprehensive about having to work on any rules, chains or cleanup if fail2ban got itself hosed somehow. Not being proficient with iptables. In contrast the rich rules are easy for me to add/remove. Also, with the plain firewall-cmd rich rules, I see no way to create a clean actionstart or actionstop section for when fail2ban is started/stopped.

FWIW, I figured out the allports seems to only be used with the recidive rule, of the rules I use. Allport banning makes the most sense for me as well.

Thanks for the reply.

getgray commented Jul 7, 2017

Agree. Thanks for confirming it is working. I'm on Centos7 and for now, I went on and used the default per port firewallcmd-ipset rule, it is working. But I'm apprehensive about having to work on any rules, chains or cleanup if fail2ban got itself hosed somehow. Not being proficient with iptables. In contrast the rich rules are easy for me to add/remove. Also, with the plain firewall-cmd rich rules, I see no way to create a clean actionstart or actionstop section for when fail2ban is started/stopped.

FWIW, I figured out the allports seems to only be used with the recidive rule, of the rules I use. Allport banning makes the most sense for me as well.

Thanks for the reply.

@Geraden07

This comment has been minimized.

Show comment
Hide comment
@Geraden07

Geraden07 Jul 7, 2017

No problem. Yes I should probably have noted my distro is Fedora 25, so should be very similar to CentOS 7.

As for the actionstart and actionstop... They aren't needed when using Firewalld. They are a byproduct of the iptables paradigm where you need to create separate tables and link them into the relevant chains before you can start adding (or rather banning) IPs.

Since my approach with firewalld (not even using the rich rules, just the zones) requires no setup or preparation before you can start with ban actions, then those actionstart and actionstop items become irrelevant. I believe it is the same if you are going to use the rich rules approach to allow for specific port blocking.

Geraden07 commented Jul 7, 2017

No problem. Yes I should probably have noted my distro is Fedora 25, so should be very similar to CentOS 7.

As for the actionstart and actionstop... They aren't needed when using Firewalld. They are a byproduct of the iptables paradigm where you need to create separate tables and link them into the relevant chains before you can start adding (or rather banning) IPs.

Since my approach with firewalld (not even using the rich rules, just the zones) requires no setup or preparation before you can start with ban actions, then those actionstart and actionstop items become irrelevant. I believe it is the same if you are going to use the rich rules approach to allow for specific port blocking.

@RobbieTheK

This comment has been minimized.

Show comment
Hide comment
@RobbieTheK

RobbieTheK Jul 26, 2017

I'm using the custom firewalld option and occasionally seeing these. Is this related to #973, as in "invalid utf-8"? Or perhaps it's as simple as this suggestion, #973 (comment), to stop fail2ban and firewalld?

2017-07-25 20:32:16,308 fail2ban.actions        [7407]: NOTICE  [pam-generic] Unban 60.190.98.50
2017-07-25 20:32:17,121 fail2ban.action         [7407]: ERROR   firewall-cmd --remove-source=60.190.98.50 --zone=drop && firewall-cmd --remove-source=60.190.98.50 --zone=drop --permanent -- stdout: b''
2017-07-25 20:32:17,122 fail2ban.action         [7407]: ERROR   firewall-cmd --remove-source=60.190.98.50 --zone=drop && firewall-cmd --remove-source=60.190.98.50 --zone=drop --permanent -- stderr: b"\x1b[91mError: UNKNOWN_SOURCE: '60.190.98.50' is not in any zone\x1b[00m\n"
2017-07-25 20:32:17,123 fail2ban.action         [7407]: ERROR   firewall-cmd --remove-source=60.190.98.50 --zone=drop && firewall-cmd --remove-source=60.190.98.50 --zone=drop --permanent -- returned 30
2017-07-25 20:32:17,124 fail2ban.actions        [7407]: ERROR   Failed to execute unban jail 'pam-generic' action 'custom-firewalld' info '{'ip': '60.190.98.50', 'failures': 3, 'matches': 'Jul 25 19:32:09 storm sshd[23434]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=60.190.98.50 Jul 25 19:32:12 storm sshd[23461]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=60.190.98.50 Jul 25 19:32:14 storm sshd[23409]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=60.190.98.50 ', 'time': 1501025535.614875}': Error unbanning 60.190.98.50

RobbieTheK commented Jul 26, 2017

I'm using the custom firewalld option and occasionally seeing these. Is this related to #973, as in "invalid utf-8"? Or perhaps it's as simple as this suggestion, #973 (comment), to stop fail2ban and firewalld?

2017-07-25 20:32:16,308 fail2ban.actions        [7407]: NOTICE  [pam-generic] Unban 60.190.98.50
2017-07-25 20:32:17,121 fail2ban.action         [7407]: ERROR   firewall-cmd --remove-source=60.190.98.50 --zone=drop && firewall-cmd --remove-source=60.190.98.50 --zone=drop --permanent -- stdout: b''
2017-07-25 20:32:17,122 fail2ban.action         [7407]: ERROR   firewall-cmd --remove-source=60.190.98.50 --zone=drop && firewall-cmd --remove-source=60.190.98.50 --zone=drop --permanent -- stderr: b"\x1b[91mError: UNKNOWN_SOURCE: '60.190.98.50' is not in any zone\x1b[00m\n"
2017-07-25 20:32:17,123 fail2ban.action         [7407]: ERROR   firewall-cmd --remove-source=60.190.98.50 --zone=drop && firewall-cmd --remove-source=60.190.98.50 --zone=drop --permanent -- returned 30
2017-07-25 20:32:17,124 fail2ban.actions        [7407]: ERROR   Failed to execute unban jail 'pam-generic' action 'custom-firewalld' info '{'ip': '60.190.98.50', 'failures': 3, 'matches': 'Jul 25 19:32:09 storm sshd[23434]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=60.190.98.50 Jul 25 19:32:12 storm sshd[23461]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=60.190.98.50 Jul 25 19:32:14 storm sshd[23409]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=60.190.98.50 ', 'time': 1501025535.614875}': Error unbanning 60.190.98.50
@Geraden07

This comment has been minimized.

Show comment
Hide comment
@Geraden07

Geraden07 Jul 26, 2017

@RobbieTheK hmm, that message seems to indicate that your unban action failed because the IP it was trying to unban wasn't already banned.

This could indicate that IPs are not correctly being banned in the first place but fail2ban thinks they are, so when it goes to unban them after configured amount of time, it can't.

Another possible reason is that maybe something else is unbanning or clearing out your firewalld zone configurations or something? If something gets unbanned without fail2ban knowing, then it will still try to unban.

You can rectify those errors if they repeat by manually forcing that IP to be banned, then waiting for fail2ban to unban it on it's next retry (if it retries again and again after that error message, I'm not sure).

Geraden07 commented Jul 26, 2017

@RobbieTheK hmm, that message seems to indicate that your unban action failed because the IP it was trying to unban wasn't already banned.

This could indicate that IPs are not correctly being banned in the first place but fail2ban thinks they are, so when it goes to unban them after configured amount of time, it can't.

Another possible reason is that maybe something else is unbanning or clearing out your firewalld zone configurations or something? If something gets unbanned without fail2ban knowing, then it will still try to unban.

You can rectify those errors if they repeat by manually forcing that IP to be banned, then waiting for fail2ban to unban it on it's next retry (if it retries again and again after that error message, I'm not sure).

@RobbieTheK

This comment has been minimized.

Show comment
Hide comment
@RobbieTheK

RobbieTheK Jul 31, 2017

This happened again. Could it be that 2 jails are catching this and the findtime and/or bantime are overlapping? Here's another example:

2017-07-31 09:11:04,308 fail2ban.filter         [7407]: INFO    [sshd] Found 103.79.143.133
2017-07-31 09:11:04,544 fail2ban.filter         [7407]: INFO    [sshd] Found 103.79.143.133
2017-07-31 09:11:05,323 fail2ban.actions        [7407]: NOTICE  [sshd] 103.79.143.133 already banned
2017-07-31 10:11:00,044 fail2ban.actions        [7407]: NOTICE  [sshd] Unban 103.79.143.133
2017-07-31 10:11:02,528 fail2ban.actions        [7407]: NOTICE  [pam-generic] Unban 103.79.143.133
2017-07-31 10:11:03,342 fail2ban.action         [7407]: ERROR   firewall-cmd --remove-source=103.79.143.133 --zone=drop && firewall-cmd --remove-source=103.79.143.133 --zone=drop --permanent -- stdout: b''
2017-07-31 10:11:03,342 fail2ban.action         [7407]: ERROR   firewall-cmd --remove-source=103.79.143.133 --zone=drop && firewall-cmd --remove-source=103.79.143.133 --zone=drop --permanent -- stderr: b"\x1b[91mError: UNKNOWN_SOURCE: '103.79.143.133' is not in any zone\x1b[00m\n"
2017-07-31 10:11:03,342 fail2ban.action         [7407]: ERROR   firewall-cmd --remove-source=103.79.143.133 --zone=drop && firewall-cmd --remove-source=103.79.143.133 --zone=drop --permanent -- returned 30
2017-07-31 10:11:03,343 fail2ban.actions        [7407]: ERROR   Failed to execute unban jail 'pam-generic' action 'custom-firewalld' info '{'ip': '103.79.143.133', 'failures': 3, 'matches': 'Jul 31 09:10:56 storm sshd[19385]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.79.143.133 Jul 31 09:10:59 storm sshd[19408]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.79.143.133 Jul 31 09:11:01 storm sshd[19364]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.79.143.133 ', 'time': 1501506661.884758}': Error unbanning 103.79.143.133

RobbieTheK commented Jul 31, 2017

This happened again. Could it be that 2 jails are catching this and the findtime and/or bantime are overlapping? Here's another example:

2017-07-31 09:11:04,308 fail2ban.filter         [7407]: INFO    [sshd] Found 103.79.143.133
2017-07-31 09:11:04,544 fail2ban.filter         [7407]: INFO    [sshd] Found 103.79.143.133
2017-07-31 09:11:05,323 fail2ban.actions        [7407]: NOTICE  [sshd] 103.79.143.133 already banned
2017-07-31 10:11:00,044 fail2ban.actions        [7407]: NOTICE  [sshd] Unban 103.79.143.133
2017-07-31 10:11:02,528 fail2ban.actions        [7407]: NOTICE  [pam-generic] Unban 103.79.143.133
2017-07-31 10:11:03,342 fail2ban.action         [7407]: ERROR   firewall-cmd --remove-source=103.79.143.133 --zone=drop && firewall-cmd --remove-source=103.79.143.133 --zone=drop --permanent -- stdout: b''
2017-07-31 10:11:03,342 fail2ban.action         [7407]: ERROR   firewall-cmd --remove-source=103.79.143.133 --zone=drop && firewall-cmd --remove-source=103.79.143.133 --zone=drop --permanent -- stderr: b"\x1b[91mError: UNKNOWN_SOURCE: '103.79.143.133' is not in any zone\x1b[00m\n"
2017-07-31 10:11:03,342 fail2ban.action         [7407]: ERROR   firewall-cmd --remove-source=103.79.143.133 --zone=drop && firewall-cmd --remove-source=103.79.143.133 --zone=drop --permanent -- returned 30
2017-07-31 10:11:03,343 fail2ban.actions        [7407]: ERROR   Failed to execute unban jail 'pam-generic' action 'custom-firewalld' info '{'ip': '103.79.143.133', 'failures': 3, 'matches': 'Jul 31 09:10:56 storm sshd[19385]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.79.143.133 Jul 31 09:10:59 storm sshd[19408]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.79.143.133 Jul 31 09:11:01 storm sshd[19364]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.79.143.133 ', 'time': 1501506661.884758}': Error unbanning 103.79.143.133
@getgray

This comment has been minimized.

Show comment
Hide comment
@getgray

getgray Jul 31, 2017

@RobbieTheK
Interesting theory, sounds right. I haven't set mine up like this (yet). But, if we:
Assume same jail time

  • jail1 (postfix for example) gets some hits and bans
  • jail2 (httpd for example) gets some hits later in the day and bans. It will already be banned, That, too will generate an error (Warning: ZONE_ALREADY_SET: 'n.n.n.n' already bound to 'public')
  • jail1 expires, unbans, OK
  • jail2 expires, will attempt unban. will get an error (Error: UNKNOWN_SOURCE: 'n.n.n.n' is not in any zone)

Can't think of a workaround. Maybe a script that checks for the ban or not.

:(

getgray commented Jul 31, 2017

@RobbieTheK
Interesting theory, sounds right. I haven't set mine up like this (yet). But, if we:
Assume same jail time

  • jail1 (postfix for example) gets some hits and bans
  • jail2 (httpd for example) gets some hits later in the day and bans. It will already be banned, That, too will generate an error (Warning: ZONE_ALREADY_SET: 'n.n.n.n' already bound to 'public')
  • jail1 expires, unbans, OK
  • jail2 expires, will attempt unban. will get an error (Error: UNKNOWN_SOURCE: 'n.n.n.n' is not in any zone)

Can't think of a workaround. Maybe a script that checks for the ban or not.

:(

@getgray

This comment has been minimized.

Show comment
Hide comment
@getgray

getgray Jul 31, 2017

How about this workaround? "--change-source" behaves like an "add source" if the source does not already exist. Tired it, seems to work w/o generating errors.

add:

actionban = firewall-cmd --change-source=<ip> --zone=drop && firewall-cmd –permanent --change-source=<ip> --zone=drop

Remove (via add then remove)

actionunban = firewall-cmd --change-source=<ip> --zone=drop && firewall-cmd –permanent --change-source=<ip> --zone=drop && firewall-cmd --remove-source=<ip> --zone=drop && firewall-cmd –permanent --remove-source=<ip> --zone=drop

getgray commented Jul 31, 2017

How about this workaround? "--change-source" behaves like an "add source" if the source does not already exist. Tired it, seems to work w/o generating errors.

add:

actionban = firewall-cmd --change-source=<ip> --zone=drop && firewall-cmd –permanent --change-source=<ip> --zone=drop

Remove (via add then remove)

actionunban = firewall-cmd --change-source=<ip> --zone=drop && firewall-cmd –permanent --change-source=<ip> --zone=drop && firewall-cmd --remove-source=<ip> --zone=drop && firewall-cmd –permanent --remove-source=<ip> --zone=drop

@getgray

This comment has been minimized.

Show comment
Hide comment
@getgray

getgray Aug 1, 2017

@Geraden07 re the start and stop actions again:

On Centos7 they built starting and stopping firewalld into the systemctl thing. So if you

systemctl stop firewalld

for example, it stops firewalld. When it stops it removes all the bans on the way out, and when you start it puts them all back. Frankly, not sure I like that, either. Seems like it would leave a window open as it logrotates and restarts both.

I need to investigate and see if I can do that with this setup... Because I still agree with you, blocking at the IP level for all parts for those malicious prods is the thing to do. In my case anyway. I could imagine some multi-tasking server were you woudl not want to lock a user out of oracle if they forgot their ssh password. But that's not my use scenario.

getgray commented Aug 1, 2017

@Geraden07 re the start and stop actions again:

On Centos7 they built starting and stopping firewalld into the systemctl thing. So if you

systemctl stop firewalld

for example, it stops firewalld. When it stops it removes all the bans on the way out, and when you start it puts them all back. Frankly, not sure I like that, either. Seems like it would leave a window open as it logrotates and restarts both.

I need to investigate and see if I can do that with this setup... Because I still agree with you, blocking at the IP level for all parts for those malicious prods is the thing to do. In my case anyway. I could imagine some multi-tasking server were you woudl not want to lock a user out of oracle if they forgot their ssh password. But that's not my use scenario.

@Geraden07

This comment has been minimized.

Show comment
Hide comment
@Geraden07

Geraden07 Aug 1, 2017

@getgray I'm not sure what you mean. You never want to run systemctl stop firewalld because that stops your entire firewall service. If you want to reload a fresh firewall config (as if you restarted the service) then firewall-cmd --reload can do that for you.

As I've said above, the initial script I wrote seems to work perfectly. When an IP gets banned, fail2ban adds that bad IP to my drop zone in firewalld in both the runtime config, and the permanent config. So if I reload my firewall at anypoint it will continue to be banned.

Then whenever fail2ban wants to unban that IP because of the ban timeout, it can do so. What you really want to avoid is having any source OTHER than fail2ban messing with these banned IPs, because that can lead to errors where fail2ban attempts to unban an IP that is no longer banned, which is what seemed to be causing the error.

Edit: I read more of the recent messages and I see what you mean about maybe possible duplicate jails overlapping. I agree that your --change-source approach first to ensure it's banned, then unban it would work. I thought of an alternative approach that would force the unban commands to both be run and simply swallow the errors likes so:

actionunban = firewall-cmd --remove-source=<ip> --zone=drop 2> /dev/null; firewall-cmd --remove-source=<ip> --zone=drop --permanent 2> /dev/null

But I don't think it's as clean as your solution which would still give error information if some OTHER error occurred unrelated to whether the IP existed in the drop zone or not.

I will update my script up above to include the --change-source approach.

Geraden07 commented Aug 1, 2017

@getgray I'm not sure what you mean. You never want to run systemctl stop firewalld because that stops your entire firewall service. If you want to reload a fresh firewall config (as if you restarted the service) then firewall-cmd --reload can do that for you.

As I've said above, the initial script I wrote seems to work perfectly. When an IP gets banned, fail2ban adds that bad IP to my drop zone in firewalld in both the runtime config, and the permanent config. So if I reload my firewall at anypoint it will continue to be banned.

Then whenever fail2ban wants to unban that IP because of the ban timeout, it can do so. What you really want to avoid is having any source OTHER than fail2ban messing with these banned IPs, because that can lead to errors where fail2ban attempts to unban an IP that is no longer banned, which is what seemed to be causing the error.

Edit: I read more of the recent messages and I see what you mean about maybe possible duplicate jails overlapping. I agree that your --change-source approach first to ensure it's banned, then unban it would work. I thought of an alternative approach that would force the unban commands to both be run and simply swallow the errors likes so:

actionunban = firewall-cmd --remove-source=<ip> --zone=drop 2> /dev/null; firewall-cmd --remove-source=<ip> --zone=drop --permanent 2> /dev/null

But I don't think it's as clean as your solution which would still give error information if some OTHER error occurred unrelated to whether the IP existed in the drop zone or not.

I will update my script up above to include the --change-source approach.

@Geraden07

This comment has been minimized.

Show comment
Hide comment
@Geraden07

Geraden07 Aug 1, 2017

PS: for the record if anyone is playing around with this, here are the commands to tell fail2ban to ban an IP and unban an IP. Good for testing.

fail2ban-client set <jail name here> banip <ip here>
fail2ban-client set <jail name here> unbanip <ip here>

Geraden07 commented Aug 1, 2017

PS: for the record if anyone is playing around with this, here are the commands to tell fail2ban to ban an IP and unban an IP. Good for testing.

fail2ban-client set <jail name here> banip <ip here>
fail2ban-client set <jail name here> unbanip <ip here>
@getgray

This comment has been minimized.

Show comment
Hide comment
@getgray

getgray Aug 1, 2017

@Geraden07 Steven:

I'm not sure what you mean. You never want to run systemctl stop firewalld because that stops your entire firewall service.

I agree. It's my distribution's installation via yum. It's apparently related to logrotate, which is restarting firewalld, which in turn restarts fail2ban. OT for this git, so I'll investigate that issue separately.

That aside, it is nice that fail2ban via ipset action backs itself out when shutdown for whatever reason.
The ipset actions start and stop appear to be turning off all bans then turning them back on when it's restarted. I don't understand what is triggering the undo and redos and calls to "fail2ban persistent database", maybe an ipset thing over my head. Oh well, can't have everything :). Cheers, Scott

Snippets from a fail2ban restart on my box:
2017-08-01 04:02:04,551 fail2ban.server [860]: INFO Stopping all jails
2017-08-01 04:02:05,412 fail2ban.jail [860]: INFO Jail 'sshd' stopped
2017-08-01 04:02:07,904 fail2ban.actions [860]: NOTICE [postfix-postscreen] Unban 82.165.27.182
2017-08-01 04:02:08,330 fail2ban.jail [860]: INFO Jail 'postfix-postscreen' stopped
2017-08-01 04:02:10,315 fail2ban.server [860]: INFO Stopping all jails
2017-08-01 04:02:10,315 fail2ban.server [860]: INFO Exiting Fail2ban
...
2017-08-01 04:02:11,178 fail2ban.server [6139]: INFO Changed logging target to /var/log/fail2ban.log for Fail2ban v0.9.6
2017-08-01 04:02:11,178 fail2ban.database [6139]: INFO Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3'
...
[for every jail it does:]
...
2017-08-01 04:02:11,580 fail2ban.jail [6139]: INFO Creating new jail 'postfix-postscreen'
2017-08-01 04:02:11,825 fail2ban.jail [6139]: INFO Jail 'postfix-postscreen' started
2017-08-01 04:02:14,459 fail2ban.actions [6139]: NOTICE [postfix-postscreen] Ban 82.165.27.182

getgray commented Aug 1, 2017

@Geraden07 Steven:

I'm not sure what you mean. You never want to run systemctl stop firewalld because that stops your entire firewall service.

I agree. It's my distribution's installation via yum. It's apparently related to logrotate, which is restarting firewalld, which in turn restarts fail2ban. OT for this git, so I'll investigate that issue separately.

That aside, it is nice that fail2ban via ipset action backs itself out when shutdown for whatever reason.
The ipset actions start and stop appear to be turning off all bans then turning them back on when it's restarted. I don't understand what is triggering the undo and redos and calls to "fail2ban persistent database", maybe an ipset thing over my head. Oh well, can't have everything :). Cheers, Scott

Snippets from a fail2ban restart on my box:
2017-08-01 04:02:04,551 fail2ban.server [860]: INFO Stopping all jails
2017-08-01 04:02:05,412 fail2ban.jail [860]: INFO Jail 'sshd' stopped
2017-08-01 04:02:07,904 fail2ban.actions [860]: NOTICE [postfix-postscreen] Unban 82.165.27.182
2017-08-01 04:02:08,330 fail2ban.jail [860]: INFO Jail 'postfix-postscreen' stopped
2017-08-01 04:02:10,315 fail2ban.server [860]: INFO Stopping all jails
2017-08-01 04:02:10,315 fail2ban.server [860]: INFO Exiting Fail2ban
...
2017-08-01 04:02:11,178 fail2ban.server [6139]: INFO Changed logging target to /var/log/fail2ban.log for Fail2ban v0.9.6
2017-08-01 04:02:11,178 fail2ban.database [6139]: INFO Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3'
...
[for every jail it does:]
...
2017-08-01 04:02:11,580 fail2ban.jail [6139]: INFO Creating new jail 'postfix-postscreen'
2017-08-01 04:02:11,825 fail2ban.jail [6139]: INFO Jail 'postfix-postscreen' started
2017-08-01 04:02:14,459 fail2ban.actions [6139]: NOTICE [postfix-postscreen] Ban 82.165.27.182

@getgray

This comment has been minimized.

Show comment
Hide comment
@getgray

getgray Aug 1, 2017

@Geraden07 When using this method, I do miss being able to see the individual ban times that ipset shows. There's no command to show that is there?

getgray commented Aug 1, 2017

@Geraden07 When using this method, I do miss being able to see the individual ban times that ipset shows. There's no command to show that is there?

@RobbieTheK

This comment has been minimized.

Show comment
Hide comment
@RobbieTheK

RobbieTheK Aug 16, 2017

You can see from the logs below that unbans are still happening even though bantime = -1. This is Fedora 26 rpm -q fail2ban: fail2ban-0.9.7-2.fc26.noarch. Here are the related jail.local options and log snips. Is there some overlap with the ‘findtime' value?

bantime = 3600
sender = fail2ban
action = custom-firewalld
backend = auto
mta = sendmail

#[Definition]

[sshd]
enabled = true
filter   = sshd
logpath  = /var/log/secure

[sshd-ddos]
enabled = true
#port    = ssh,sftp
filter = sshd-ddos
logpath  = /var/log/secure
maxretry = 5

[pam-generic]
enabled  = true
# pam-generic filter can be customized to monitor specific subset of 'tty's
filter   = pam-generic
# port actually must be irrelevant but lets leave it all for some possible uses
port     = all
banaction = iptables-allports
logpath  = /var/log/secure
maxretry = 3

[recidive]
enabled  = true
filter   = recidive
logpath  = /var/log/fail2ban.log
action   = custom-firewalld[name=recidive]
           sendmail-whois-lines[name=recidive, logpath=/var/log/fail2ban.log]
bantime  = -1 ; forever
findtime = 86400   ; 1 day
maxretry = 5
 cat /var/log/fail2ban.log| grep 164.215.170.34
2017-08-15 16:41:26,524 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:26,530 fail2ban.filter         [1996]: INFO    [pam-generic] Found 164.215.170.34
2017-08-15 16:41:26,534 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:29,132 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:31,763 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:35,006 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:35,744 fail2ban.actions        [1996]: NOTICE  [sshd] Ban 164.215.170.34
2017-08-15 16:41:35,746 fail2ban.filter         [1996]: INFO    [recidive] Found 164.215.170.34
2017-08-15 16:41:36,811 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:39,525 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:42,563 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:45,622 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:48,273 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:48,277 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:48,527 fail2ban.actions        [1996]: NOTICE  [sshd] 164.215.170.34 already banned
2017-08-15 17:41:35,914 fail2ban.actions        [1996]: NOTICE  [sshd] Unban 164.215.170.34
fail2ban-client status | grep "Jail list:" | sed "s/ //g" | awk '{split($2,a,",");for(i in a) system("fail2ban-client status " a[i])}' | grep "Status\|IP list" | grep 164.215.170.34

does not return anything.

RobbieTheK commented Aug 16, 2017

You can see from the logs below that unbans are still happening even though bantime = -1. This is Fedora 26 rpm -q fail2ban: fail2ban-0.9.7-2.fc26.noarch. Here are the related jail.local options and log snips. Is there some overlap with the ‘findtime' value?

bantime = 3600
sender = fail2ban
action = custom-firewalld
backend = auto
mta = sendmail

#[Definition]

[sshd]
enabled = true
filter   = sshd
logpath  = /var/log/secure

[sshd-ddos]
enabled = true
#port    = ssh,sftp
filter = sshd-ddos
logpath  = /var/log/secure
maxretry = 5

[pam-generic]
enabled  = true
# pam-generic filter can be customized to monitor specific subset of 'tty's
filter   = pam-generic
# port actually must be irrelevant but lets leave it all for some possible uses
port     = all
banaction = iptables-allports
logpath  = /var/log/secure
maxretry = 3

[recidive]
enabled  = true
filter   = recidive
logpath  = /var/log/fail2ban.log
action   = custom-firewalld[name=recidive]
           sendmail-whois-lines[name=recidive, logpath=/var/log/fail2ban.log]
bantime  = -1 ; forever
findtime = 86400   ; 1 day
maxretry = 5
 cat /var/log/fail2ban.log| grep 164.215.170.34
2017-08-15 16:41:26,524 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:26,530 fail2ban.filter         [1996]: INFO    [pam-generic] Found 164.215.170.34
2017-08-15 16:41:26,534 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:29,132 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:31,763 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:35,006 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:35,744 fail2ban.actions        [1996]: NOTICE  [sshd] Ban 164.215.170.34
2017-08-15 16:41:35,746 fail2ban.filter         [1996]: INFO    [recidive] Found 164.215.170.34
2017-08-15 16:41:36,811 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:39,525 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:42,563 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:45,622 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:48,273 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:48,277 fail2ban.filter         [1996]: INFO    [sshd] Found 164.215.170.34
2017-08-15 16:41:48,527 fail2ban.actions        [1996]: NOTICE  [sshd] 164.215.170.34 already banned
2017-08-15 17:41:35,914 fail2ban.actions        [1996]: NOTICE  [sshd] Unban 164.215.170.34
fail2ban-client status | grep "Jail list:" | sed "s/ //g" | awk '{split($2,a,",");for(i in a) system("fail2ban-client status " a[i])}' | grep "Status\|IP list" | grep 164.215.170.34

does not return anything.

@sebres

This comment has been minimized.

Show comment
Hide comment
@sebres

sebres Aug 16, 2017

Member

@RobbieTheK
You've specified bantime = -1 for jail recidive only.
For jail sshd the bantime is not specified, thus default (1h according your config).

According to the log-excerpt jail recidive has found 1 failure but expected 5 (maxretry) to ban this IP.
Filter recidive search the IP banned with fail2ban, therefore it counts the bans not the failures (attempts).

Member

sebres commented Aug 16, 2017

@RobbieTheK
You've specified bantime = -1 for jail recidive only.
For jail sshd the bantime is not specified, thus default (1h according your config).

According to the log-excerpt jail recidive has found 1 failure but expected 5 (maxretry) to ban this IP.
Filter recidive search the IP banned with fail2ban, therefore it counts the bans not the failures (attempts).

@RobbieTheK

This comment has been minimized.

Show comment
Hide comment
@RobbieTheK

RobbieTheK Aug 16, 2017

@sebres thanks so is there a best practice or recommended setting to allow recidive to permanently ban these IPs? There's no way to include already banned from a fail2ban.actions NOTICE?

RobbieTheK commented Aug 16, 2017

@sebres thanks so is there a best practice or recommended setting to allow recidive to permanently ban these IPs? There's no way to include already banned from a fail2ban.actions NOTICE?

@sebres

This comment has been minimized.

Show comment
Hide comment
@sebres

sebres Aug 16, 2017

Member

best practice or recommended setting to allow recidive to permanently ban these IPs

I don't really understand what you exactly needed:

  • recidive jail is envisaged for the bad IPs, that makes repeated attempts, after these was already banned (thus counts bans from any other jails).
  • corresponding your last configuration the IP will be banned in recidive jail after 5 (maxretry) attempts (in this case after 5 bans in any other jails) within time interval of 1 day (findtime 86400 seconds). If you want another handling change maxretry and/or findtime. Note: normally recidive bans for all ports, thus may be very evil for legal users that just trying something resp. wrong recognized as bad through misconfiguration (or for you self, if your IP will be banned there).
  • if you want to permanent ban in some other jail, just set bantime = -1 to this jail. Note: this will ban this IPs for ports specified for this jails only.
  • if you want to count any fail attempt (not ban) in recidive jail, you should rewrite the regex from NOTICE ... Ban to INFO ... Found (but recidive self should be ignored like INFO\s+[recidive] Found, because of cyclic recursion otherwise).
Member

sebres commented Aug 16, 2017

best practice or recommended setting to allow recidive to permanently ban these IPs

I don't really understand what you exactly needed:

  • recidive jail is envisaged for the bad IPs, that makes repeated attempts, after these was already banned (thus counts bans from any other jails).
  • corresponding your last configuration the IP will be banned in recidive jail after 5 (maxretry) attempts (in this case after 5 bans in any other jails) within time interval of 1 day (findtime 86400 seconds). If you want another handling change maxretry and/or findtime. Note: normally recidive bans for all ports, thus may be very evil for legal users that just trying something resp. wrong recognized as bad through misconfiguration (or for you self, if your IP will be banned there).
  • if you want to permanent ban in some other jail, just set bantime = -1 to this jail. Note: this will ban this IPs for ports specified for this jails only.
  • if you want to count any fail attempt (not ban) in recidive jail, you should rewrite the regex from NOTICE ... Ban to INFO ... Found (but recidive self should be ignored like INFO\s+[recidive] Found, because of cyclic recursion otherwise).
@davama

This comment has been minimized.

Show comment
Hide comment
@davama

davama Dec 5, 2017

@Geraden07 Thank you very much for your custom action on #1474 (comment)

On our enviroment, using git fail2banv0.11.0.dev0, firewall-ipset.conf stopped banning. Fail2ban-client would show a banned ip but ipset did not. Centos7.4.1708.

Not a fan of custom config but hopefully this feature can be added to the base release.

davama commented Dec 5, 2017

@Geraden07 Thank you very much for your custom action on #1474 (comment)

On our enviroment, using git fail2banv0.11.0.dev0, firewall-ipset.conf stopped banning. Fail2ban-client would show a banned ip but ipset did not. Centos7.4.1708.

Not a fan of custom config but hopefully this feature can be added to the base release.

@sebres

This comment has been minimized.

Show comment
Hide comment
@sebres

sebres Dec 21, 2017

Member

Well, I had got time to check suggested action.d/custom-firewalld.conf:

  • this looks like a single jail action using all protocols and all ports banning, which have several problems:
    • using it with multiple jails (because no name bindings, one jail can unban ticket banned from another jail if bantime resp. start of bans are different).
    • not possible to ban only specified ports (allports action) for different jails;
    • the same applies to the different protocols;
  • IPv6 capability?
Member

sebres commented Dec 21, 2017

Well, I had got time to check suggested action.d/custom-firewalld.conf:

  • this looks like a single jail action using all protocols and all ports banning, which have several problems:
    • using it with multiple jails (because no name bindings, one jail can unban ticket banned from another jail if bantime resp. start of bans are different).
    • not possible to ban only specified ports (allports action) for different jails;
    • the same applies to the different protocols;
  • IPv6 capability?
@Geraden07

This comment has been minimized.

Show comment
Hide comment
@Geraden07

Geraden07 Dec 24, 2017

Hi @sebres

In response to your points:

  1. Using this with multiple jails can cause the problem where one jail unbans while another jail still has it banned, and the firewall would no longer ban the IP. This is a problem with management, and from a design perspective I would argue that this should be in the application logic of fail2ban, not a firewall functionality. From a division of responsibilities perspective, the firewall's job is to manage traffic according to the rules you give it. Fail2ban's responsibility is to add/manage/remove rules telling the firewall what to do.

In the old style, it was possible to setup chains in iptables so that you could have an IP banned more than once in the chain. After the first one is removed (by one of the jails) the other ban would still be present in the chain and so you don't need to worry about multiple jails when unbanning.

With firewalld, they've simplified the logic so you give it a single command to ban and another one to unban. This inherently means the fail2ban application will need to account for this. Since firewalld isn't going anywhere, I would imagine that you'll be forced to accommodate this sooner or later.

  1. The desire to ban ports specifically confuses me a little bit... We were already talking about this further up this thread, but essentially if an IP address gets jailed, it's considered a bad actor and not to be trusted. Why would you continue wanting to accept packets from that same IP on different ports... If you are running many servers, this protects them all from login attempts when the IP gets banned, and doesn't allow the attacker to just move on to the next port they want to try.

That being said, if you are absolutely set on offering this functionality, it IS possible (in a roundabout and somewhat dangerous way)... By setting a zone to accept all, then adding the IP and port you want to ban, you would get the desired result. But trusting all traffic by default and only banning that which you identify as bad is a really dangerous idea.

There may be a better alternative in doing this via the rich rules, but this is the only way using the zones functionality.

  1. Different protocols are essentially the same problem as the specific ports, so see my point number 2 above.

To your last point, though, there is good news. Firewalld is fully compatible with IPv6. Like I said in my very first post on this thread, using the respective zones results in the following:

If the 'block' zone were to be used, the response to banned IPs would be an icmp-host-prohibited message for IPv4 and icmp6-adm-prohibited for IPv6. If the 'drop' zone were used, the packets would be dropped with no response.

So you would not need to change the behaviour in anyway depending on IPv4 vs IPv6. That's hidden behind firewalld.

If you have any questions, let me know.

Geraden07 commented Dec 24, 2017

Hi @sebres

In response to your points:

  1. Using this with multiple jails can cause the problem where one jail unbans while another jail still has it banned, and the firewall would no longer ban the IP. This is a problem with management, and from a design perspective I would argue that this should be in the application logic of fail2ban, not a firewall functionality. From a division of responsibilities perspective, the firewall's job is to manage traffic according to the rules you give it. Fail2ban's responsibility is to add/manage/remove rules telling the firewall what to do.

In the old style, it was possible to setup chains in iptables so that you could have an IP banned more than once in the chain. After the first one is removed (by one of the jails) the other ban would still be present in the chain and so you don't need to worry about multiple jails when unbanning.

With firewalld, they've simplified the logic so you give it a single command to ban and another one to unban. This inherently means the fail2ban application will need to account for this. Since firewalld isn't going anywhere, I would imagine that you'll be forced to accommodate this sooner or later.

  1. The desire to ban ports specifically confuses me a little bit... We were already talking about this further up this thread, but essentially if an IP address gets jailed, it's considered a bad actor and not to be trusted. Why would you continue wanting to accept packets from that same IP on different ports... If you are running many servers, this protects them all from login attempts when the IP gets banned, and doesn't allow the attacker to just move on to the next port they want to try.

That being said, if you are absolutely set on offering this functionality, it IS possible (in a roundabout and somewhat dangerous way)... By setting a zone to accept all, then adding the IP and port you want to ban, you would get the desired result. But trusting all traffic by default and only banning that which you identify as bad is a really dangerous idea.

There may be a better alternative in doing this via the rich rules, but this is the only way using the zones functionality.

  1. Different protocols are essentially the same problem as the specific ports, so see my point number 2 above.

To your last point, though, there is good news. Firewalld is fully compatible with IPv6. Like I said in my very first post on this thread, using the respective zones results in the following:

If the 'block' zone were to be used, the response to banned IPs would be an icmp-host-prohibited message for IPv4 and icmp6-adm-prohibited for IPv6. If the 'drop' zone were used, the packets would be dropped with no response.

So you would not need to change the behaviour in anyway depending on IPv4 vs IPv6. That's hidden behind firewalld.

If you have any questions, let me know.

@sebres

This comment has been minimized.

Show comment
Hide comment
@sebres

sebres Dec 26, 2017

Member
  1. The jail identifies normally a chain (table, etc) by its name, where it is banned. The jails can have different timeouts, thus unbanning of the ticket in jail A should not cause an unbanning of it in jail B and vice versa.
  2. Which ports affected can (and should be) defined in respective jail. This is not necessarily the attackers only that going banned, possibly it is just a "normal" user, that has forgot the password. Or you self by testing of something, etc. Banning port-less is normally only prerogative of allports-jails like pam-generic resp. recidive.
Member

sebres commented Dec 26, 2017

  1. The jail identifies normally a chain (table, etc) by its name, where it is banned. The jails can have different timeouts, thus unbanning of the ticket in jail A should not cause an unbanning of it in jail B and vice versa.
  2. Which ports affected can (and should be) defined in respective jail. This is not necessarily the attackers only that going banned, possibly it is just a "normal" user, that has forgot the password. Or you self by testing of something, etc. Banning port-less is normally only prerogative of allports-jails like pam-generic resp. recidive.
@Geraden07

This comment has been minimized.

Show comment
Hide comment
@Geraden07

Geraden07 Apr 24, 2018

So just an update for anyone who reads this (will also be updating the script up top)...

I found that using
actionban = firewall-cmd --change-source <ip> && firewall-cmd --change-source <ip> --permanent
and
actionunban = firewall-cmd --change-source <ip> && firewall-cmd --change-source <ip> --permanent && firewall-cmd --remove-source <ip> && firewall-cmd --remove-source <ip> --permanent
were giving me performance problems when I started increasing my jail time lengths. With hundreds of IPs it would take a long time to shutdown (unbanning each of them individually) and usually timeout, orphaning any that weren't unbanned as being permanently banned.

This isn't hard to figure out though. Obviously, 2 commands to ban is okay, as large number of IPs are not regularly banned all at once, other than startup, and at startup nothing is blocked as it does so.

But shutting down, yikes that's a lot! 2 commands to make sure the IP is there, then another 2 to remove them. And again, under normal operation it's not a problem but while it shuts down, this blocks.

Obviously we can get marginally better performance by changing the unban to this:
actionunban = firewall-cmd --remove-source <ip> --zone=drop && firewall-cmd --remove-source <ip> --zone=drop --permanent || echo 0

We are down to 2 commands. But it still takes a long time during shutdown.

Luckily, I found #1743 from the devs have added the actionflush which, when defined as some non-empty command will stop the service from single unbanning everything as it shutsdown, and run the flush command instead.

Unfortunately there is not a single command to flush the blocked IPs from the drop zone. At least in the config that is loaded. We CAN cheat and modify the "permanent" config which is stored in a file in /etc/firewalld/zones/drop.xml (at least on my Fedora distro).

so by adding actionflush = sed -i '/<source address=/d' /etc/firewalld/zones/drop.xml we can stop the shutdown from blocking, and remove all blocked IPs from the permanent zone config.

My thinking is that if the service is going down due to a restart, this works just as well (the config will be loaded on boot). If you are going to be reloading the firewall config and wanted the fail2ban stopped first, again, same thing.

There IS the possibility that you are stopping and starting the fail2ban service without reloading the firewall. In this case, it will simply re-add all those banned IPs back to the permanent and runtime configs. There IS the slight possibility that during the time the fail2ban service was stopped, an IP was to be unbanned. So when the service starts again it won't re-ban that IP, but the IP will still be left in the runtime config. However, crucially, it will still be unbanned in the permanent config so a restart or reload of the firewall rule will stop a permanent ban from accidentally happening.

Anyway, if at this point anyone is still reading, I will be updating my script up top to reflect the latest version that I am running with. There is that very slight possibility of an IP leak, where IPs are not properly unbanned in the runtime while the fail2ban service is stopped. But since it's not permanent, I figure it's alright for the performance gains it brings.

Geraden07 commented Apr 24, 2018

So just an update for anyone who reads this (will also be updating the script up top)...

I found that using
actionban = firewall-cmd --change-source <ip> && firewall-cmd --change-source <ip> --permanent
and
actionunban = firewall-cmd --change-source <ip> && firewall-cmd --change-source <ip> --permanent && firewall-cmd --remove-source <ip> && firewall-cmd --remove-source <ip> --permanent
were giving me performance problems when I started increasing my jail time lengths. With hundreds of IPs it would take a long time to shutdown (unbanning each of them individually) and usually timeout, orphaning any that weren't unbanned as being permanently banned.

This isn't hard to figure out though. Obviously, 2 commands to ban is okay, as large number of IPs are not regularly banned all at once, other than startup, and at startup nothing is blocked as it does so.

But shutting down, yikes that's a lot! 2 commands to make sure the IP is there, then another 2 to remove them. And again, under normal operation it's not a problem but while it shuts down, this blocks.

Obviously we can get marginally better performance by changing the unban to this:
actionunban = firewall-cmd --remove-source <ip> --zone=drop && firewall-cmd --remove-source <ip> --zone=drop --permanent || echo 0

We are down to 2 commands. But it still takes a long time during shutdown.

Luckily, I found #1743 from the devs have added the actionflush which, when defined as some non-empty command will stop the service from single unbanning everything as it shutsdown, and run the flush command instead.

Unfortunately there is not a single command to flush the blocked IPs from the drop zone. At least in the config that is loaded. We CAN cheat and modify the "permanent" config which is stored in a file in /etc/firewalld/zones/drop.xml (at least on my Fedora distro).

so by adding actionflush = sed -i '/<source address=/d' /etc/firewalld/zones/drop.xml we can stop the shutdown from blocking, and remove all blocked IPs from the permanent zone config.

My thinking is that if the service is going down due to a restart, this works just as well (the config will be loaded on boot). If you are going to be reloading the firewall config and wanted the fail2ban stopped first, again, same thing.

There IS the possibility that you are stopping and starting the fail2ban service without reloading the firewall. In this case, it will simply re-add all those banned IPs back to the permanent and runtime configs. There IS the slight possibility that during the time the fail2ban service was stopped, an IP was to be unbanned. So when the service starts again it won't re-ban that IP, but the IP will still be left in the runtime config. However, crucially, it will still be unbanned in the permanent config so a restart or reload of the firewall rule will stop a permanent ban from accidentally happening.

Anyway, if at this point anyone is still reading, I will be updating my script up top to reflect the latest version that I am running with. There is that very slight possibility of an IP leak, where IPs are not properly unbanned in the runtime while the fail2ban service is stopped. But since it's not permanent, I figure it's alright for the performance gains it brings.

@sedstr

This comment has been minimized.

Show comment
Hide comment
@sedstr

sedstr May 6, 2018

I've applied your suggestions to my fail2ban configs, but I'm up to 5037 banned IPs.
This takes a lot of time to process on startup (and linux seems to need rebooting as often as windows nowadays).
Have you found a solution to help fail2ban ban the IPs again quicker on startup? It seems like its only able to ban round 3 IPs per second.
Its taking fail2ban + firewalld ~30 minutes to reban all the IPs again after reboot.
Surely there is a better way for fail2ban to work with firewalld?

sedstr commented May 6, 2018

I've applied your suggestions to my fail2ban configs, but I'm up to 5037 banned IPs.
This takes a lot of time to process on startup (and linux seems to need rebooting as often as windows nowadays).
Have you found a solution to help fail2ban ban the IPs again quicker on startup? It seems like its only able to ban round 3 IPs per second.
Its taking fail2ban + firewalld ~30 minutes to reban all the IPs again after reboot.
Surely there is a better way for fail2ban to work with firewalld?

@Geraden07

This comment has been minimized.

Show comment
Hide comment
@Geraden07

Geraden07 May 7, 2018

@sedstr yes this is an issue. Unfortunately I don't see any way to speed things up. No matter which way you cut it, we NEED the ban commands as they are since they are whats used when a new IP is detect and needs to be banned.

On startup, when there are huge numbers that need to be banned, the only thing I can think of is if there were an actionpopulate action similar but opposite to the actionflush that would help populate the initial bans on service start. To my knowledge this doesn't currently exist, though.

I think this is still a somewhat limited problem though. You have to remember the idea of fail2ban is not to stop all bad IPs from contacting you. This is impossible due to systems like Dynamic DNS, SSH tunneling, etc. where bad actors can simply look like they are coming from new/different IP addresses.

Instead, it's meant to detect repeated high volume login attempts in a short interval and ban to stop brute force attacks. Spot-checking attacks (admin/admin login attempts for example) will not be stopped, but that shouldn't really be an issue.

Even keeping the ban interval fairly short, like an hour, is enough to stop a brute force because the rate of dozens or hundreds of attempts per second if no banning takes place is now down to say 5 per hour (or whatever you configure). That's such a drastic reduction it basically makes brute forcing not viable and the attackers generally move on to more susceptible targets. I think the math works out to be something like a 99.98% reduction in attempt rate. (I assume what I think is a very conservative 5 attempts/sec brute force rate for 18000/hr. If this drops down to 5/hr, then we get a comparison of 5/18000 = 0.000214 or 0.02% this drops even more if you assume a much higher brute force rate). Extending the ban window out to weeks doesn't provide much benefit but does start to incur performance concerns like we are considering here.

So my personal recommendation is to shorten your ban windows to a matter of hours. Otherwise, you can still live with the 30min ban spin up when the service starts, as I also don't think that's too big a security concern when all matters are considered.

Geraden07 commented May 7, 2018

@sedstr yes this is an issue. Unfortunately I don't see any way to speed things up. No matter which way you cut it, we NEED the ban commands as they are since they are whats used when a new IP is detect and needs to be banned.

On startup, when there are huge numbers that need to be banned, the only thing I can think of is if there were an actionpopulate action similar but opposite to the actionflush that would help populate the initial bans on service start. To my knowledge this doesn't currently exist, though.

I think this is still a somewhat limited problem though. You have to remember the idea of fail2ban is not to stop all bad IPs from contacting you. This is impossible due to systems like Dynamic DNS, SSH tunneling, etc. where bad actors can simply look like they are coming from new/different IP addresses.

Instead, it's meant to detect repeated high volume login attempts in a short interval and ban to stop brute force attacks. Spot-checking attacks (admin/admin login attempts for example) will not be stopped, but that shouldn't really be an issue.

Even keeping the ban interval fairly short, like an hour, is enough to stop a brute force because the rate of dozens or hundreds of attempts per second if no banning takes place is now down to say 5 per hour (or whatever you configure). That's such a drastic reduction it basically makes brute forcing not viable and the attackers generally move on to more susceptible targets. I think the math works out to be something like a 99.98% reduction in attempt rate. (I assume what I think is a very conservative 5 attempts/sec brute force rate for 18000/hr. If this drops down to 5/hr, then we get a comparison of 5/18000 = 0.000214 or 0.02% this drops even more if you assume a much higher brute force rate). Extending the ban window out to weeks doesn't provide much benefit but does start to incur performance concerns like we are considering here.

So my personal recommendation is to shorten your ban windows to a matter of hours. Otherwise, you can still live with the 30min ban spin up when the service starts, as I also don't think that's too big a security concern when all matters are considered.

@sebres

This comment has been minimized.

Show comment
Hide comment
@sebres

sebres May 7, 2018

Member

I've actionbulkban functionality (quasi as counterpart to actionflush) within my own strict development branch, which can help with the initial banning after fail2ban restart (even as a result of the bulk banning), but primary it was developed also to ban more as one ticket (after some acceptable latency) to avoid too large banning-overhead e. g. under DDOS attack similar conditions.
Still no time ATM to back-port it to major branches. So currently I cannot say when it may be ready...

Member

sebres commented May 7, 2018

I've actionbulkban functionality (quasi as counterpart to actionflush) within my own strict development branch, which can help with the initial banning after fail2ban restart (even as a result of the bulk banning), but primary it was developed also to ban more as one ticket (after some acceptable latency) to avoid too large banning-overhead e. g. under DDOS attack similar conditions.
Still no time ATM to back-port it to major branches. So currently I cannot say when it may be ready...

@unioncos

This comment has been minimized.

Show comment
Hide comment
@unioncos

unioncos Aug 22, 2018

I am late to the party and I use CentOS 7 on a VPS so I do not screw around with ports at all - I just permanently block the IP with a reject/. Some VPS systems like those that use OpenVZ for instance do not support ipset in their configuration so ports and protocols do nothing and do not work and you won't know they are not working unless you use the --permanent option using the firewallcmd-rich-rules.conf file option and use it to write a rich rule and then reload firewalld OR you do a systemctl status firewalld.service. Then you will discover the ban action while being applied is not working at all and is ignored because ipset is not enabled in the firewall

One has to then modify the from " blocktype = reject type='icmp-port-unreachable' " to a simple " blocktype = 'reject' " of course without the outside quotes in the firewallcmd-rich-rules.conf and then change the jail.conf of course to jail.local and point it the firewallcmd-rich-rules.conf instead of the iptables-multiport.conf file, (and I make the ban permanent there too ) but if you do not fail2ban will append entries to the .xml file of the wrong format that does not work - for whatever zone is the default and they will be non-working rich text rules that firewalld will ignore on a reload whenever that happens.

At #1474 (comment) Geraden07 wrote in part

"The desire to ban ports specifically confuses me a little bit... We were already talking about this further up this thread, but essentially if an IP address gets jailed, it's considered a bad actor and not to be trusted. Why would you continue wanting to accept packets from that same IP on different ports... If you are running many servers, this protects them all from login attempts when the IP gets banned, and doesn't allow the attacker to just move on to the next port they want to try."

I agree and a simple reject/ drops their packet at the door to your server and they cannot access any port to try anything to put any load on the server to reject them at all. My logs have quieted across the board from mail to secure because offenders are offenders and they try all kinds of crap not just to ssh into the server. I am currently working on a fix for those who try to login the mail servers now using this same approach - most offenders of course come from non US ISP's

I read this post and the thing missing is how to get BOTH a permanent effect and a reload of the firewall at the same time. NOT using the --permanent option causes an immediate application of the rule, but it does not survive a reboot, so in that case then I watch fail2ban eat large amounts of CPU applying IP's from the permanent ban BLACKLIST FILE I use.

Currently it also looks like fail2ban will add multiple entries of the same IP if they come back and try again, and then firewalld will ignore the duplicate entry

Currently as I write this my public.xml file is growing as fail2ban writes a new SANE and PROPER rich text rules format xml file for permanent banning upon reload and it has gone from 24kb to 190kb (and still growing) and before I acted on the incorrect rich rule adds - it had grown to 1536kb at fail2ban's hand and a firewall-cmd --reload took about 7 minutes and ended in "success" but with all the ignored rich text rules due to ipset not being enabled on my VPS - which I can do nothing about as that is the OpenVZ setup the hosting company uses - so the simple reject/ command is all that would work anyway.

My actionban statement in the firewallcmd-rich-rules.conf I modified to be . . .

actionban = ports="<port>"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='<ip>' <blocktype>"; done if ! grep -Fq <ip> /etc/fail2ban/ip.blacklist; then echo "<ip> # fail2ban/$( date '+%%Y-%%m-%%d %%T' ): auto-add for repeat offender" >> /etc/fail2ban/ip.blacklist; fi

This way it creates a permanent blacklist file which currently has 21347 lines and each one is an IP - or CIDR subnet I added for whole countries ISP's and it is 432kb right now and the last entry in the blacklist file reads

xx.xx.xx.xx # fail2ban/2018-08-21 18:44:19: auto-add for repeat offender

. . . for the last jack-tail that tried to ssh into the server which was also caught and blocked by denyhosts too - which conveniently emails me that the new IP was added.

Now fail2ban is going through the entire blacklist file and creating working permanent rich rules for the xml file to simply reject the offending IP at the door to the server - period - and there is no unban use needed - though it should still work if called.

When I do a reload of firewalld - these IPs will become permanently blocked IP's that survive a reboot as read from the firewalld zone xml file

I just cannot edit the xml file while fail2ban is running because the version saves by two different sources to disk cause a loss of a proper format to the xml file. If I find duplicate IP's as reported that a rich rule is already enabled to an IP they are in the blacklist file and I need to change them there first, but each reload the firewall takes longer and longer - though right now only about 3 minutes with a 250kb xml zone file. I can reload it periodically as the system takes the snapshot of the xml file and reloads that

So I am asking about the actionbulkban dev work mentioned above at #1474 (comment)

unioncos commented Aug 22, 2018

I am late to the party and I use CentOS 7 on a VPS so I do not screw around with ports at all - I just permanently block the IP with a reject/. Some VPS systems like those that use OpenVZ for instance do not support ipset in their configuration so ports and protocols do nothing and do not work and you won't know they are not working unless you use the --permanent option using the firewallcmd-rich-rules.conf file option and use it to write a rich rule and then reload firewalld OR you do a systemctl status firewalld.service. Then you will discover the ban action while being applied is not working at all and is ignored because ipset is not enabled in the firewall

One has to then modify the from " blocktype = reject type='icmp-port-unreachable' " to a simple " blocktype = 'reject' " of course without the outside quotes in the firewallcmd-rich-rules.conf and then change the jail.conf of course to jail.local and point it the firewallcmd-rich-rules.conf instead of the iptables-multiport.conf file, (and I make the ban permanent there too ) but if you do not fail2ban will append entries to the .xml file of the wrong format that does not work - for whatever zone is the default and they will be non-working rich text rules that firewalld will ignore on a reload whenever that happens.

At #1474 (comment) Geraden07 wrote in part

"The desire to ban ports specifically confuses me a little bit... We were already talking about this further up this thread, but essentially if an IP address gets jailed, it's considered a bad actor and not to be trusted. Why would you continue wanting to accept packets from that same IP on different ports... If you are running many servers, this protects them all from login attempts when the IP gets banned, and doesn't allow the attacker to just move on to the next port they want to try."

I agree and a simple reject/ drops their packet at the door to your server and they cannot access any port to try anything to put any load on the server to reject them at all. My logs have quieted across the board from mail to secure because offenders are offenders and they try all kinds of crap not just to ssh into the server. I am currently working on a fix for those who try to login the mail servers now using this same approach - most offenders of course come from non US ISP's

I read this post and the thing missing is how to get BOTH a permanent effect and a reload of the firewall at the same time. NOT using the --permanent option causes an immediate application of the rule, but it does not survive a reboot, so in that case then I watch fail2ban eat large amounts of CPU applying IP's from the permanent ban BLACKLIST FILE I use.

Currently it also looks like fail2ban will add multiple entries of the same IP if they come back and try again, and then firewalld will ignore the duplicate entry

Currently as I write this my public.xml file is growing as fail2ban writes a new SANE and PROPER rich text rules format xml file for permanent banning upon reload and it has gone from 24kb to 190kb (and still growing) and before I acted on the incorrect rich rule adds - it had grown to 1536kb at fail2ban's hand and a firewall-cmd --reload took about 7 minutes and ended in "success" but with all the ignored rich text rules due to ipset not being enabled on my VPS - which I can do nothing about as that is the OpenVZ setup the hosting company uses - so the simple reject/ command is all that would work anyway.

My actionban statement in the firewallcmd-rich-rules.conf I modified to be . . .

actionban = ports="<port>"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='<ip>' <blocktype>"; done if ! grep -Fq <ip> /etc/fail2ban/ip.blacklist; then echo "<ip> # fail2ban/$( date '+%%Y-%%m-%%d %%T' ): auto-add for repeat offender" >> /etc/fail2ban/ip.blacklist; fi

This way it creates a permanent blacklist file which currently has 21347 lines and each one is an IP - or CIDR subnet I added for whole countries ISP's and it is 432kb right now and the last entry in the blacklist file reads

xx.xx.xx.xx # fail2ban/2018-08-21 18:44:19: auto-add for repeat offender

. . . for the last jack-tail that tried to ssh into the server which was also caught and blocked by denyhosts too - which conveniently emails me that the new IP was added.

Now fail2ban is going through the entire blacklist file and creating working permanent rich rules for the xml file to simply reject the offending IP at the door to the server - period - and there is no unban use needed - though it should still work if called.

When I do a reload of firewalld - these IPs will become permanently blocked IP's that survive a reboot as read from the firewalld zone xml file

I just cannot edit the xml file while fail2ban is running because the version saves by two different sources to disk cause a loss of a proper format to the xml file. If I find duplicate IP's as reported that a rich rule is already enabled to an IP they are in the blacklist file and I need to change them there first, but each reload the firewall takes longer and longer - though right now only about 3 minutes with a 250kb xml zone file. I can reload it periodically as the system takes the snapshot of the xml file and reloads that

So I am asking about the actionbulkban dev work mentioned above at #1474 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment