bug report: fail2ban only works when I add the IP manually on CentOS 7

[cottonthread]

📝 Preliminary Checks

• I tried searching for an existing issue and followed the debugging docs advice, but still need assistance.

👀 What Happened?

When I use fail2ban manually the system takes effect but when it comes from the mail system itself, it generates an error in adding the IP to the list, so it is not possible to block IPs automatically.

I have tried changing the fail2ban-jail.cf file, so that instead of running nftables it uses iptables as explained here, but I think I'm doing it wrong because the error is getting worse. (https://docker-mailserver.github.io/docker-mailserver/latest/config/debugging/)

Could it be that I am still running an old operating system like CentOS 7 with firewall-cmd as firewall?

👟 Reproduction Steps

"docker exec -it mail setup fail2ban ban IP" works, but when I do a test for example with "telnet IP 25" and generate several login errors, fail2ban detects this attempt but fails to block the IP.

🐋 DMS Version

v12.1.0

💻 Operating System and Architecture

CentOS 7

⚙️ Container configuration files

ports:
    -   "25:25"
    -   "110:110"
    -   "143:143"
    -   "587:587"
    -   "993:993"
    -   "995:995"
    -   "465:465"
    volumes:
    - $HOME/mailserver/mail/:/var/mail/
    - $HOME/mailserver/mail-state/:/var/mail-state/
    - $HOME/mailserver/logs/:/var/log/mail/
    - $HOME/mailserver/config/:/tmp/docker-mailserver/
    environment:
    -   "TZ=Europe/Madrid"
    -   "OVERRIDE_HOSTNAME=****"
    -   "DMS_DEBUG=1"
    -   "LOG_LEVEL=info"
    -   "SUPERVISOR_LOGLEVEL=info"
    -   "ONE_DIR=1"
    -   "POSTMASTER_ADDRESS=****"
    -   "PERMIT_DOCKER=connected-networks"
    -   "TLS_LEVEL="
    -   "SPOOF_PROTECTION="
    -   "ENABLE_SRS=0"
    -   "ENABLE_POP3=1"
    -   "ENABLE_CLAMAV=0"
    -   "ENABLE_FAIL2BAN=1"
    -   "ENABLE_MANAGESIEVE="
    -   "POSTSCREEN_ACTION=enforce"
    -   "SMTP_ONLY="
    -   "SSL_TYPE=manual"
    -   "SSL_CERT_PATH=/tmp/ssl/DNSCerts/live/****/fullchain.pem"
    -   "SSL_KEY_PATH=/tmp/ssl/DNSCerts/live/****/privkey.pem"
    -   "VIRUSMAILS_DELETE_DELAY="
    -   "ENABLE_POSTFIX_VIRTUAL_TRANSPORT="
    -   "POSTFIX_DAGENT="
    -   "POSTFIX_MAILBOX_SIZE_LIMIT=5368706371"
    -   "POSTFIX_MESSAGE_SIZE_LIMIT=26214400"
    -   "PFLOGSUMM_TRIGGER=daily_cron"
    -   "PFLOGSUMM_RECIPIENT=****"
    -   "PFLOGSUMM_SENDER="
    -   "LOGWATCH_INTERVAL=weekly"
    -   "LOGWATCH_RECIPIENT="
    -   "REPORT_RECIPIENT=0"
    -   "REPORT_SENDER="
    -   "REPORT_INTERVAL=weekly"
    -   "POSTFIX_INET_PROTOCOLS=ipv4"
    -   "ENABLE_SPAMASSASSIN=1"
    -   "SPAMASSASSIN_SPAM_TO_INBOX=1"
    -   "MOVE_SPAM_TO_JUNK=1"
    -   "SA_TAG=2.0"
    -   "SA_TAG2=6.31"
    -   "SA_KILL=6.31"
    -   "SA_SPAM_SUBJECT=***SPAM*****"
    -   "ENABLE_FETCHMAIL=0"
    -   "FETCHMAIL_POLL=300"
    -   "ENABLE_POSTGREY=0"
    -   "POSTGREY_DELAY=300"
    -   "POSTGREY_MAX_AGE=35"
    -   "POSTGREY_TEXT=Delayed by postgrey"
    -   "POSTGREY_AUTO_WHITELIST_CLIENTS=5"
    -   "SASL_PASSWD="
    -   "SRS_SENDER_CLASSES=envelope_sender"
    -   "SRS_EXCLUDE_DOMAINS="
    -   "SRS_SECRET="
    -   "DEFAULT_RELAY_HOST="
    -   "RELAY_HOST="
    -   "RELAY_PORT="
    -   "RELAY_USER="
    -   "RELAY_PASSWORD="
    -   "ENABLE_QUOTAS=1"

📜 Relevant log output

2023-10-09 01:15:26,064 fail2ban.actions        [788]: NOTICE  [dovecot] Restore Ban *IP*
2023-10-09 01:15:26,077 fail2ban.utils          [788]: ERROR   7f545417b5a0 -- exec: nft add table inet f2b-table
nft -- add chain inet f2b-table f2b-chain \{ type filter hook input priority -1 \; \}
nft add set inet f2b-table addr-set-dovecot \{ type ipv4_addr\; flags interval\; \}
for proto in $(echo 'tcp' | sed 's/,/ /g'); do
nft add rule inet f2b-table f2b-chain $proto dport \{ $(echo 'pop3,pop3s,imap,imaps,submission,465,sieve' | sed s/:/-/g) \} ip saddr @addr-set-dovecot drop
done
2023-10-09 01:15:26,077 fail2ban.utils          [788]: ERROR   7f545417b5a0 -- stderr: 'Error: Could not process rule: Numerical result out of range'
2023-10-09 01:15:26,077 fail2ban.utils          [788]: ERROR   7f545417b5a0 -- stderr: 'add set inet f2b-table addr-set-dovecot { type ipv4_addr; flags interval; }'
2023-10-09 01:15:26,077 fail2ban.utils          [788]: ERROR   7f545417b5a0 -- stderr: '^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^'
2023-10-09 01:15:26,077 fail2ban.utils          [788]: ERROR   7f545417b5a0 -- stderr: 'Error: No such file or directory'
2023-10-09 01:15:26,077 fail2ban.utils          [788]: ERROR   7f545417b5a0 -- stderr: 'add rule inet f2b-table f2b-chain tcp dport { pop3,pop3s,imap,imaps,submission,465,sieve } ip saddr @addr-set-dovecot drop'
2023-10-09 01:15:26,077 fail2ban.utils          [788]: ERROR   7f545417b5a0 -- stderr: '                                                                                                    ^^^^^^^^^^^^^^^^^'
2023-10-09 01:15:26,077 fail2ban.utils          [788]: ERROR   7f545417b5a0 -- returned 1
2023-10-09 01:15:26,077 fail2ban.actions        [788]: ERROR   Failed to execute ban jail 'dovecot' action 'nftables-multiport' info 'ActionInfo({'ip': '*IP*', 'family': 'inet4', 'fid': <function Actions.ActionInfo.<lambda> at 0x7f5455826f70>, 'raw-ticket': <function Actions.ActionInfo.<lambda> at 0x7f5455825670>})': Error starting action Jail('dovecot')/nftables-multiport: 'Script error'

Improvements to this form?

No response

[polarathene]

• You've not provided the full compose.yaml / docker-compose.yml config, that would probably be helpful.
• Your environment settings seems odd. Many of these are either defaults or you're explicitly unsetting them? You even have some that aren't valid anymore such as SASL_PASSWD (removed since the April release of v12).

Don't forget to enable the needed capability:

docker-mailserver/compose.yaml

Lines 26 to 27 in 82c38f2

	# cap_add:
	# - NET_ADMIN

---

• Error: Could not process rule: Numerical result out of range
• Error: No such file or directory
• Error starting action Jail('dovecot')/nftables-multiport: 'Script error'

The ban command from our setup fail2ban ban <ip address here> support internally runs this:

docker-mailserver/target/bin/fail2ban

Line 43 in 82c38f2

RESULT=$(fail2ban-client set custom banip "${@}")

And uses this custom jail:

docker-mailserver/target/fail2ban/jail.local

Lines 35 to 40 in 82c38f2

	# This jail is used for manual bans.
	# To ban an IP address use: setup.sh fail2ban ban <IP>
	[custom]
	enabled = true
	bantime = 180d
	port = smtp,pop3,pop3s,imap,imaps,submission,submissions,sieve

I'm assuming the quoted log errors above are from the fail2ban service trying to ban implicitly for you? That should be using the dovecot jail the logs mention, but this isn't that different jail config wise:

docker-mailserver/target/fail2ban/jail.local

Lines 23 to 24 in 82c38f2

	[dovecot]
	enabled = true

Not quite sure what to make of the logged errors other than possibly compatibility issues with your Docker host.

---

Could it be that I am still running an old operating system like CentOS 7

Quite possibly. That will be officially EOL in less than a years time.

Please share your kernel version and docker + compose versions. Additionally if you can the systemd version, which may be quite old.

It's possible your kernel and systemd versions don't support cgroups v2, which may not matter here but had some importance in the debug docs IIRC (see the :edge version of the debugging docs for many more insights on compatibility gotchas). I know there was some nftables related improvements mentioned that older kernels would lack too.

[cottonthread]

• You've not provided the full compose.yaml / docker-compose.yml config, that would probably be helpful.
• Your environment settings seems odd. Many of these are either defaults or you're explicitly unsetting them? You even have some that aren't valid anymore such as SASL_PASSWD (removed since the April release of v12).

Don't forget to enable the needed capability:

docker-mailserver/compose.yaml

Lines 26 to 27 in 82c38f2

	# cap_add:
	# - NET_ADMIN

I started using DMS since version 7 or something like that, and I have always tried to keep these parameters, I imagine they are now very obsolete, I will try to update them, as of today my file is concretely like this:

version: '3'
services:
  mail:
    image: mailserver/docker-mailserver:12.1.0
    hostname: mail
    domainname: *hidden*
    container_name: mail
    ports:
    -   "25:25"
    -   "110:110"
    -   "143:143"
    -   "587:587"
    -   "993:993"
    -   "995:995"
    -   "465:465"
    volumes:
    - $HOME/mailserver/mail/:/var/mail/
    - $HOME/mailserver/mail-state/:/var/mail-state/
    - $HOME/mailserver/logs/:/var/log/mail/
    - $HOME/mailserver/config/:/tmp/docker-mailserver/
    - $HOME/www/DNSCerts/:/tmp/ssl/DNSCerts/:ro
    environment:
    -   "TZ=Europe/Madrid"
    -   "OVERRIDE_HOSTNAME=mail.*hidden*"
    -   "DMS_DEBUG=1"
    -   "LOG_LEVEL=info"
    -   "SUPERVISOR_LOGLEVEL=info"
    -   "ONE_DIR=1"
    -   "POSTMASTER_ADDRESS=*hidden*@*hidden*"
    -   "PERMIT_DOCKER=connected-networks"
    -   "TLS_LEVEL="
    -   "SPOOF_PROTECTION="
    -   "ENABLE_SRS=0"
    -   "ENABLE_POP3=1"
    -   "ENABLE_CLAMAV=0"
    -   "ENABLE_FAIL2BAN=1"
    -   "ENABLE_MANAGESIEVE="
    -   "POSTSCREEN_ACTION=enforce"
    -   "SMTP_ONLY="
    -   "SSL_TYPE=manual"
    -   "SSL_CERT_PATH=/tmp/ssl/DNSCerts/live/*hidden*/fullchain.pem"
    -   "SSL_KEY_PATH=/tmp/ssl/DNSCerts/live/*hidden*/privkey.pem"
    -   "VIRUSMAILS_DELETE_DELAY="
    -   "ENABLE_POSTFIX_VIRTUAL_TRANSPORT="
    -   "POSTFIX_DAGENT="
    -   "POSTFIX_MAILBOX_SIZE_LIMIT=5368706371"
    -   "POSTFIX_MESSAGE_SIZE_LIMIT=26214400"
    -   "PFLOGSUMM_TRIGGER=daily_cron"
    -   "PFLOGSUMM_RECIPIENT=*hidden*@*hidden*"
    -   "PFLOGSUMM_SENDER="
    -   "LOGWATCH_INTERVAL=weekly"
    -   "LOGWATCH_RECIPIENT="
    -   "REPORT_RECIPIENT=0"
    -   "REPORT_SENDER="
    -   "REPORT_INTERVAL=weekly"
    -   "POSTFIX_INET_PROTOCOLS=ipv4"
    -   "ENABLE_SPAMASSASSIN=1"
    -   "SPAMASSASSIN_SPAM_TO_INBOX=1"
    -   "MOVE_SPAM_TO_JUNK=1"
    -   "SA_TAG=2.0"
    -   "SA_TAG2=6.31"
    -   "SA_KILL=6.31"
    -   "SA_SPAM_SUBJECT=***SPAM*****"
    -   "ENABLE_FETCHMAIL=0"
    -   "FETCHMAIL_POLL=300"
    -   "ENABLE_POSTGREY=0"
    -   "POSTGREY_DELAY=300"
    -   "POSTGREY_MAX_AGE=35"
    -   "POSTGREY_TEXT=Delayed by postgrey"
    -   "POSTGREY_AUTO_WHITELIST_CLIENTS=5"
    -   "SASL_PASSWD="
    -   "SRS_SENDER_CLASSES=envelope_sender"
    -   "SRS_EXCLUDE_DOMAINS="
    -   "SRS_SECRET="
    -   "DEFAULT_RELAY_HOST="
    -   "RELAY_HOST="
    -   "RELAY_PORT="
    -   "RELAY_USER="
    -   "RELAY_PASSWORD="
    -   "ENABLE_QUOTAS=1"
    cap_add:
    - NET_ADMIN
    - SYS_PTRACE
    restart: always
    stop_grace_period: 1m30s

  db:
    image: mysql:latest
    container_name: mysql
    command: --default-authentication-plugin=mysql_native_password
    restart: always
    environment:
    -   "TZ=Europe/Madrid"
    -   "MYSQL_ROOT_PASSWORD=*hidden*"
    volumes:
    - $HOME/www/MySQL:/var/lib/mysql
    security_opt:
    - seccomp:unconfined

  adminer:
    image: adminer
    container_name: adminer
    restart: always
    volumes:
    - /etc/localtime:/etc/localtime:ro
    ports:
    - "8088:8080"

  roundcube:
    image: roundcube/roundcubemail:latest
    container_name: roundcube
    hostname: *hidden*
    volumes:
    - $HOME/www/RoundCube/config.inc.php:/var/roundcube/config/config.inc.php
    - $HOME/www/RoundCube/logo.svg:/var/www/html/logo.svg
    - $HOME/www/RoundCube/CSS/elastic/:/var/www/html/skins/elastic/
    # - $HOME/www/RoundCube/plugin.password.config.inc.php:/var/www/html/plugins/password/config.inc.php
    - /etc/localtime:/etc/localtime:ro
    restart: always
    environment:
    -   "ROUNDCUBEMAIL_DEFAULT_HOST=tls://mail"
    -   "ROUNDCUBEMAIL_SMTP_SERVER=tls://mail"
    -   "ROUNDCUBEMAIL_PLUGINS=archive,zipdownload,emoticons,vcard_attachments,filesystem_attachments,newmail_notifier"
    -   "ROUNDCUBEMAIL_DB_TYPE=mysql"
    -   "ROUNDCUBEMAIL_DB_HOST=mysql"
    -   "ROUNDCUBEMAIL_DB_NAME=roundcubemail"
    -   "ROUNDCUBEMAIL_DB_PORT=3306"
    -   "ROUNDCUBEMAIL_DB_USER=roundcube"
    -   "ROUNDCUBEMAIL_DB_PASSWORD=*hidden*"
    -   "ROUNDCUBEMAIL_UPLOAD_MAX_FILESIZE=20M"
  caddy:
    image: abiosoft/caddy:php
    container_name: caddy
    hostname: *hidden*
    volumes:
    - $HOME/www/Caddyfile:/etc/Caddyfile:ro
    - $HOME/www/Certfications:/root/.caddy
    - $HOME/www/DNSCerts/:/DNSCerts/
    - $HOME/www/ExternalCerts/:/ExternalCerts/
    - $HOME/www/WebSites:/srv
    - $HOME/www/php.ini:/etc/php7/conf.d/php.ini
    restart: always
    environment:
    -   "ACME_AGREE=true"
    -   "ENABLE_TELEMETRY=false"
    -   "TZ=Europe/Madrid"
    ports:
    -   "80:80"
    -   "443:443"
    depends_on:
    -   roundcube
    -   db
    -   mail

  ftp:
    image: stilliard/pure-ftpd:latest
    container_name: ftp
    environment:
    -   "PUBLICHOST=*hidden*"
    -   "FTP_USER_HOME=/home/ftpusers"
    -   "ADDED_FLAGS=-s -A -Z -H -4 -X -x -d -d -O w3c:/var/log/pure-ftpd/transfer.log"
    -   "TZ=Europe/Madrid"
    volumes:
    - $HOME/www/WebSites:/home/ftpusers
    - $HOME/www/FTP/passwd:/etc/pure-ftpd/passwd
    # - ./FTP/logs/pureftpd.log:/var/log/pure-ftpd/pureftpd.log
    # - ./FTP/logs/transfer.log:/var/log/pure-ftpd/transfer.log
    restart: always
    ports:
    -   "21:21"
    -   "30000-30009:30000-30009"

• Error: Could not process rule: Numerical result out of range
• Error: No such file or directory
• Error starting action Jail('dovecot')/nftables-multiport: 'Script error'

The ban command from our setup fail2ban ban <ip address here> support internally runs this:

docker-mailserver/target/bin/fail2ban

Line 43 in 82c38f2

RESULT=$(fail2ban-client set custom banip "${@}")

And uses this custom jail:

docker-mailserver/target/fail2ban/jail.local

Lines 35 to 40 in 82c38f2

	# This jail is used for manual bans.
	# To ban an IP address use: setup.sh fail2ban ban <IP>
	[custom]
	enabled = true
	bantime = 180d
	port = smtp,pop3,pop3s,imap,imaps,submission,submissions,sieve

I'm assuming the quoted log errors above are from the fail2ban service trying to ban implicitly for you? That should be using the dovecot jail the logs mention, but this isn't that different jail config wise:

docker-mailserver/target/fail2ban/jail.local

Lines 23 to 24 in 82c38f2

	[dovecot]
	enabled = true

Not quite sure what to make of the logged errors other than possibly compatibility issues with your Docker host.

When I get this error is when DMS trying to recover banned IPs or when Fail2ban manages on its own (for failed attempts by dovecot account for example), this kind of error appears. But when I do it explicitly, it works fine.

Could it be that I am still running an old operating system like CentOS 7

Quite possibly. That will be officially EOL in less than a years time.

Please share your kernel version and docker + compose versions. Additionally if you can the systemd version, which may be quite old.

Kernel:
	4.4.248-1.el7.elrepo.x86_64
Systemd:
	systemd 219 +PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN
Docker:
	 Client: Docker Engine - Community
	 Version:           24.0.6
	 API version:       1.43
	 Go version:        go1.20.7
	 Git commit:        ed223bc
	 Built:             Mon Sep  4 12:35:25 2023
	 OS/Arch:           linux/amd64
	 Context:           default
	Server: Docker Engine - Community
	 Engine:
	  Version:          24.0.6
	  API version:      1.43 (minimum version 1.12)
	  Go version:       go1.20.7
	  Git commit:       1a79695
	  Built:            Mon Sep  4 12:34:28 2023
	  OS/Arch:          linux/amd64
	  Experimental:     false
	 containerd:
	  Version:          1.6.24
	  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
	 runc:
	  Version:          1.1.9
	  GitCommit:        v1.1.9-0-gccaecfc
	 docker-init:
	  Version:          0.19.0
	  GitCommit:        de40ad0
Docker-compose:
	docker-compose version 1.29.2, build unknown
	docker-py version: 5.0.0
	CPython version: 3.6.8
	OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017
Firewalld:
	0.6.3

It's possible your kernel and systemd versions don't support cgroups v2, which may not matter here but had some importance in the debug docs IIRC (see the :edge version of the debugging docs for many more insights on compatibility gotchas). I know there was some nftables related improvements mentioned that older kernels would lack too.

Reading on https://docker-mailserver.github.io/docker-mailserver/edge/config/debugging/ it seems to me that my system is too obsolete in making DMS v12 work well with Fail2ban, especially in the aspect of firewalld and cgroups v2. Could you tell me, in this case, DMS will be affected in other aspects? For me doing a system upgrade from CentOS 7 to an Ubuntu 22.04 LTS would be very risky.

[polarathene]

For me doing a system upgrade from CentOS 7 to an Ubuntu 22.04 LTS would be very risky.

You're going to want to get around to this at some point though.

Kernel 4.4 went EOL in Feb 2022 with version 4.4.302, yours is quite behind even that. CentOS 7 will be EOL next year too.

Your OpenSSL is likewise quite outdated. What risks are you concerned about? Are you able to make a backup of your data? Your system will be vulnerable over time without security fixes for exploits that are discovered.

Docker Compose is likewise outdated now, you'd want the 2.x series. Not sure how well that'll continue to work with the latest Docker releases that you're keeping up with.

Perhaps you can hold out until Ubuntu 24.04 LTS, but you should plan to migrate by sometime next year.

---

Could you tell me, in this case, DMS will be affected in other aspects?

I can only refer you to the Debugging docs page you've seen. I wrote that with information based on information with prior bug reports. Some systems had compatibility issues for example due to how old they were. That's unfortunately a risk you take with continuing to use software (and soon OS release) that go EOL, we cannot provide much support for that.

---

When I get this error is when DMS trying to recover banned IPs or when Fail2ban manages on its own (for failed attempts by dovecot account for example), this kind of error appears. But when I do it explicitly, it works fine.

Fail2Ban isn't an area of my expertise in the project. That would be @casperklein who I've assigned, if they've got time they might be able to respond with why explicit bans work and implicit ones fail. It may be your kernel, or something related to SELinux / firewalld. Hard to say 😅

[casperklein]

Unfortunately I don't have an idea how to fix that 😞 I second @polarathene advice to try a more recent OS.

[github-actions]

This issue has become stale because it has been open for 20 days without activity.
This issue will be closed in 10 days automatically unless:

• a maintainer removes the meta/stale label or adds the stale-bot/ignore label
• new activity occurs, such as a new comment

[github-actions]

This issue was closed due to inactivity.