[BR]: fail2ban does not start on some debian/ubuntu systems - backend should probably be set to systemd on all systemd-based distros

[tobixen]

Environment:

• Fail2Ban master branch, as well as version 0.11.1 on Ubuntu Focal and many others
• OS, including release name/version : Ubuntu Focal. Allegedly, Ubuntu Xenial is also affected, as well as some Debian installations.
• Fail2Ban installed via OS/distribution mechanisms
• You have not applied any additional foreign patches to the codebase
• Some customizations were done to the configuration (provide details below is so)

The issue:

On modern systemd-based distros, like newer releases of Ubuntu, Debian, Archlinux, RHEL, Fedora, etc, services like sshd logs to the systemd journal. Optionally rsyslog or syslog can be installed and run, and logs will also be available i.e. in /var/log/auth.log or /var/log/secure.log.

In the files /etc/fail2ban/paths-{arch|fedora|opensuse} there is a section like this:

syslog_backend = systemd
sshd_backend = systemd
dropbear_backend = systemd
proftpd_backend = systemd
pureftpd_backend = systemd
wuftpd_backend = systemd
postfix_backend = systemd
dovecot_backend = systemd

... and because of this, fail2ban works on arch, fedora (with derivatives) and opensuse. However, it fails on debian and ubuntu, unless the syslog package is installed and the service is running. Apparently this was reported as early as 2014 at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=770171 and it's also reported for Ubuntu 16.04 at https://bugs.launchpad.net/ubuntu/+source/fail2ban/+bug/1696591 ... we're running Ubuntu and EL. On our baseimages with Ubuntu 16.04, fail2ban installs and runs because we have syslog running there, on Ubuntu 20.04 we've had to hand-tune configuration to get fail2ban run on the services above. On EL it also works due to the paths-fedora.conf-file.

Steps to reproduce

• Install Ubuntu 20.04
• Observe that the OS installation comes without any rsyslog/syslog package installed (I suppose YMMV, dependent on how it's installed)
• Install fail2ban (sudo apt-get install fail2ban)
• Enable the sshd jail: echo -e "[sshd]\nenabled=true" | sudo tee /etc/fail2ban/jail.local
• Start up fail2ban: sudo systemctl start fail2ban
• Observe that it's not running: sudo systemctl status fail2ban - the error message looks like ERROR Failed during configuration: Have not found any log file for sshd jail.

Suggestion

I suggest creating a /etc/fail2ban/paths-systemd containing only the lines *_backend = systemd, and make sure it's run from any operating system having systemd installed.

Configuration, dump and another helpful excerpts

Any customizations done to /etc/fail2ban/ configuration

$ sudo cat /etc/fail2ban/jail.local 
[sshd]
enabled=true

[sebres]

It looks to me like a pretty maintainer issue. @fail2ban/maintainers, @sylvestre Any idea?

From other point of view it is a configuration issue and can be surely simply fixed using:

- echo -e "[sshd]\nenabled=true" | sudo tee /etc/fail2ban/jail.local
+ echo -e "[sshd]\nbackend=systemd\nenabled=true" | sudo tee /etc/fail2ban/jail.local

With other words:

  $ sudo cat /etc/fail2ban/jail.local 
  [sshd]
+ backend=systemd
  enabled = true

Pretty simply, isn't?

... and because of this, fail2ban works on arch, fedora (with derivatives) and opensuse

Well, it may work for sshd jail but could fail for some other jail if it gets enabled, but much more important is vice versa situation...

I suggest creating a /etc/fail2ban/paths-systemd containing only the lines *_backend = systemd, and make sure it's run from any operating system having systemd installed.

... that the issue with such a configuration would be - the system having systemd installed doesn't guarantee that every service affected by that include for which the jail would use systemd backend by default will also log into systemd journal.
Moreover it may be dangerous to do that by default, because:

1. if the log file is missing you'd be notified by fail2ban about and so can fix it (e. g. by changing the backend)
2. but if the backend is configured to systemd incorrectly for some jail, fail2ban is unable to detect whether the service logging there or not, so the jail would just not find any message from the service, simply because it will be not written to the journal (ability to read from journal doesn't imply that something, fail2ban is waiting for, writes to the journal at all). With other words if fail2ban starts without errors it would not necessarily mean that it "works" in that case.

The former is unpleasant but safe against possible misconfiguration, whereas the later is pretty ugly and hardly noticeable.

However I'm not against the suggestion to make /etc/fail2ban/paths-systemd as a part of fail2ban.

[tobixen]

It looks to me like a pretty maintainer issue. @fail2ban/maintainers, @sylvestre Any idea?

I thought so at first ... but since the path-*.conf is part of the fail2ban git repository, I feel it should be handled here.

As mentioned, this has also been flagged as a debian bug since 2014 ...

With other words:

  $ sudo cat /etc/fail2ban/jail.local 
  [sshd]
+ backend=systemd
  enabled = true

Pretty simply, isn't?

Well, for you and me ... sure, trivial ... but for anyone setting up fail2ban without any knowledge of the package, and just following the instructions (at least as they are at the top of jail.conf), I think it's a bad user experience when it doesn't work. Actually, it did take me quite some time digging to figure out that the trick is backend=systemd.

... that the issue with such a configuration would be - the system having systemd installed doesn't guarantee that every service affected by that include for which the jail would use systemd backend by default will also log into systemd journal. Moreover it may be dangerous to do that by default, because:

1. if the log file is missing you'd be notified by fail2ban about and so can fix it (e. g. by changing the backend)
2. but if the backend is configured to systemd incorrectly for some jail, fail2ban is unable to detect whether the service logging there or not, so the jail would just not find any message from the service, simply because it will be not written to the journal (ability to read from journal doesn't imply that something, fail2ban is waiting for, writes to the journal at all). With other words if fail2ban starts without errors it would not necessarily mean that it "works" in that case.

You may have a point, I'm just not sure if it is significant. I strongly believe that:

• For all combinations of OS distributions coming with systemd and not being EOL'ed a very long time ago and services on the list (sshd, dropbear, proftpd, pureftpd, wuftpd, postfix, dovecot), one can reliably find the logging in the journal, unless one has deliberately installed and configured things in a "weird" manner.
• One cannot trust fail2ban to work as intended without doing a minimum of testing.
• The risk that people will not use fail2ban because it's too difficult to set up is greater than the risk of fail2ban silently failing because the failed login information was only found in some logfile while fail2ban expected it to be in the journal.
• There is also a risk that the log files fail2ban is searching for exists, but that the information is only written to the journal
• There will always be people that are aiming to shoot themselves in the foot, and pulling the trigger, it's pointless to try to think of all the ways it can be done. My colleague was setting up fail2ban on Ubuntu the other day, did not verify that it was working as it should - but he also did not verify that the service was running. In that case, it would definitively have been better that fail2ban would have been just working "out of the box".

I've been looking a bit around... unfortunately I have no Debian to test with, but I have plenty of Ubuntu flavors, plus some CentOS and RHEL.

• I have an EL7-box with HAProxy running, it logs to syslog but not to journal - meaning that your scenario is possible. Then again, haproxy is not on the list. In the default configuration included in fail2ban the log path is hardcoded to /var/log/haproxy.log with no backend given.
• Ubuntu Precise does not have systemd installed, and it's also very EOL, so it can be ignored.
• Ubuntu Trusty does not come with systemd, but it seems possible to install it. I didn't test. I probably should have tested. Trusty is EOL, but it's possible to buy "extended support" until April 2024.
• Ubuntu Xenial comes with systemd installed. postfix and ssh logs to journal. I have rsyslog installed as well, it ensures the logs ends up in the files that fail2ban expects - but only as long as rsyslog is running! If the rsyslog service is stopped, logging to the files will stop, but logging to the journal will continue.

I could certainly do more testing, like checking every service on the list (sshd, dropbear, proftpd, pureftpd, wuftpd, postfix, dovecot ... syslog is also on the list, perhaps that one should be taken out). Of course it could be all down to "confirmation bias", but what I've seen just makes me stronger in my belief that systemd is the more correct backend for all those services for all systems running systemd :-)

[artfulrobot]

I just came across this today on a fresh Debian 12 install.

It seems a (packaging) bug to me that installing the fail2ban package results in a crashed service. It's taken a bit of searching to get here, and I still haven't got fail2ban running.

I have installed in the usual way:

apt install fail2ban

Then I created /etc/fail2ban/jail.local

[DEFAULT]
# Debian 12 has no log files, just journalctl
backend = systemd

# "bantime" is the number of seconds that a host is banned.
bantime  = 1d
# "maxretry" is the number of failures before a host get banned.
maxretry = 5
# A host is banned if it has generated "maxretry" during the last "findtime"
findtime  = 1h

[sshd]
enabled = true

And then tried to start the service again, but it crashes.

× fail2ban.service - Fail2Ban Service
     Loaded: loaded (/lib/systemd/system/fail2ban.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Tue 2023-08-15 13:07:06 BST; 4s ago
   Duration: 807ms
       Docs: man:fail2ban(1)
    Process: 32501 ExecStart=/usr/bin/fail2ban-server -xf start (code=exited, status=255/EXCEPTION)
   Main PID: 32501 (code=exited, status=255/EXCEPTION)
        CPU: 516ms

Aug 15 13:07:05 ar11 systemd[1]: Started fail2ban.service - Fail2Ban Service.
Aug 15 13:07:06 ar11 fail2ban-server[32501]: 2023-08-15 13:07:06,056 fail2ban.configreader   [32501]: WARNING 'allowipv6' not defined in 'Definition'. Using default one: 'auto'
Aug 15 13:07:06 ar11 fail2ban-server[32501]: Server ready
Aug 15 13:07:06 ar11 systemd[1]: fail2ban.service: Main process exited, code=exited, status=255/EXCEPTION
Aug 15 13:07:06 ar11 systemd[1]: fail2ban.service: Failed with result 'exit-code'.

I note that there's /var/log/fail2ban.log which contains:

2023-08-15 13:07:06,212 fail2ban.server         [32501]: INFO    Starting Fail2ban v1.0.2
2023-08-15 13:07:06,213 fail2ban.observer       [32501]: INFO    Observer start...
2023-08-15 13:07:06,219 fail2ban.database       [32501]: INFO    Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3'
2023-08-15 13:07:06,219 fail2ban.jail           [32501]: INFO    Creating new jail 'sshd'
2023-08-15 13:07:06,424 fail2ban.jail           [32501]: ERROR   Backend 'systemd' failed to initialize due to No module named 'systemd'
... absolutely HUGE stack trace here ...
2023-08-15 13:07:06,428 fail2ban                [32501]: ERROR   NOK: ("Failed to initialize any backend for Jail 'sshd'",)
2023-08-15 13:07:06,429 fail2ban.server         [32501]: INFO    Shutdown in progress...
2023-08-15 13:07:06,429 fail2ban.observer       [32501]: INFO    Observer stop ... try to end queue 5 seconds
2023-08-15 13:07:06,450 fail2ban.observer       [32501]: INFO    Observer stopped, 0 events remaining.
2023-08-15 13:07:06,490 fail2ban.server         [32501]: INFO    Stopping all jails
2023-08-15 13:07:06,490 fail2ban.database       [32501]: INFO    Connection to database closed.
2023-08-15 13:07:06,691 fail2ban.server         [32501]: INFO    Exiting Fail2ban

Finally I realised I also needed to install python3-systemd package. And now it appears to be running.

TL;DR: to install on debian 12:

• apt install python3-systemd
• include backend=systemd in the jail.local as above
• and since Debian 12 is well and truly systemd (by default) now: it should have fail2ban log to systemd journal instead of the logfile. (logtarget = SYSTEMD-JOURNAL) in the [DEFAULT] section.

[sebres]

Because we get so many requests related to this issue...

@fail2ban/maintainers, @sylvestre would it make sense to add:

sshd_backend = systemd

to https://github.com/fail2ban/fail2ban/blob/master/config/paths-debian.conf
(and probably few other *_backend used in jail.local, quasi to fulfill #1225 for current debian)

Or would you rather want to add it to debian repository?

From other side, I remember an idea, how we probably can "fix" this and similar issues once and for all - if one would extend backend = auto with the logic: when there is no matches for logpath, and systemd journal is there - switch to systemd backend instead of error. Just I don't know right now how we can avoid a timing issue (for instance if fail2ban starts before some service and this service didn't create a log-file yet, so such a switch to journal monitoring would be wrong.
I must think a bit about that.

[anthonyborriello]

On Debian 12 and Raspberry OS 12
I managed to make fail2ban work writing
backend = systemd in jail.conf
and installing ip6tables to make the sshd work
Anyway log files are not saved!
Every time i reboot the system, the sshd stats return to zero..
Is there a way to make it work?