how to ignore an error "Failed during configuration: Have not found any log file for <jail-name> jail"?

[gomin1d]

on my hosts, the availability of services can be optional and even chaotic. how to make fail2ban normally relate to the fact that some logs may not exist? in that case, he could just do nothing

Fail2Ban version v0.11.2-dev

[sebres]

I still not fulfilled #1379, but it is planned that fail2ban is waiting for log-file existence (instead of failing).
At the moment you can:

• either disable such jail;
• or switch to systemd-journal;
• or create such log-file (before fail2ban starting and in logrotate script);
• or create empty file and add its name as last name in logpath like below (so at least one path can be valid):

[jail]
logpath = %(known/logpath)s
          /etc/fail2ban/emptylog

But note that at the moment newly created files are not picked up by fail2ban unless it (or the jail) is restarted.

[ChuckNorrison]

solved in jail.conf with backend = systemd

[Mugane]

None of these solutions is sufficient - fail2ban needs to have a non-blocking fallback when a log file is not found (e.g. default to systemd and log a warning instead of failing to start). Am I the only one who thinks this is a pretty obvious solution?

[sebres]

fail2ban needs to have a non-blocking fallback when a log file is not found

Yes (and #1379 targeting this among other things, just persistently no time to fulfill it).
However it remains questionable approach for many users. Many still prefer that an error occurs if no one log-file is found for the jail.

default to systemd and log a warning instead of failing to start

So in case of some typo by the logpath of some jail it should switch to systemd backend (regardless which backend is specified for the jail)?
I don't think it'd be wise by default.

[Mugane]

Ok I get it but the blocking nature makes it basically impossible to rely on in automated scripts

[sebres]

but the blocking nature makes it basically impossible to rely on in automated scripts

Which kind of scripts? I don't understand the necessity to enable jails for non existing services (no matter per scripts or manually). Also to specify wrong logpath or backend.
As for the automated script, it is surely able to check existence of the logfile, as well as set proper path or else start or enable the jails only for existing services. I don't follow what exactly may be impossible there.

Although I did not say "no" for the RFE, I just for the well thought handling (matter of #1379).

[Mugane]

Basically any modern server configuration script (AWS image template, Linode stackscript, etc). But more importantly I'm not sure that we need to know what kind of scripts... it's outside the scope of what the program should be concerned about. It would make sense to expect that there may be scripts or other programs that do not expect it to be blocking without imposing expectations of deep knowledge of the workings of the blocking nature of fail2ban in order to use it. Basically, you are by decree forcing the absolute termination of everything because of a set of conditions that is not necessarily critical. A warning is sufficient. Anything else is going too far; there is no way for you to know if blocking is the desirable outcome (there may be other more important concerns than the propper logging of fail2ban altogether). If it is still a worry for you then it would be better to have an execution mode/flag that was aggressive & blocked (even if that's the default) and a mode that is non-blocking.

[rocketskatesninja]

Also, when literally all of the documentation tells you 1) do not edit the jail.conf file because fail2ban will default to jail.local and 2) to enable a jail you have to add "enable = true" AND YET fail2ban will not work unless you go in and add "enable = false" to all of the jails that point to non-extant logs in jails.conf is really frustrating.

Just spent 2 days trying to figure out why it's not working, and turns out it was just the documentation. How old is this service?

[gomin1d]

Also, when literally all of the documentation tells you 1) do not edit the jail.conf file because fail2ban will default to jail.local and 2) to enable a jail you have to add "enable = true" AND YET fail2ban will not work unless you go in and add "enable = false" to all of the jails that point to non-extant logs in jails.conf is really frustrating.

Just spent 2 days trying to figure out why it's not working, and turns out it was just the documentation. How old is this service?

@rocketskates-ninja use /etc/fail2ban/emptylog

[jail]
logpath = %(known/logpath)s
          /etc/fail2ban/emptylog

[sebres]

Enabling of all jails is a bad idea - it can simply cause hundred of (unused) threads that would basically monitor nothing.

The issue with "Have not found any log file" also may have several reasons:

1. there is really no logfile to monitor, e. g. it is systemd jouornal, so the jail must be configured with backend = systemd
2. the path deviates for some reason, so it expect logpath = /some/path/to/log
3. there is really no such service and the jail is unneeded.

Adding of empty-log to logpath can circumvent the error by start, but it wouldn't solve the reason actually.

And last but not least, the documentation cannot include everything. Basically the larger the documentation the impenetrable it becomes. One cannot simply write every possible potential issue in the docu and hope the people would find it. This hope is ridiculous.
This is why there is the community and FAQ in wiki (which can be extended by everyone from GH, so one'd save that 2 days trying for others).

[rocketskatesninja]

This is very likely to be caused by the aws template, as while this is the first time I've installed and configured fail2ban, I have set up a few ubuntu-server LAMPs on my home system and have never had as many issues as this aws is giving me. I am going to have to read the documentation and will be anxious to find the section that explains under what conditions the exact opposite of what is in the header comments is true.

When I install fail2ban and try running it out of the box without a jail.local , it fails because it cant find the logs of the first service listed that I'm not using. Maybe it's because of the missing .jail.local.

I cp jail.conf to jail.local. Just for the sake of saving time here imagine I removed all of the jails except [sshd] from jail.local. Should work now, right?

I run it. Again, it cant get passed jail.conf, which is strange because none of the jails have "enable = true" , but they do have a blank line where it could be. So I guess it checks for log files for jails that are not enabled? Or not defining enabled is the same as setting it true?

If I go through and remove all of the jails in jail.conf for services I am not running, it goes right on to jail.local. If I go and set each of the jails in jail.conf to enable = false, (which seems much better than pointing them all at a blank log) , it goes right on to jail.local. Either will work, but the multiple warnings about the jail.conf getting reset on updates is a bit concerning, Also, because I am setting this up for someone else, having to explain all of this to them is a bit concerning.

I was just thinking someone went in and took out all of the "enable = false" for each of the jails in the default config. but looking forward to finding the documentation that explains all this. Maybe a solution would be to put 'enable = false' into all of the jails in the default config, so that users would have to enable the ones that they actually need one by one.

[sebres]

All jails are disabled by default, so I don't follow why one shall put enabled = false into the default section.

[rocketskatesninja]

So to clarify, if I have this jail in my jail.conf :

[openhab-auth]

filter = openhab
banaction = %(banaction_allports)s
logpath = /opt/openhab/logs/request.log

I definitely should not be getting this error message:

May 08 18:13:29 my.server.net fail2ban-server[3588]: 2023-05-08 18:13:29,643 fail2ban [3588]: ERROR Failed during configuration: Have not found any log file for openhab-auth jail

Right?

[sebres]

Normally, no, you'd not.
But is there somewhere in config something enabling the jails by default, then one gets it.
And I guess you have it somewhere.
Just grep /etc/fail2ban for enabled and see where and what enables it.

[rocketskatesninja]

Purging fail2ban and reinstalling it (again) has it working properly. I'm not sure what the difference is, but I'm betting its this aws image. It's really weird. Thanks for your time and help, sorry for being grumpy about it.

[flaviovs]

@sebres, this issue breaks automated provisioning of servers (for example, using Ansible) when the jail must be setup and enabled, but the service hasn't been started and/or created its first log files.

Currently the only reliable alternative is for the provisioning script to create the log file in advance, but this is error prone and far from elegant: the service might change logging in the future, need to pay attention to permission, etc. Although it may work, adding an empty file (or, perhaphs, /dev/null) to make the error to go away is ugly hack, and a waste of resources IMO.

I agree that it is important to alert about a jail for which no log file was found during startup, so what about just turning the incident into a WARNING, and not aborting the whole service or refusing to start? I don't know the internals of fail2ban, but if it checks periodically for new files (e.g. for wildcard logpath), then this could be a trivial change.

Another alternative would be to have a per-jail ensure_log_exists boolean option that defaults to true and that would abort with an error (true) or continue with a warning (false) if no logs were found for that jail.

[sebres]

this issue breaks automated provisioning of servers (for example, using Ansible) when the jail must be setup and enabled

If you can setup jail and/or enable it, why can't you start the jail after service gets started?
To me it looks like an attempt to put your issue on us.

but the service hasn't been started and/or created its first log files.

More or less it is the matter of dependencies or responsibility. In any way if you definitely know that service will write the log there, just create empty log (and set appropriate permissions), exactly like postrotate handler of logrotate would do that.

Currently the only reliable alternative is for the provisioning script to create the log file in advance, but this is error prone and far from elegant: the service might change logging in the future, need to pay attention to permission, etc.

Do I understand correctly:

• for one thing you want no error by start and silently search for a log (eternally?)
• on the other hand by wrong configuration (through Ansible or whatever) the error should be detected.

Don't you see certain contradiction here? :)
How an eternal seek for logpath is better than an empty log?

if it checks periodically for new files (e.g. for wildcard logpath), then this could be a trivial change

Nothing is trivial here. However if you think to know the truth, welcome with PR.

For how many jails and their logpath do you want to allow that? For hundert jails with potentially thousand logs too?
What is about SELinux?..
How long this periodic search shall happen (e. g. even after few hours and missing logs it must continue further or consider it as some error)?
If you found 1 logfile /xxx/a.log for logpath = /xxx/*.log, is it enough and shall you stop trying to find more files (with other words is it possible that 1 minutes later we'd find xxx/b.log).
Although more or less good solution is imaginable for pyinotify backend, I don't think it is good idea for polling backend at all.

[flaviovs]

If you can setup jail and/or enable it, why can't you start the jail after service gets started?

@sebres, let me describe a use case so maybe you understand the issue:

1. A script setups a Linux server with some default jails (e.g. sshd) and starts fail2ban -- everything is fine
2. Later a custom app install script drops an app.conf file to jail.d on the server, and tries to reload fail2ban, which exits and fails to restart because the log file doesn't exist

The thing is: the log file monitored by app.conf is from a web application which is generated only after someone opens the web app and a security issue is detected.

More or less it is the matter of dependencies or responsibility. In any way if you definitely know that service will write the log there, just create empty log (and set appropriate permissions), exactly like postrotate handler of logrotate would do that.

As I mentioned, that can be done. I'm just not sure if it's the most elegant way, since it requires the script dropping the jail.d file to know details about the app it shouldn't be bothered to now. For example, what if a new version of the app requires a specific file mode for the log files? Or the app now runs under a different UID? As you said, it is the matter of dependencies, and I hope we agree that generally the less dependencies (coupling) the better.

Do I understand correctly:

• for one thing you want no error by start and silently search for a log (eternally?)

No.

As I've written, I agree that starting a jail with non-existent log files should never be ignored.

• on the other hand by wrong configuration (through Ansible or whatever) the error should be detected.

I agree, and I hope that the example I gave clarify that it's a reasonable use case, not a "wrong configuration" scenario.

How an eternal seek for logpath is better than an empty log?
Nothing is trivial here. However if you think to know the truth, welcome with PR.

I was under the impression that F2B would always monitor for new files (for supporting wildcards, for example), but just by looking at the package I now see that's not the case, so the solution-path I was thinking may not be as trivial as I thought.

Anyways, I don't mean any disrespect when I say that it could be a trivial change.

All that said, there's also the simpler option of turning the error+abort into a warning. IMHO, issuing a warning about a mis-configured jail, ignoring it, but keep monitoring makes more sense than to abort the daemon altogether, which then stops all jails. If you think it would be an improvement to your software, let me now -- I'm more than happy to give the PR a try.

[sebres]

I hope that the example I gave clarify that it's a reasonable use case, not a "wrong configuration" scenario.

The request was already basically clear to me, I was only answering to the "this is error prone and far from elegant: the service might change logging in the future, need to pay attention to permission, etc.".
This is completely orthogonal to the issue and neither the pre-creation of a file, nor the eternal seek for a file would change something here (it'd be wrong in both cases).

The thing is: the log file monitored by app.conf is from a web application which is generated only after someone opens the web app and a security issue is detected.

This is questionable practice - what is if "security issue is detected" but log-file cannot be created due to permissions error, or because the path is incorrect. But...
Again, this all is valid unless you would either pre-create the empty one, or you'd reload fail2ban config after application would start (if it'd create the log immediately).

Anyway, for the enhancement there is already an issue/FR #1379, and I have an experimental branch working in that way, however with 2 restrictions: it is based on fully experimental (sebres-edition) f2b-similar IDS, and it supports only pyinotify since not implemented for polling (or gamin, which is anyway removed in f2b-1.0/python3). I just don't get time to back-port it here (persistently busy).

Regarding new log-files, if we speaking about certain service configuring fail2ban (like Ansible and co), at the moment there is only two major possibilities to notify fail2ban about new files:

• if it matches logpath (or after you added new line to logpath in jail.local), using command-line|
  fail2ban-client reload ?$jail?
• directly with command-line
  fail2ban-client set $jail addlogpath $logfile

Or both variants using pickle protocol of fail2ban directly to the f2b-socket.