What tags can I use with ignorecommand? Can I get the full line from the Apache Log sent to my ignorecommand script?

[dfreeburn]

Environment:

• Fail2Ban version (including any possible distribution suffixes):
• OS, including release name/version: Ubuntu 16.04
• Fail2Ban installed via OS/distribution mechanisms Fail2Ban 0.10.4 installed from https://github.com/fail2ban/fail2ban/archive/0.10.4.tar.gz
• You have not applied any additional foreign patches to the codebase
• Some customizations were done to the configuration (provide details below is so)

The issue:

According to the ChangeLog for v0.10.4

• ignorecommand extended to use actions-similar replacement (capable to interpolate all possible tags like <ip-host>, <family>, <fid>, F-USER etc.)

We need to send the full line from the Apache Log (e.g. <matches>) to our ignorecommand. We need to determine whether an Apache log line is from a user who has logged into our application, to prevent them from getting banned from too many 404s.

Why are they getting 404's? It's a ticketing system where guest-submitted emails turn into tickets that the logged-in users can respond to. If the guest-submitted email contains an image, it often has a relative link that turns into a 404 when opened in our ticket system.

So, we need non-logged-in guests to get banned for too many 404's, but logged-in users to be ignored. Now that everyone is working from home, keeping an up-to-date list of users' IPs to ignore is proving infeasible.

Our system already gives logged-in users a session ID, so we will get Apache to pass that into the Apache Log, and then Fail2Ban will pass that to our ignorecommand, which will compare it to our current list of active session IDs. If the Session ID from the Apache Log matches an active Session ID, any 404s will be ignored.

Problem is, <matches> and <match> don't appear to be working with ignorecommand.

Which tags can we use? How can we get the full line from the Apache log sent into our ignorecommand script?

Steps to reproduce

This works:

[apache-404]
banaction = iptables-multiport
enabled = true
port = http,https
filter = apache-404
logpath = /var/log/apache2/*access*log
bantime = 3600
findtime = 300
maxretry = 10
ignorecommand = /etc/fail2ban/filter.d/ignorecommands/check_session.sh <ip>

This doesn't:

[apache-404]
banaction = iptables-multiport
enabled = true
port = http,https
filter = apache-404
logpath = /var/log/apache2/*access*log
bantime = 3600
findtime = 300
maxretry = 10
ignorecommand = /etc/fail2ban/filter.d/ignorecommands/check_session.sh <matches>

Expected behavior

Expect to receive the full matched line from the Apache Log as an argument to our ignorecommand

Observed behavior

ignorecommand is never called

Relevant parts of /var/log/fail2ban.log file:

Received this in fail2ban.log:

2020-12-02 13:45:20,033 fail2ban.utils          [9896]: #39-Lev. 7fdec0c04b28 -- exec: /etc/fail2ban/filter.d/ignorecommands/check_session.sh <MATCH>
2020-12-02 13:45:20,033 fail2ban.utils          [9896]: ERROR   7fdec0c04b28 -- stderr: '/bin/sh: 1: Syntax error: end of file unexpected'
2020-12-02 13:45:20,033 fail2ban.utils          [9896]: ERROR   7fdec0c04b28 -- returned 2

[sebres]

The issue is unclear to me:

1. you have to put tag <matches> into the quotes to make a single argument from:

-ignorecommand = /etc/fail2ban/filter.d/ignorecommands/check_session.sh <matches>
+ignorecommand = /etc/fail2ban/filter.d/ignorecommands/check_session.sh "<matches>"

2. The error shows it is occurred as you used tags match and not matches, please provide an error (if happened) for correct tag:

2020-12-02 13:45:20,033 fail2ban.utils [9896]: #39-Lev. 7fdec0c04b28 -- exec: /etc/fail2ban/filter.d/ignorecommands/check_session.sh <MATCH>

What tags can I use with ignorecommand?

Here are all the tags that could be used in 0.10.4 - https://github.com/fail2ban/fail2ban/blob/0.10.4/fail2ban/server/actions.py#L296-L314

And last but not least - you don't need to supply whole matched lines from log to check user is logged.
Normally you can simply exclude it in failregex. Assuming you have 2 lines in log (first should match second should not):

[07/Nov/2020:22:25:01 +0100] 192.0.2.1 - - "POST /12345.php HTTP/1.1" 404 209
[07/Nov/2020:22:25:01 +0100] 192.0.2.1 - username "POST /12345.php HTTP/1.1" 404 209

Changing the regex like here:

-failregex = ^(?:\[\])?\s*<ADDR>\s+\S+\s+\S+\s+"[^"]+"\s+404\s
+failregex = ^(?:\[\])?\s*<ADDR>\s+\S+\s+-\s+"[^"]+"\s+404\s

would match only unauthenticated requests (without username in log).
So you would save a lot of resources to store the ticket in fail2ban failures queue, parse matching lines and check the user via ignorecommand.

Another (fewer optimal) possibility would be to extend filter to extract username using tags <F-USER>\S+</F-USER> and check it in ignorecommand like:

ignorecommand = [ "<F-USER>" != "-" ]