hostsdeny actionunban does not seem to work

[gavsiu]

OS: FreeBSD 12.2-RELEASE-p9
Package: py38-fail2ban-0.11.2_2

Through testing, I realized hostsdeny action adds my IP to the hosts.deny file when I trigger the sshd filter. I try to unban myself using fail2ban-client set sshd unbanip IP.ADDRESS and it says I'm unbanned, but the entry is not removed from the hosts.deny file.

2021-09-26 23:45:28,465 fail2ban.actions        [23946]: NOTICE  [sshd] Ban IP.ADDRESS
2021-09-26 23:45:41,302 fail2ban.actions        [23946]: NOTICE  [sshd] Unban IP.ADDRESS
2021-09-26 23:45:41,308 fail2ban.utils          [23946]: ERROR   8019949f0 -- exec: IP=$(echo "IP.ADDRESS" | /usr/bin/sed 's/[][\.]/\\\0/g') && /usr/bin/sed -i "/^ALL: $IP$/d" /usr/local/etc/fail2ban/hosts.d/hosts.deny
2021-09-26 23:45:41,309 fail2ban.utils          [23946]: ERROR   8019949f0 -- stderr: 'sed: 1: "/usr/local/etc/fail2ban ...": extra characters at the end of l command'
2021-09-26 23:45:41,309 fail2ban.utils          [23946]: ERROR   8019949f0 -- returned 1
2021-09-26 23:45:41,309 fail2ban.actions        [23946]: ERROR   Failed to execute unban jail 'sshd' action 'hostsdeny' info 'ActionInfo({'ip': 'IP.ADDRESS', 'family': 'inet4', 'fid': <function Actions.ActionInfo.<lambda> at 0x8019804c0>, 'raw-ticket': <function Actions.ActionInfo.<lambda> at 0x801980b80>})': Error unbanning IP.ADDRESS

fail2ban-client status sshd will show that my IP is no longer banned from the internal database, but the entry still exists in hosts.deny.

actionunban was not modified and matches the latest here on GitHub.

actionunban = IP=$(echo "<ip_value>" | /usr/bin/sed 's/[][\.]/\\\0/g') && /usr/bin/sed -i "/^<daemon_list>: $IP$/d" <file>

I found this and followed it. I got rid of the error, but the entry was still not removed from hosts.deny file.

[sebres]

See https://stackoverflow.com/a/35620381/7161854

Shortly: FreeBSD sed requires an argument after -i to rename the original file to.

You have to change

fail2ban/config/action.d/hostsdeny.conf

Line 42 in 5093ff8

actionunban = IP=$(echo "<ip_value>" | sed 's/[][\.]/\\\0/g') && sed -i "/^<daemon_list>: $IP$/d" <file>

(or rewrite it in /etc/fail2ban/action.d/hostsdeny.local)

- actionunban = IP=$(echo "<ip_value>" | sed 's/[][\.]/\\\0/g') && sed -i "/^<daemon_list>: $IP$/d" <file>
+ actionunban = IP=$(echo "<ip_value>" | sed 's/[][\.]/\\\0/g') && sed -i .bak "/^<daemon_list>: $IP$/d" <file>

@fail2ban/bsd, @fail2ban/maintainers do we need some customized action here (or can it be generalized using parameter --in-place=.bak to be compatible to all platforms).

[gavsiu]

Yes, the TrueNAS Iocage jail is using csh.

New test:

root@Fail2ban:/tmp/test # echo 'Unban'; IP=$(echo "192.0.2.1" | sed 's/[][\.]/\\\0/g') && sed -i .bak "/^ALL: $IP"'$/d' ./hosts.deny; cat ./hosts.deny
Illegal variable name.

New actionunban:

2021-09-27 19:25:17,716 fail2ban.jail           [53195]: INFO    Jail 'sshd' started
2021-09-27 19:25:23,188 fail2ban.filter         [53195]: INFO    [sshd] Found IP.ADDRESS - 2021-09-27 19:25:23
2021-09-27 19:25:24,893 fail2ban.filter         [53195]: INFO    [sshd] Found IP.ADDRESS - 2021-09-27 19:25:24
2021-09-27 19:25:26,069 fail2ban.filter         [53195]: INFO    [sshd] Found IP.ADDRESS - 2021-09-27 19:25:26
2021-09-27 19:25:26,167 fail2ban.actions        [53195]: NOTICE  [sshd] Ban IP.ADDRESS
2021-09-27 19:25:38,223 fail2ban.actions        [53195]: NOTICE  [sshd] Unban IP.ADDRESS
2021-09-27 19:25:38,229 fail2ban.utils          [53195]: ERROR   8028bd8a0 -- exec: IP=$(echo "IP.ADDRESS" | sed 's/[][\.]/\\\0/g') && sed -i .bak "/^ALL: $IP"'$/d' <file
2021-09-27 19:25:38,229 fail2ban.utils          [53195]: ERROR   8028bd8a0 -- stderr: '/bin/sh: cannot open file: No such file or directory'
2021-09-27 19:25:38,230 fail2ban.utils          [53195]: ERROR   8028bd8a0 -- returned 2
2021-09-27 19:25:38,230 fail2ban.actions        [53195]: ERROR   Failed to execute unban jail 'sshd' action 'hostsdeny' info 'ActionInfo({'ip': 'IP.ADDRESS', 'family': 'inet4', 'fid': <function Actions.ActionInfo.<lambda> at 0x8019814c0>, 'F-*': {'matches': [('', 'Sep 27 19:25:23', ' FTP sshd[53227]: Invalid user user@email.com from IP.ADDRESS port 47778'), 'Sep 27 19:25:24 FTP sshd[53229]: Invalid user user@email.com from IP.ADDRESS port 47780', 'Sep 27 19:25:26 FTP sshd[53231]: Invalid user user@email.com from IP.ADDRESS port 47782'], 'failures': 3, 'mlfid': ' FTP sshd[53227]: ', 'user': 'user@email.com', 'ip4': 'IP.ADDRESS', 'users': {'user@email.com'}}, 'raw-ticket': <function Actions.ActionInfo.<lambda> at 0x801981b80>})': Error unbanning IP.ADDRESS

Current shell:

root@Fail2ban:/tmp/test # echo $SHELL
/bin/csh

Available shells:

root@Fail2ban:/tmp/test # cat /etc/shells
# $FreeBSD: releng/12.2/lib/libc/gen/shells 336840 2018-07-28 20:21:23Z brd $
#
# List of acceptable shells for chpass(1).
# Ftpd will not allow users to connect who are not using
# one of these shells.

/bin/sh
/bin/csh
/bin/tcsh
/usr/local/bin/bash
/usr/local/bin/rbash

[gavsiu]

I changed my shell to bash and ran the first test script you provided.

No errors, but /tmp/test/hosts.deny still had the entry.

Same thing with the hostsdeny.local file and fail2ban-client. No errors, but entry still exists.

[sebres]

Illegal variable name.

You could try to find proper way to escape $ char, I don't use csh and don't have it to hand, so cannot help you here.

2021-09-27 19:25:38,229 fail2ban.utils [53195]: ERROR 8028bd8a0 -- exec: IP=$(echo "IP.ADDRESS" | sed 's/[][\.]/\\\0/g') && sed -i .bak "/^ALL: $IP"'$/d' <file
2021-09-27 19:25:38,229 fail2ban.utils [53195]: ERROR 8028bd8a0 -- stderr: '/bin/sh: cannot open file: No such file or directory'

Please take carefully what do you change in the action - > character after file seems to be missed:

- actionunban = ... <file
+ actionunban = ... <file>

[gavsiu]

Please take carefully what do you change in the action - > character after file seems to be missed:

- actionunban = ... <file
+ actionunban = ... <file>

Sorry, overlooked that. GitHub copy button keeps covering some of the code.

As I said yesterday, running the unban test in bash, there were no errors, but it also didn't remove the entry. I fiddled with it again today and noticed that's because the variable IP is set to 192\00\02\01, which won't match the entry in hosts.deny 192.0.2.1.

I changed the test from

echo 'Unban'; IP=$(echo "192.0.2.1" | sed 's/[][\.]/\\\0/g') && sed -i "" "/^ALL: $IP$/d" ./hosts.deny; cat ./hosts.deny

to

echo 'Unban'; IP=$(echo "192.0.2.1") && sed -i "" "/^ALL: $IP$/d" ./hosts.deny; cat ./hosts.deny

And it removed the entry. Was there a reason for sed 's/[][\.]/\\\0/g' in the first place?

Looking at the hostsdeny.conf file actionban = printf %%b "<daemon_list>: <ip_value>\n" >> <file> adds <daemon_list>: <ip_value>\n to the file, but actionunban changes the <ip_value> before trying to remove it.

I added this to hostsdeny.local and switched back to csh shell and actionunban works now.

actionunban = sed -i "" "/^<daemon_list>: <ip_value>$/d" <file>

[sebres]

because the variable IP is set to 192\00\02\01

How so? It should create 192\.0\.2\.1, PoC:

$ echo "192.0.2.1" | sed 's/[][\.]/\\\0/g'
192\.0\.2\.1

This conversion is necessary - it is called an "escape" for the "." and [] tokens, which (if unescaped) in world of regular expressions would match any or set of characters and not the dot or bracket chars only.
The goal of such escape is to avoid accidental incorrect match on "similar" IP-strings, where . erroneously matches some digit or [::123] would match colon and 1, 2, and 3 chars.

For example see:

$ printf "%s\n%s\nEOF\n" "ALL: 192.0.2.1" "ALL: 192.0.221.5"
ALL: 192.0.2.1
ALL: 192.0.221.5
EOF
$ printf "%s\n%s\nEOF\n" "ALL: 192.0.2.1" "ALL: 192.0.221.5" | sed '/^ALL: 192\.0\.2\.1/d'; # with escape
ALL: 192.0.221.5
EOF
$ printf "%s\n%s\nEOF\n" "ALL: 192.0.2.1" "ALL: 192.0.221.5" | sed '/^ALL: 192.0.2.1/d';    # without escape
EOF

Anyway for proper usage of RE one have to escape the dot and square brackets characters with a backslash.

I don't know why your sed version does not support back reference (by \0), so you have to adjust the escaping RE for your sed version (or use something else like awk, perl, python or whatever)...
If \0 is not working only back reference, you could try sed -E 's/([][\.])/\\\1/g', for instance:

$ echo "192.0.2.1, [::1]" | sed -E 's/([][\.])/\\\1/g'
192\.0\.2\.1, \[::1\]

I have neither the time nor the interest to find it out for every platform and distribution (or sed version) across the world.