fail2ban isn't adding repeat offender to the files and getting error on ip.blacklist / fail2ban.action

[rpilman]

Environment:

Fill out and check ([x]) the boxes which apply. If your Fail2Ban version is outdated,
and you can't verify that the issue persists in the recent release, better seek support
from the distribution you obtained Fail2Ban from

• Fail2Ban version (including any possible distribution suffixes):
  Ran service fail2ban --version = Service Ver. 0.91-ubuntu1
• OS, including release name/version:
  Ubuntu 16.04.01 LTS (GNU/Linux 4.4.0-31-genertic x86_64)
• Fail2Ban installed via OS/distribution mechanisms
• You have not applied any additional foreign patches to the codebase
• Some customizations were done to the configuration (provide details below is so)

The issue:

I followed the steps from the website below permanently ban repeat offenders, but checking my log file I'm seeing an error which is:

copy and paste url into address bar and it will work. https://wireflare.com/blog/permanently-ban-repeat-offenders-with-fail2ban/

2016-08-09 04:39:18,098 fail2ban.action         [2573]: ERROR   iptables -w -I f2b-sshd 1 -s 121.18.238.29 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.29' >> /etc/fail2ban/ip.blacklist -- stdout: b''
2016-08-09 04:39:18,098 fail2ban.action         [2573]: ERROR   iptables -w -I f2b-sshd 1 -s 121.18.238.29 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.29' >> /etc/fail2ban/ip.blacklist -- stderr: b'/bin/sh: 3: eco: not found\n'
2016-08-09 04:39:18,100 fail2ban.action         [2573]: ERROR   iptables -w -I f2b-sshd 1 -s 121.18.238.29 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.29' >> /etc/fail2ban/ip.blacklist -- returned 127
2016-08-09 04:39:18,100 fail2ban.action         [2573]: INFO    HINT on 127: "Command not found".  Make sure that all commands in "iptables -w -I f2b-sshd 1 -s 121.18.238.29 -j REJECT --reject-with icmp-port-un$
2016-08-09 04:39:18,100 fail2ban.actions        [2573]: ERROR   Failed to execute ban jail 'sshd' action 'iptables-multiport' info 'CallingMap({'ipjailmatches': <function Actions.__checkBan.<locals>.<lambda> at$
2016-08-09 04:39:18,751 fail2ban.filter         [2573]: INFO    [sshd] Found 121.18.238.29

Steps to reproduce:

Follow the steps in the above link

Expected behavior

To have repeat offender banned and logged to a file ip.blacklist / ip.blacklist.repeatoffender / ip.blacklist.offender

Observed behavior

I'm getting emails showing fail2ban is banning ip addresses, but the repeat offenders aren't getting logged and perma banned.

Any additional information

I'm new to ubuntu and cmd line so I appreciate the any help - I've searched and even on the link above asked the main guy to help but he's stopped answering me.

Configuration, dump and another helpful excerpts

http://drops.articulate.com/Egof

Any customizations done to /etc/fail2ban/ configuration

http://drops.articulate.com/Egof

Relevant parts of /var/log/fail2ban.log file:

preferably obtained while running fail2ban with loglevel = 4

2016-08-09 11:56:16,573 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.22
2016-08-09 11:56:18,649 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.22
2016-08-09 11:56:27,436 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.22
2016-08-09 11:56:27,578 fail2ban.actions [2573]: NOTICE [sshd] Ban 121.18.238.22
2016-08-09 11:56:27,792 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 121.18.238.22 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.22' >> /etc/fail2ban/ip.blacklist -- stdout: b''
2016-08-09 11:56:27,793 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 121.18.238.22 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.22' >> /etc/fail2ban/ip.blacklist -- stderr: b'/bin/sh: 3: eco: not found\n'
2016-08-09 11:56:27,794 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 121.18.238.22 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.22' >> /etc/fail2ban/ip.blacklist -- returned 127
2016-08-09 11:56:27,795 fail2ban.action [2573]: INFO HINT on 127: "Command not found". Make sure that all commands in "iptables -w -I f2b-sshd 1 -s 121.18.238.22 -j REJECT --reject-with icmp-port-unreachable\n\neco '121.18.238.22' >> /etc/fail2ban/ip.blacklist" are in the PATH of fail2ban-server process (grep -a PATH= /proc/pidof -x fail2ban-server/environ). You may want to start "fail2ban-server -f" separately, initiate it with "fail2ban-client reload" in another shell session and observe if additional informative error messages appear in the terminals.
2016-08-09 11:56:27,795 fail2ban.actions [2573]: ERROR Failed to execute ban jail 'sshd' action 'iptables-multiport' info 'CallingMap({'ipjailmatches': <function Actions.__checkBan.. at 0x7f2e7c549c80>, 'ipfailures': <function Actions.__checkBan.. at 0x7f2e7c549d08>, 'time': 1470761787.5782888, 'ipjailfailures': <function Actions.__checkBan.. at 0x7f2e7c549510>, 'failures': 5, 'matches': 'Aug 9 11:56:05 pathfinder sshd[7550]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.22 user=root\nAug 9 11:56:07 pathfinder sshd[7550]: Failed password for root from 121.18.238.22 port 38511 ssh2\nAug 9 11:56:16 pathfinder sshd[7598]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.22 user=root\nAug 9 11:56:18 pathfinder sshd[7598]: Failed password for root from 121.18.238.22 port 32907 ssh2\nAug 9 11:56:27 pathfinder sshd[7667]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.22 user=root', 'ip': '121.18.238.22', 'ipmatches': <function Actions.__checkBan.. at 0x7f2e7c549b70>})': Error banning 121.18.238.22
2016-08-09 11:56:29,357 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.22
2016-08-09 11:58:55,464 fail2ban.actions [2573]: NOTICE [sshd] Unban 221.194.44.227
2016-08-09 12:00:55,779 fail2ban.filter [2573]: INFO [sshd] Found 221.194.44.223
2016-08-09 12:00:58,156 fail2ban.filter [2573]: INFO [sshd] Found 221.194.44.223
2016-08-09 12:01:01,343 fail2ban.filter [2573]: INFO [sshd] Found 221.194.44.223
2016-08-09 12:01:03,732 fail2ban.filter [2573]: INFO [sshd] Found 221.194.44.223
2016-08-09 12:01:08,849 fail2ban.filter [2573]: INFO [sshd] Found 221.194.44.223
2016-08-09 12:01:09,816 fail2ban.actions [2573]: NOTICE [sshd] Ban 221.194.44.223
2016-08-09 12:01:10,032 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 221.194.44.223 -j REJECT --reject-with icmp-port-unreachable

eco '221.194.44.223' >> /etc/fail2ban/ip.blacklist -- stdout: b''
2016-08-09 12:01:10,033 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 221.194.44.223 -j REJECT --reject-with icmp-port-unreachable

eco '221.194.44.223' >> /etc/fail2ban/ip.blacklist -- stderr: b'/bin/sh: 3: eco: not found\n'
2016-08-09 12:01:10,034 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 221.194.44.223 -j REJECT --reject-with icmp-port-unreachable

eco '221.194.44.223' >> /etc/fail2ban/ip.blacklist -- returned 127
2016-08-09 12:01:10,034 fail2ban.action [2573]: INFO HINT on 127: "Command not found". Make sure that all commands in "iptables -w -I f2b-sshd 1 -s 221.194.44.223 -j REJECT --reject-with icmp-port-unreachable\n\neco '221.194.44.223' >> /etc/fail2ban/ip.blacklist" are in the PATH of fail2ban-server process (grep -a PATH= /proc/pidof -x fail2ban-server/environ). You may want to start "fail2ban-server -f" separately, initiate it with "fail2ban-client reload" in another shell session and observe if additional informative error messages appear in the terminals.
2016-08-09 12:01:10,035 fail2ban.actions [2573]: ERROR Failed to execute ban jail 'sshd' action 'iptables-multiport' info 'CallingMap({'ipjailmatches': <function Actions.__checkBan.. at 0x7f2e7c549c80>, 'ipfailures': <function Actions.__checkBan.. at 0x7f2e7c549e18>, 'time': 1470762069.8165634, 'ipjailfailures': <function Actions.__checkBan.. at 0x7f2e7c549d90>, 'failures': 5, 'matches': 'Aug 9 12:00:55 pathfinder sshd[9006]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=221.194.44.223 user=root\nAug 9 12:00:58 pathfinder sshd[9006]: Failed password for root from 221.194.44.223 port 52666 ssh2\nAug 9 12:01:00 pathfinder sshd[9006]: Failed password for root from 221.194.44.223 port 52666 ssh2\nAug 9 12:01:03 pathfinder sshd[9006]: Failed password for root from 221.194.44.223 port 52666 ssh2\nAug 9 12:01:08 pathfinder sshd[9078]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=221.194.44.223 user=root', 'ip': '221.194.44.223', 'ipmatches': <function Actions.__checkBan.. at 0x7f2e7c549d08>})': Error banning 221.194.44.223
2016-08-09 12:01:10,343 fail2ban.filter [2573]: INFO [sshd] Found 221.194.44.223
2016-08-09 12:01:35,656 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.32
2016-08-09 12:01:38,190 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.32
2016-08-09 12:01:45,591 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.32
2016-08-09 12:01:47,496 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.32
2016-08-09 12:01:57,545 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.32
2016-08-09 12:01:58,497 fail2ban.actions [2573]: NOTICE [sshd] Ban 121.18.238.32
2016-08-09 12:01:58,711 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 121.18.238.32 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.32' >> /etc/fail2ban/ip.blacklist -- stdout: b''
2016-08-09 12:01:58,712 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 121.18.238.32 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.32' >> /etc/fail2ban/ip.blacklist -- stderr: b'/bin/sh: 3: eco: not found\n'
2016-08-09 12:01:58,713 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 121.18.238.32 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.32' >> /etc/fail2ban/ip.blacklist -- returned 127
2016-08-09 12:01:58,713 fail2ban.action [2573]: INFO HINT on 127: "Command not found". Make sure that all commands in "iptables -w -I f2b-sshd 1 -s 121.18.238.32 -j REJECT --reject-with icmp-port-unreachable\n\neco '121.18.238.32' >> /etc/fail2ban/ip.blacklist" are in the PATH of fail2ban-server process (grep -a PATH= /proc/pidof -x fail2ban-server/environ). You may want to start "fail2ban-server -f" separately, initiate it with "fail2ban-client reload" in another shell session and observe if additional informative error messages appear in the terminals.
2016-08-09 12:01:58,713 fail2ban.actions [2573]: ERROR Failed to execute ban jail 'sshd' action 'iptables-multiport' info 'CallingMap({'ipjailmatches': <function Actions.__checkBan.. at 0x7f2e7c5498c8>, 'ipfailures': <function Actions.__checkBan.. at 0x7f2e7c549ae8>, 'time': 1470762118.4975297, 'ipjailfailures': <function Actions.__checkBan.. at 0x7f2e7c5499d8>, 'failures': 5, 'matches': 'Aug 9 12:01:35 pathfinder sshd[9234]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.32 user=root\nAug 9 12:01:38 pathfinder sshd[9234]: Failed password for root from 121.18.238.32 port 54479 ssh2\nAug 9 12:01:45 pathfinder sshd[9291]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.32 user=root\nAug 9 12:01:47 pathfinder sshd[9291]: Failed password for root from 121.18.238.32 port 41172 ssh2\nAug 9 12:01:57 pathfinder sshd[9338]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.32 user=root', 'ip': '121.18.238.32', 'ipmatches': <function Actions.__checkBan.. at 0x7f2e7c549b70>})': Error banning 121.18.238.32
2016-08-09 12:01:59,431 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.32
2016-08-09 12:04:07,160 fail2ban.actions [2573]: NOTICE [sshd] Unban 221.194.44.218
2016-08-09 12:09:29,718 fail2ban.actions [2573]: NOTICE [sshd] Unban 121.18.238.29
2016-08-09 12:11:54,893 fail2ban.filter [2573]: INFO [sshd] Found 185.110.132.201
2016-08-09 12:11:57,405 fail2ban.filter [2573]: INFO [sshd] Found 185.110.132.201
2016-08-09 12:14:33,218 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.29
2016-08-09 12:14:34,557 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.29
2016-08-09 12:14:44,256 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.29
2016-08-09 12:14:46,107 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.29
2016-08-09 12:14:48,269 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.29
2016-08-09 12:14:48,287 fail2ban.actions [2573]: NOTICE [sshd] Ban 121.18.238.29
2016-08-09 12:14:48,504 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 121.18.238.29 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.29' >> /etc/fail2ban/ip.blacklist -- stdout: b''
2016-08-09 12:14:48,504 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 121.18.238.29 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.29' >> /etc/fail2ban/ip.blacklist -- stderr: b'/bin/sh: 3: eco: not found\n'
2016-08-09 12:14:48,506 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 121.18.238.29 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.29' >> /etc/fail2ban/ip.blacklist -- returned 127
2016-08-09 12:14:48,506 fail2ban.action [2573]: INFO HINT on 127: "Command not found". Make sure that all commands in "iptables -w -I f2b-sshd 1 -s 121.18.238.29 -j REJECT --reject-with icmp-port-unreachable\n\neco '121.18.238.29' >> /etc/fail2ban/ip.blacklist" are in the PATH of fail2ban-server process (grep -a PATH= /proc/pidof -x fail2ban-server/environ). You may want to start "fail2ban-server -f" separately, initiate it with "fail2ban-client reload" in another shell session and observe if additional informative error messages appear in the terminals.
2016-08-09 12:14:48,506 fail2ban.actions [2573]: ERROR Failed to execute ban jail 'sshd' action 'iptables-multiport' info 'CallingMap({'ipjailmatches': <function Actions.__checkBan.. at 0x7f2e7c549730>, 'ipfailures': <function Actions.__checkBan.. at 0x7f2e7c5499d8>, 'time': 1470762888.2872748, 'ipjailfailures': <function Actions.__checkBan.. at 0x7f2e7c5498c8>, 'failures': 5, 'matches': 'Aug 9 12:14:33 pathfinder sshd[12941]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.29 user=root\nAug 9 12:14:34 pathfinder sshd[12941]: Failed password for root from 121.18.238.29 port 42464 ssh2\nAug 9 12:14:44 pathfinder sshd[12989]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.29 user=root\nAug 9 12:14:46 pathfinder sshd[12989]: Failed password for root from 121.18.238.29 port 58930 ssh2\nAug 9 12:14:48 pathfinder sshd[12989]: Failed password for root from 121.18.238.29 port 58930 ssh2', 'ip': '121.18.238.29', 'ipmatches': <function Actions.__checkBan.. at 0x7f2e7c549d08>})': Error banning 121.18.238.29
2016-08-09 12:18:14,003 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.19
2016-08-09 12:18:16,214 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.19
2016-08-09 12:18:31,445 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.19
2016-08-09 12:18:33,461 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.19
2016-08-09 12:18:38,180 fail2ban.actions [2573]: NOTICE [sshd] Unban 221.194.44.216
2016-08-09 12:18:46,122 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.19
2016-08-09 12:18:46,402 fail2ban.actions [2573]: NOTICE [sshd] Ban 121.18.238.19
2016-08-09 12:18:46,621 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 121.18.238.19 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.19' >> /etc/fail2ban/ip.blacklist -- stdout: b''
2016-08-09 12:18:46,621 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 121.18.238.19 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.19' >> /etc/fail2ban/ip.blacklist -- stderr: b'/bin/sh: 3: eco: not found\n'
2016-08-09 12:18:46,623 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 121.18.238.19 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.19' >> /etc/fail2ban/ip.blacklist -- returned 127
2016-08-09 12:18:46,623 fail2ban.action [2573]: INFO HINT on 127: "Command not found". Make sure that all commands in "iptables -w -I f2b-sshd 1 -s 121.18.238.19 -j REJECT --reject-with icmp-port-unreachable\n\neco '121.18.238.19' >> /etc/fail2ban/ip.blacklist" are in the PATH of fail2ban-server process (grep -a PATH= /proc/pidof -x fail2ban-server/environ). You may want to start "fail2ban-server -f" separately, initiate it with "fail2ban-client reload" in another shell session and observe if additional informative error messages appear in the terminals.
2016-08-09 12:18:46,623 fail2ban.actions [2573]: ERROR Failed to execute ban jail 'sshd' action 'iptables-multiport' info 'CallingMap({'ipjailmatches': <function Actions.__checkBan.. at 0x7f2e7c549d90>, 'ipfailures': <function Actions.__checkBan.. at 0x7f2e7c549730>, 'time': 1470763126.4027042, 'ipjailfailures': <function Actions.__checkBan.. at 0x7f2e7c549b70>, 'failures': 5, 'matches': 'Aug 9 12:18:14 pathfinder sshd[13777]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.19 user=root\nAug 9 12:18:16 pathfinder sshd[13777]: Failed password for root from 121.18.238.19 port 51549 ssh2\nAug 9 12:18:31 pathfinder sshd[13836]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.19 user=root\nAug 9 12:18:33 pathfinder sshd[13836]: Failed password for root from 121.18.238.19 port 43650 ssh2\nAug 9 12:18:46 pathfinder sshd[13914]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.19 user=root', 'ip': '121.18.238.19', 'ipmatches': <function Actions.__checkBan.. at 0x7f2e7c549e18>})': Error banning 121.18.238.19
2016-08-09 12:18:48,530 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.19
2016-08-09 12:26:28,550 fail2ban.actions [2573]: NOTICE [sshd] Unban 121.18.238.22
2016-08-09 12:31:10,028 fail2ban.actions [2573]: NOTICE [sshd] Unban 221.194.44.223
2016-08-09 12:31:59,294 fail2ban.actions [2573]: NOTICE [sshd] Unban 121.18.238.32
2016-08-09 12:44:48,350 fail2ban.actions [2573]: NOTICE [sshd] Unban 121.18.238.29
2016-08-09 12:47:56,760 fail2ban.filter [2573]: INFO [sshd] Found 185.110.132.201
2016-08-09 12:47:56,767 fail2ban.filter [2573]: INFO [sshd] Found 185.110.132.201
2016-08-09 12:47:58,748 fail2ban.filter [2573]: INFO [sshd] Found 185.110.132.201
2016-08-09 12:48:46,831 fail2ban.actions [2573]: NOTICE [sshd] Unban 121.18.238.19
2016-08-09 13:12:19,315 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.22
2016-08-09 13:12:20,809 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.22
2016-08-09 13:12:37,431 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.22
2016-08-09 13:12:39,065 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.22
2016-08-09 13:12:47,579 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.22
2016-08-09 13:12:48,533 fail2ban.actions [2573]: NOTICE [sshd] Ban 121.18.238.22
2016-08-09 13:12:48,751 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 121.18.238.22 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.22' >> /etc/fail2ban/ip.blacklist -- stdout: b''
2016-08-09 13:12:48,752 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 121.18.238.22 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.22' >> /etc/fail2ban/ip.blacklist -- stderr: b'/bin/sh: 3: eco: not found\n'
2016-08-09 13:12:48,752 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 121.18.238.22 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.22' >> /etc/fail2ban/ip.blacklist -- returned 127
2016-08-09 13:12:48,752 fail2ban.action [2573]: INFO HINT on 127: "Command not found". Make sure that all commands in "iptables -w -I f2b-sshd 1 -s 121.18.238.22 -j REJECT --reject-with icmp-port-unreachable\n\neco '121.18.238.22' >> /etc/fail2ban/ip.blacklist" are in the PATH of fail2ban-server process (grep -a PATH= /proc/pidof -x fail2ban-server/environ). You may want to start "fail2ban-server -f" separately, initiate it with "fail2ban-client reload" in another shell session and observe if additional informative error messages appear in the terminals.
2016-08-09 13:12:48,752 fail2ban.actions [2573]: ERROR Failed to execute ban jail 'sshd' action 'iptables-multiport' info 'CallingMap({'ipjailmatches': <function Actions.__checkBan.. at 0x7f2e7c549bf8>, 'ipfailures': <function Actions.__checkBan.. at 0x7f2e7c549b70>, 'time': 1470766368.5335803, 'ipjailfailures': <function Actions.__checkBan.. at 0x7f2e7c549730>, 'failures': 5, 'matches': 'Aug 9 13:12:19 pathfinder sshd[27814]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.22 user=root\nAug 9 13:12:20 pathfinder sshd[27814]: Failed password for root from 121.18.238.22 port 52045 ssh2\nAug 9 13:12:37 pathfinder sshd[27877]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.22 user=root\nAug 9 13:12:39 pathfinder sshd[27877]: Failed password for root from 121.18.238.22 port 41204 ssh2\nAug 9 13:12:47 pathfinder sshd[27952]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.22 user=root', 'ip': '121.18.238.22', 'ipmatches': <function Actions.__checkBan.. at 0x7f2e7c5498c8>})': Error banning 121.18.238.22
2016-08-09 13:12:49,585 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.22

Relevant lines from monitored log files in question:

eco '121.18.238.19' >> /etc/fail2ban/ip.blacklist -- stdout: b''
2016-08-09 12:18:46,621 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 121.18.238.19 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.19' >> /etc/fail2ban/ip.blacklist -- stderr: b'/bin/sh: 3: eco: not found\n'
2016-08-09 12:18:46,623 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 121.18.238.19 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.19' >> /etc/fail2ban/ip.blacklist -- returned 127
2016-08-09 12:18:46,623 fail2ban.action [2573]: INFO HINT on 127: "Command not found". Make sure that all commands in "iptables -w -I f2b-sshd 1 -s 121.18.238.19 -j REJECT --reject-with icmp-port-unreachable\n\neco '121.18.238.19' >> /etc/fail2ban/ip.blacklist" are in the PATH of fail2ban-server process (grep -a PATH= /proc/pidof -x fail2ban-server/environ). You may want to start "fail2ban-server -f" separately, initiate it with "fail2ban-client reload" in another shell session and observe if additional informative error messages appear in the terminals.
2016-08-09 12:18:46,623 fail2ban.actions [2573]: ERROR Failed to execute ban jail 'sshd' action 'iptables-multiport' info 'CallingMap({'ipjailmatches': <function Actions.__checkBan.. at 0x7f2e7c549d90>, 'ipfailures': <function Actions.__checkBan.. at 0x7f2e7c549730>, 'time': 1470763126.4027042, 'ipjailfailures': <function Actions.__checkBan.. at 0x7f2e7c549b70>, 'failures': 5, 'matches': 'Aug 9 12:18:14 pathfinder sshd[13777]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.19 user=root\nAug 9 12:18:16 pathfinder sshd[13777]: Failed password for root from 121.18.238.19 port 51549 ssh2\nAug 9 12:18:31 pathfinder sshd[13836]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.19 user=root\nAug 9 12:18:33 pathfinder sshd[13836]: Failed password for root from 121.18.238.19 port 43650 ssh2\nAug 9 12:18:46 pathfinder sshd[13914]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.19 user=root', 'ip': '121.18.238.19', 'ipmatches': <function Actions.__checkBan.. at 0x7f2e7c549e18>})': Error banning 121.18.238.19
2016-08-09 12:18:48,530 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.19
2016-08-09 12:26:28,550 fail2ban.actions [2573]: NOTICE [sshd] Unban 121.18.238.22
2016-08-09 12:31:10,028 fail2ban.actions [2573]: NOTICE [sshd] Unban 221.194.44.223
2016-08-09 12:31:59,294 fail2ban.actions [2573]: NOTICE [sshd] Unban 121.18.238.32
2016-08-09 12:44:48,350 fail2ban.actions [2573]: NOTICE [sshd] Unban 121.18.238.29
2016-08-09 12:47:56,760 fail2ban.filter [2573]: INFO [sshd] Found 185.110.132.201
2016-08-09 12:47:56,767 fail2ban.filter [2573]: INFO [sshd] Found 185.110.132.201
2016-08-09 12:47:58,748 fail2ban.filter [2573]: INFO [sshd] Found 185.110.132.201
2016-08-09 12:48:46,831 fail2ban.actions [2573]: NOTICE [sshd] Unban 121.18.238.19
2016-08-09 13:12:19,315 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.22
2016-08-09 13:12:20,809 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.22
2016-08-09 13:12:37,431 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.22
2016-08-09 13:12:39,065 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.22
2016-08-09 13:12:47,579 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.22
2016-08-09 13:12:48,533 fail2ban.actions [2573]: NOTICE [sshd] Ban 121.18.238.22
2016-08-09 13:12:48,751 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 121.18.238.22 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.22' >> /etc/fail2ban/ip.blacklist -- stdout: b''
2016-08-09 13:12:48,752 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 121.18.238.22 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.22' >> /etc/fail2ban/ip.blacklist -- stderr: b'/bin/sh: 3: eco: not found\n'
2016-08-09 13:12:48,752 fail2ban.action [2573]: ERROR iptables -w -I f2b-sshd 1 -s 121.18.238.22 -j REJECT --reject-with icmp-port-unreachable

eco '121.18.238.22' >> /etc/fail2ban/ip.blacklist -- returned 127
2016-08-09 13:12:48,752 fail2ban.action [2573]: INFO HINT on 127: "Command not found". Make sure that all commands in "iptables -w -I f2b-sshd 1 -s 121.18.238.22 -j REJECT --reject-with icmp-port-unreachable\n\neco '121.18.238.22' >> /etc/fail2ban/ip.blacklist" are in the PATH of fail2ban-server process (grep -a PATH= /proc/pidof -x fail2ban-server/environ). You may want to start "fail2ban-server -f" separately, initiate it with "fail2ban-client reload" in another shell session and observe if additional informative error messages appear in the terminals.
2016-08-09 13:12:48,752 fail2ban.actions [2573]: ERROR Failed to execute ban jail 'sshd' action 'iptables-multiport' info 'CallingMap({'ipjailmatches': <function Actions.__checkBan.. at 0x7f2e7c549bf8>, 'ipfailures': <function Actions.__checkBan.. at 0x7f2e7c549b70>, 'time': 1470766368.5335803, 'ipjailfailures': <function Actions.__checkBan.. at 0x7f2e7c549730>, 'failures': 5, 'matches': 'Aug 9 13:12:19 pathfinder sshd[27814]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.22 user=root\nAug 9 13:12:20 pathfinder sshd[27814]: Failed password for root from 121.18.238.22 port 52045 ssh2\nAug 9 13:12:37 pathfinder sshd[27877]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.22 user=root\nAug 9 13:12:39 pathfinder sshd[27877]: Failed password for root from 121.18.238.22 port 41204 ssh2\nAug 9 13:12:47 pathfinder sshd[27952]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=121.18.238.22 user=root', 'ip': '121.18.238.22', 'ipmatches': <function Actions.__checkBan.. at 0x7f2e7c5498c8>})': Error banning 121.18.238.22
2016-08-09 13:12:49,585 fail2ban.filter [2573]: INFO [sshd] Found 121.18.238.22

[sebres]

1. Your log says sometimes that eco '...' >> /etc/fail2ban/ip.blacklist is wrong.
   Should it be not echo?
2. It says every time "Command not found" by executing of iptables.
   What did you see, if you start iptables --version from shell? If you really see "command not found", you should specify another default banaction (allowed or available for your system) or even install package iptables.
3. The errors occured by banning of IP in jail "sshd", not (as you described) in jail "repeatoffender". Please provide possibly customization you've done as regards to jail "sshd" or actions that are used there.
4. Your log excerpt was possibly copied from console (so some interresting long lines are truncated, ended with $ characters). Please provide this.

[rpilman]

1. In which file would I need to fix this type of 'eco' to 'echo'
2. after typing in iptables --version i get:

iptables v1.6.0

1. Would you like copies of these files? I can make a copy and ftp in and transfer from the server to my pc?
2. You're right it was a copy and paste. How do I provide you with the full lines? want me to export the error log?

[rpilman]

Here are all the files that I edited to follow the guide from the link above to perform perma ban on repeat offender.

Please download the conf files and log files from this link - http://drops.articulate.com/Egof

fail2ban.txt

[sebres]

1. grep -Rw "eco" /etc/fail2ban/ ?
2. Strange, why it would be not found from fail2ban. It is running under root?
3. You should know which customization you've made or not? It may be a lot of work, to do it instead of you...
4. just attach the log-file via github...

[rpilman]

I did, attached is the log file above and the link from droplr is the fail2ban.conf jail.conf etc.. all the config files

Again, I'm very new to this I was following a gude =[

[rpilman]

I ran the cmd to find eco and it shows this

root@pathfinder:# grep -Rw "eco" /etc/fail2ban/
/etc/fail2ban/action.d/iptables-multiport.conf: eco '' >> /etc/fail2ban/ip.blacklist
root@pathfinder:#

[sebres]

Have you customized iptables-multiport.conf ?
Because I can't remember, that such a bug was ever in our config files.

[rpilman]

Nope the only files i touched were:
/etc/fail2ban/fail2ban.conf
/etc/logrotate.d/fail2ban
/etc/fail2ban/jail.local
/etc/fail2ban/filter.d/repeatoffender.conf
/etc/fail2ban/action.d/repeatoffender.conf
added script - removeoffender.sh

I fixed the eco to echo

[sebres]

Another errors you did:

• you should not copy the whole file jail.conf into jail.local. Only your changes should be made there. Example:

[sshd]
enabled = true

[pam-generic]
enabled = true

[my-jail]
filter=my-filter
action=my-action
enabled = true

• the same as above for /etc/fail2ban/fail2ban.local; and don't modify /etc/fail2ban/fail2ban.conf, because it can be overwritten after package upgrade;

[sebres]

Nope the only files i touched were...

What?

Can you please take a look to https://github.com/fail2ban/fail2ban/blob/master/config/action.d/iptables-multiport.conf and compare it with your "supposedly not changed" iptables-multiport.conf?

It may be few differences, like f2b- instead of fail2ban-, etc.
But I'm sure, we had never such things like eco '...' >> /etc/fail2ban/ip.blacklist.

I don't know what for changes have made the maintainers of Ubuntu package, but it sounds very strange...

[rpilman]

Interesting I checked the master config file of /action.d/iptables-multiport.conf and did see differences..

I also didn't copy the entire file of jail.conf into jail.local =/ I don't even know how to do that.. I just followed the steps in the digitalocean guide on how to secure my newly cloud hosted ubuntu server.

then found a guide which i linked above to perma ban repeat offender... Is there a way i can start over and delete everything related to fail2ban and re-download it onto my server?

[rpilman]

I dont see a fail2ban.local under /etc/fail2ban

just:
fail2ban.conf
ip.blacklist
ip.blocklist.offender
ip.blocklist.repeatoffender
jail.conf
jail.local
paths-common.conf
paths-debian.conf
*removeoffender.sh

and god to honest the only files I touched were the ones I listed as I only followed that guide on how to do perma ban on repeat offenders. I swear ✋ 🆙 ☁️ 🎌 ❤️ 💉 👀

[rpilman]

My ip.blacklist file is not populating with ip addresses, but the ip.blocklist.offender and ip.blocklist.repeatoffender is still empty

[rpilman]

@sebres - Someone stated that the error shown is

"This error has nothing to do with the repeat offender jail that we created. This looks like it might be because you have two jails using the same iptables chains. Try to restart iptables as well to clear the existing chains."

[rpilman]

First off I had no clue how to have my jails using the same iptables chains... I did restart iptables and I don't know how to clear existing chains.

[sebres]

multiple same iptables chains is an aftereffect of missconfiguring...
we had several issues corresponding this, see for example #980 and referenced issues.
But as already said, it is an aftereffect.

The best way to make all clean:

• stop f2b
• restart iptables
• backup your config-dir (/etc/fail2ban), just to compare (don't use it hereafter)
• uninstall f2b
• remove your config-dir (/etc/fail2ban),
• install f2b
• create your own fail2ban.local, jail.local and make there the changes, enable jails etc.
• start f2b

[sebres]

sorry, write from mobile. send too early, I corrected comment above

[rpilman]

no problem, I sent you an email as I dont want to keep commenting on this since it's closed.