Included filter.d/sshd.conf does not identify failed public keys

[graysky2]

Environment:

Fill out and check ([x]) the boxes which apply. If your Fail2Ban version is outdated,
and you can't verify that the issue persists in the recent release, better seek support
from the distribution you obtained Fail2Ban from

• Fail2Ban version (including any possible distribution suffixes):
• OS, including release name/version:
• Fail2Ban installed via OS/distribution mechanisms
• You have not applied any additional foreign patches to the codebase
• Some customizations were done to the configuration (provide details below is so)

The issue:

Fail2Ban does not trigger a ban when connection attempts are made using an invalid public key. On this box, sshd setup running on port 15001 and only allows keyfiles on Arch Linux using Fail2Ban v0.9.4.

Steps to reproduce

1. Enable the fail2ban systemd service using the below /etc/fail2ban/jail.local
2. Connect from an external IP using an unauthorized key to ssh.

Expected behavior

After 3 attempts, F2B should lock out the offending IP.

Observed behavior

No ban is triggered.

Any customizations done to /etc/fail2ban/ configuration

The only modification I have made is to create /etc/fail2ban/jail.local

[DEFAULT]
ignoreip = 192.168.1.1/24
bantime  = 1200
maxretry = 3
backend  = auto

[sshd]
enabled = true
filter  = sshd
backend = systemd

Relevant parts of /var/log/fail2ban.log file:

preferably obtained while running fail2ban with loglevel = 4

2016-07-05 18:22:29,760 fail2ban.server         [4636]: INFO    Changed logging target to /var/log/fail2ban.log for Fail2ban v0.9.4
2016-07-05 18:22:29,764 fail2ban.database       [4636]: INFO    Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3'
2016-07-05 18:22:29,987 fail2ban.database       [4636]: WARNING New database created. Version '2'
2016-07-05 18:22:29,994 fail2ban.jail           [4636]: INFO    Creating new jail 'sshd'
2016-07-05 18:22:30,127 fail2ban.jail           [4636]: INFO    Jail 'sshd' uses systemd
2016-07-05 18:22:30,412 fail2ban.jail           [4636]: INFO    Initiated 'systemd' backend
2016-07-05 18:22:30,440 fail2ban.filter         [4636]: INFO    Set findtime = 600
2016-07-05 18:22:30,442 fail2ban.filter         [4636]: INFO    Set maxRetry = 3
2016-07-05 18:22:30,444 fail2ban.actions        [4636]: INFO    Set banTime = 1200
2016-07-05 18:22:30,451 fail2ban.filter         [4636]: INFO    Set maxlines = 10
2016-07-05 18:22:31,231 fail2ban.filtersystemd  [4636]: INFO    Added journal match for: '_SYSTEMD_UNIT=sshd.service + _COMM=sshd'
2016-07-05 18:22:31,318 fail2ban.jail           [4636]: INFO    Jail 'sshd' started

Below is the output from journalctl -u sshd while I am pounding the box with an unauthorized key:

Jul 05 18:22:41 mercury sshd[4659]: Failed publickey for graysky from 76.58.20.166 port 37944 ssh2: RSA SHA256:v3dpapGleDaUKf$4V1vKyR9ZyUgjaJAmoCTcb2PLljI
Jul 05 18:22:41 mercury sshd[4659]: Connection closed by 76.58.20.166 port 37944 [preauth]
Jul 05 18:22:41 mercury sshd[4661]: Connection from 76.58.20.166 port 37946 on 192.168.1.110 port 15001
Jul 05 18:22:42 mercury sshd[4661]: Failed publickey for graysky from 76.58.20.166 port 37946 ssh2: RSA SHA256:v3dpapGleDaUKf$4V1vKyR9ZyUgjaJAmoCTcb2PLljI
Jul 05 18:22:42 mercury sshd[4661]: Connection closed by 76.58.20.166 port 37946 [preauth]
Jul 05 18:22:42 mercury sshd[4663]: Connection from 76.58.20.166 port 37948 on 192.168.1.110 port 15001
Jul 05 18:22:42 mercury sshd[4663]: Failed publickey for graysky from 76.58.20.166 port 37948 ssh2: RSA SHA256:v3dpapGleDaUKf$4V1vKyR9ZyUgjaJAmoCTcb2PLljI
Jul 05 18:22:42 mercury sshd[4663]: Connection closed by 76.58.20.166 port 37948 [preauth]
Jul 05 18:22:42 mercury sshd[4665]: Connection from 76.58.20.166 port 37950 on 192.168.1.110 port 15001
Jul 05 18:22:43 mercury sshd[4665]: Failed publickey for graysky from 76.58.20.166 port 37950 ssh2: RSA SHA256:v3dpapGleDaUKf$4V1vKyR9ZyUgjaJAmoCTcb2PLljI
Jul 05 18:22:43 mercury sshd[4665]: Connection closed by 76.58.20.166 port 37950 [preauth]
Jul 05 18:22:43 mercury sshd[4667]: Connection from 76.58.20.166 port 37952 on 192.168.1.110 port 15001
Jul 05 18:22:43 mercury sshd[4667]: Failed publickey for graysky from 76.58.20.166 port 37952 ssh2: RSA SHA256:v3dpapGleDaUKf$4V1vKyR9ZyUgjaJAmoCTcb2PLljI
Jul 05 18:22:43 mercury sshd[4667]: Connection closed by 76.58.20.166 port 37952 [preauth]
Jul 05 18:22:43 mercury sshd[4669]: Connection from 76.58.20.166 port 37954 on 192.168.1.110 port 15001
Jul 05 18:22:44 mercury sshd[4669]: Failed publickey for graysky from 76.58.20.166 port 37954 ssh2: RSA SHA256:v3dpapGleDaUKf$4V1vKyR9ZyUgjaJAmoCTcb2PLljI
Jul 05 18:22:44 mercury sshd[4669]: Connection closed by 76.58.20.166 port 37954 [preauth]

[graysky2]

I seem to have it working with the following line:

^%(__prefix_line)sFailed publickey for .+ from <HOST> port \d+ ssh2: .*$

However, when I run the test script, I get failures but I do not understand why.

% fail2ban-regex systemd-journal /etc/fail2ban/filter.d/sshd-arch.conf

Running tests
=============

Use   failregex filter file : sshd-arch, basedir: /etc/fail2ban
Use         maxlines : 10
Use    journal match : _SYSTEMD_UNIT=sshd.service + _COMM=sshd


Results
=======

Failregex: 18 total
|-  #) [# of hits] regular expression
|   1) [8] ^\s*(<[^.]+\.[^.]+>)?\s*(?:\S+ )?(?:kernel: \[ *\d+\.\d+\] )?(?:@vserver_\S+ )?(?:(?:\[\d+\])?:\s+[\[\(]?sshd(?:\(\S+\))?[\]\)]?:?|[\[\(]?sshd(?:\(\S+\))?[\]\)]?:?(?:\[\d+\])?:?)?\s(?:\[ID \d+ \S+\])?\s*Connection closed by <HOST> port \d+ \[preauth\]$
|   4) [10] ^\s*(<[^.]+\.[^.]+>)?\s*(?:\S+ )?(?:kernel: \[ *\d+\.\d+\] )?(?:@vserver_\S+ )?(?:(?:\[\d+\])?:\s+[\[\(]?sshd(?:\(\S+\))?[\]\)]?:?|[\[\(]?sshd(?:\(\S+\))?[\]\)]?:?(?:\[\d+\])?:?)?\s(?:\[ID \d+ \S+\])?\s*Failed \S+ for .*? from <HOST>(?: port \d*)?(?: ssh\d*)?(: (ruser .*|(\S+ ID \S+ \(serial \d+\) CA )?\S+ (?:[\da-f]{2}:){15}[\da-f]{2}(, client user ".*", client host ".*")?))?\s*$
`-

Ignoreregex: 0 total

Lines: 47 lines, 0 ignored, 18 matched, 29 missed
[processed in 0.30 sec]

Missed line(s): too many to print.  Use --print-all-missed to print all 29 lines

[sebres]

Fixed within #1479

[baptx]

@sebres in which fail2ban version was the fix added? I am using fail2ban 0.10.2-2.1 from Debian 10. After adding LogLevel VERBOSE in /etc/ssh/sshd_config and restarting SSH, I can see Failed publickey for messages in /var/log/auth.log when I use a wrong public key but fail2ban does not block my IP address after several failed attempts.
My file /etc/fail2ban/jail.local just contains this:

[sshd]
enabled = true

[sebres]

I use a wrong public key but fail2ban does not block my IP address after several failed attempts.

I assume your user name is valid, so the messages, that you mean, are not Failed publickey for invalid user, emphasis on invalid.

in which fail2ban version was the fix added?

Shortly, I don't know, because the sshd filter changes continuously. The case is simple - we have to find a balance between:

• don't recognize legitimate users attempting to login using multiple keys with resulting success access as failed attempts (see fail2ban bans legitimate SSH users with multiple public keys #1263);
• don't recognize single attempt producing several messages in same session that cause failure as multiple failures;
• and block everything that did not gain access at end of auth-phase;

I can only say, in the the last version (0.10.5 / 0.11.1) it is fixed "ultimately" (at least unless the logging changes or we get again some new findings).
In this versions it works in this way:

• the message Failed publickey for invalid user ... causes a failure immediately;
• the message Failed publickey for ... (valid user) causes a failure only if (in the same session) it is followed by:
  • some other failure (like Disconnecting: Too many authentication failures)
  • user trying different username (legitimate user trying brute-force on some account and hereafter doing a login that is intended to "forget" failures by success login)
• the messages Failed publickey for tester (valid user) followed by Accepted publickey for tester don't result to a failure unless it doesn't try to change user name during login.
  This looks like

  fail2ban/fail2ban/tests/files/logs/sshd

  Lines 152 to 162 in ec681a3

  	# Ignore tries of legitimate users with multiple public keys (gh-1263):
  	# failJSON: { "match": false }
  	Nov 28 09:16:03 srv sshd[32307]: Failed publickey for git from 192.0.2.1 port 57904 ssh2: ECDSA 0e:ff:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
  	# failJSON: { "match": false }
  	Nov 28 09:16:03 srv sshd[32307]: Failed publickey for git from 192.0.2.1 port 57904 ssh2: RSA 04:bc:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
  	# failJSON: { "match": false }
  	Nov 28 09:16:03 srv sshd[32307]: Postponed publickey for git from 192.0.2.1 port 57904 ssh2 [preauth]
  	# failJSON: { "match": false }
  	Nov 28 09:16:03 srv sshd[32307]: Accepted publickey for git from 192.0.2.1 port 57904 ssh2: DSA 36:48:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
  	# failJSON: { "match": false, "desc": "Should be forgotten by success/accepted public key" }
  	Nov 28 09:16:03 srv sshd[32307]: Connection closed by 192.0.2.1

Also note that even with old (not "fixed") variants of sshd-filter you are "safe", because the key authentication method is extremely stable against brute-force - even 1024-bit RSA key would expect approximately 2.7 * 10¹⁵¹ primes tries what is equal a number with 150 zeros attempts to hit your key with 100% guaranty or approximately 10⁷⁵ to hit it with even chance (50%).
This has almost no chances to succeed, but don't forget that either it will cause Too many authentication failures (so gets banned at some point) or the intruder should make it still slower (what would shrink this microscopic chance to the absolute 0).

[baptx]

@sebres thanks for your reply. Yes, I meant Failed publickey for ... (valid user). So it is not possible to ban an IP address by just using a wrong public key several times on a valid user? Would it not improve security against public / private key bruteforce? I don't see the error message Too many authentication failures in /var/log/auth.log if I try more than 5 times with a wrong public key on a valid user.

[sebres]

Would it not improve security against public / private key bruteforce?

Did you read at all what I wrote above about a chance to success using brute force?

I don't see the error message Too many authentication failures in /var/log/auth.log if I try more than 5 times with a wrong public key on a valid user.

This is normally configurable (and I don't know how many attempts is possible on your system before you get that).

So it is not possible to ban an IP address by just using a wrong public key several times on a valid user?

So I guess you did not read what I wrote...
In current filter implementation (fail2ban versions) it would be recognized as failure, but "retarded", because quasi waiting for Accepted publickey for ... (and if gets that then ignores all them). If you get other message like Connection closed without Accepted, it should cause a failure(s) and then ban after N (maxretry) attempts.

[baptx]

@sebres yes I read that it is nearly impossible to bruteforce but I talked about improving since it is always possible to improve security. For my other question, I asked that because my IP address was not banned even if I have a Connection closed message after each Failed publickey for ... (valid user) message and without having Accepted messages. I guess my current fail2ban version includes the fix since the fix was released in 2016.

[sebres]

I guess my current fail2ban version includes the fix since it was released in 2016.

This fix is probably outdated.
I wrote you above:

1. the sshd filter changes continuously.
2. I can only say, in the the last version (0.10.5 / 0.11.1) it is fixed "ultimately" (at least unless the logging changes or we get again some new findings).

And the logic I described above is valid for latest version only (e. g. last month fix - 9137c7b).

To cut a long story short, it is always possible to provide your own RE's if needed, so:

[sshd]
enabled = true
failregex = ...your-own-RE...
       %(known/failregex)s

For other things install latest version.

[GlebKazantsev]

@sebres yes I read that it is nearly impossible to bruteforce but I talked about improving since it is always possible to improve security. For my other question, I asked that because my IP address was not banned even if I have a Connection closed message after each Failed publickey for ... (valid user) message and without having Accepted messages. I guess my current fail2ban version includes the fix since the fix was released in 2016.

I also ran into the problem that an attempt to connect with the wrong certificate or without it, but with the correct user is not banned. I configured ssh in such a way that there was a login by certificate and login by password.
If I have a certificate, it asks for a password from it, if not, it asks for a user password.
If an attacker tries to log in without a certificate, he receives an invitation to enter the correct login and password with a single attempt, and the password consists of +32 characters with letters, numbers and special characters. That is, those who do not have a certificate will be banned by the wrong login and password.