Allow fail2ban protection

[DirkHoffmann]

# Ticket imported from Trac

Fail2ban (​http://www.fail2ban.org/) allows to block IPs temporarily in case they try brute-force attacks on login/passwords. That system is quite universal and uses logfile entries.
We would like to use it, but in order to work correctly, it must write the originating IP into the logfile, which is not the case presently. (Only the uid is printed with timestamp and error text.)
Be careful to make sure the log text (analysed with regex) cannot trigger fake alerts and lock the site admins out. It must be safe against injection by trying false logins with UID="Login failed for 'hoffmann' from IP=127.0.0.1" for example.

This request is relevant for Local and LDAP authentication, probably not for SSO (which has its own brute-force hacker filter) and maybe for NICE.

[DirkHoffmann]

From #1530 (for reference).

[DirkHoffmann]

Update: There are two places in the code that should trigger fail2ban:

• Login dialog (see User passwords in clear in "admin logfile" #1530 for reference) - exception for invalid credentials in https://github.com/DirkHoffmann/indico/blob/85529ca/indico/MaKaC/authentication/LDAPAuthentication.py#L224
• Account creation (see Passwords still exposed #1740 reference) - Somewhere in the process method of https://github.com/indico/indico/blob/3a2dba3/indico/MaKaC/webinterface/rh/base.py#L545

[DirkHoffmann]

Trying to figure how I can get the REMOTE_ADDR from within the log routine.
Seems that the good old environment variable can be used request.get_environ("REMOTE_ADDR"). But is that compatible with Windows? Present InDiCo code seems to use it in several places

• /MaKaC/webinterface/session/base.py
• /indico/web/wsgi/indico_wsgi_handler.py

But also req.remote_addr should work - which is used in two or three places elsewhere in the sources.

tbc…

[ThiefMaster]

Both files are obsolete since v1.2.

The correct way to get the IP address is request.remote_addr (with request being imported using from flask import request)

[DirkHoffmann]

OK, thanks for the information. Thus I should have all ingredients in hand to do the job.
On the time-scale of a couple of months, I can commit to implement it. So if someone else starts to work on the issue, let me know, in order to avoid us double work!

[DirkHoffmann]

Sorry, I have to ask for advice again. I have to implement something in v1.1(.2), so I concluded that the logger.exception() for INVALID_CREDENTIALS in LDAPAuthentication.py should be replaced by a lower level like logger.warning().
I see, you came to the same conclusion in the v1.2 sources, but you prefer logger.info() instead. I do not see info messages in /opt/indico/log/indico.log. So where do they go?

[pferreir]

Maybe your debug level is misconfigured (logging.conf)?

[DirkHoffmann]

You are probably right. I was looking in indico.conf, but the level is WARNING in logging.conf.
However it is NOTSET (i.e. "all message levels") specifically for [handler_indico], which should be the one used by LDAPAuthentication, shouldn't it?

[pferreir]

Yep. Indeed. We really have to set a better default. NOTSET is just plain silly 😟

Edit: See #1829

[DirkHoffmann]

I have a working solution for v1.1, working with
failregex = indico.auth.ldap: WARNING - Username: .* - invalid credentials. RemoteAddr: <HOST>$
in /etc/fail2ban/filter.d/indico.conf (Hehe, yet another indico.conf 😈 to confuse users.) Needed that urgently, and v1.2 is not yet installed on our system. (LDAP difficulties.)
I think I will pull-request it to v1.1 for the sake of it. (Shouldn't I?)
The way to do it in v1.2 and later will be completely different (due to refactoring), but if we could keep the messages compatible for fail2ban, that would be great. If you want to change nevertheless: The difficult part of designing a "good" message is to exclude all possible ways to inject a bunch of false error messages (by "clever" choice of user name or URL options) that would lock out sysadmins or anybody else.

[ThiefMaster]

Hm, I don't think it makes lots of sense to merge anything into v1.1 - that version is outdated and I doubt we'll release a v1.1.3 considering that we want people to update to v1.2...

[DirkHoffmann]

v1.1: I know, it is a philosophical question. But what is the alternative? Keeping forever a better version (which is in production) of v1.1 in my own git repository? Maybe it just feels bad for me, as I am not used to decentralised repos. But it feels indeed very bad to have the knowledge of the best solutions spread around the world and not one repo with the ideal, even if outdated, solution.

[ThiefMaster]

Well at some point a version reaches its end of life. It's the same with other software.

Anyway, I believe that is exactly the beauty with Git: You can have your own repo, with a branch based on v1.1.2 that contains your fix so you'll never lose it and if e.g. someone on the mailing list asks for the same thing+version you could point him to that commit.

[DirkHoffmann]

Yes, but it supposes that I will be around when someone is asking or searching loudly. And my repository will not live forever, whereas a central Indico repository is likely to live on until noone uses Indico any more, and even when git will be replaced by a NG code versioning system.

[ThiefMaster]

indico/flask-multipass#2

I don't think it makes much sense to use fail2ban for this; locking people out on the firewall level is not very user-friendly and I don't know any popular web service that does this. Showing a "login attempts exceeded, wait x minutes" is nicer for sure.

[DirkHoffmann]

I don't think it makes much sense to use fail2ban for this; locking people out on the firewall level is not very user-friendly and I don't know any popular web service that does this.

Sorry, but I think you should leave this to other admins. Some sites may have rules to use fail2ban or other mechanisms on all their hosted websites. It is their business to be unpopular or not.

Showing a "login attempts exceeded, wait x minutes" is nicer for sure.

That is fine. Should I understand that it is implemented in v2.0? Otherwise this ticket should remain open, as it is not solved. Actually it was not about fail2ban or not, it was about having a more standard and stable error message format available to do whatever the admin wants to do with it.

[ThiefMaster]

I closed the issue because rate limiting would rather be a feature of the Flask-Multipass extension (also developed by us) than of Indico itself, hence the reference to the issue in that repo.

But I guess logging the IP address on Indico's side in case of a failed authentication attempt would be useful too...

[DirkHoffmann]

Any timescale for this or indico/flask-multipass#2?
This is to estimate, if we transfer our own patch into the latest version again or wait for a quick solution from mainstream.

[ThiefMaster]

Not sure about builtin rate limiting, but improved logging of user-caused login failures (ie only invalid user/credentials, not other errors outside the user's scope) will most likely be in the next 1.9.11 dev version. They'll be logged as a WARNING and include the IP address, which should be sufficient to run fail2ban on them.

[ThiefMaster]

I guess something like this should be suitable for fail2ban logging:

2017-07-24 10:52:00,471  WARNING  74032185fe0d49f0  indico.auth               Invalid credentials (ip=2001:1234::dead:beef, provider=cern-ldap): No such user
2017-07-24 10:52:03,364  WARNING  5d2142643aa742a1  indico.auth               Invalid credentials (ip=2001:1234::dead:beef, provider=cern-ldap): No such user
2017-07-24 10:52:12,991  WARNING  b890a2ab5be644f1  indico.auth               Invalid credentials (ip=2001:1234::dead:beef, provider=dummy): Invalid credentials
2017-07-24 10:52:16,106  WARNING  44e0a4610d174f5d  indico.auth               Invalid credentials (ip=2001:1234::dead:beef, provider=dummy): No such user
2017-07-24 10:52:31,607  WARNING  57a2d34b679f43d0  indico.auth               Invalid credentials (ip=2001:1234::dead:beef, provider=indico): No such user
2017-07-24 10:52:37,890  WARNING  a4989ac204d24e29  indico.auth               Invalid credentials (ip=2001:1234::dead:beef, provider=indico): Invalid credentials

[DirkHoffmann]

Today (1:54am) Adrian wrote:

I guess something like this should be suitable for fail2ban logging: 2017-07-24 10:52:00,471 WARNING 74032185fe0d49f0 indico.multipass Invalid credentials (ip=2001🔢:dead:beef, provider=cern-ldap): No such user 2017-07-24 10:52:03,364 WARNING 5d2142643aa742a1 indico.multipass Invalid credentials (ip=2001🔢:dead:beef, provider=cern-ldap): No such user 2017-07-24 10:52:12,991 WARNING b890a2ab5be644f1 indico.multipass Invalid credentials (ip=2001🔢:dead:beef, provider=dummy): Invalid cre dentials 2017-07-24 10:52:16,106 WARNING 44e0a4610d174f5d indico.multipass Invalid credentials (ip=2001🔢:dead:beef, provider=dummy): No such use r 2017-07-24 10:52:31,607 WARNING 57a2d34b679f43d0 indico.multipass Invalid credentials (ip=2001🔢:dead:beef, provider=indico): No such us er 2017-07-24 10:52:37,890 WARNING a4989ac204d24e29 indico.multipass Invalid credentials (ip=2001🔢:dead:beef, provider=indico): Invalid cr edentials

I am not familiar with this format: `ip=2001🔢:dead:beef`. Are these hexadecimal codings for client::server? Up to now we used the numerical address like `(\d{1,3}).(\d{1,3}){3}`, but I think **fail2ban** would also get along with a DNS host. The [fail2ban manual](http://www.fail2ban.org/wiki/index.php/MANUAL_0_8#Filters) may tell more. Obviously this needs to be tested thoroughly. Because it would be fatal to believe that the protection is active, while it fails for a syntax error or special case. (The manual states that `<HOST>` is IPv6 compliant, if that was your intention of the above examples.) One points worries me more, though: I interprete your example as the successive tests of three providers (cern-ldap, dummy, indico). Does this mean that in case of a valid UID/pwd, there would in general be two failures and one success, thus two warnings issued? Then I am afraid this is not exploitable by **fail2ban**.

[ThiefMaster]

That's an IPv6 address, since this is the future and more and more people use IPv6. If you do not support IPv6 then your regex would probably work. The log message never contains a DNS host.

If you use an IPv4 address the IP in the log message is using the normal IPv4 style:

2017-07-24 11:19:00,420  WARNING  895426c126b240e2  indico.auth               Invalid credentials (ip=128.141.12.34, provider=cern-ldap): No such user

Does this mean that in case of a valid UID/pwd, there would in general be two failures and one success

I simply tried logging in to each provider with an invalid user and with a valid user but an invalid password. So there's exactly one log message for each invalid attempt. You can safely ignore anything after the ip address, so in the regex I would only look at the ... Invalid credentials (ip=..., part.

[DirkHoffmann]

indico.multipass (#1550 (comment)) or indico.auth (#1550 (comment))?

The expression
failregex = WARNING .* indico.auth * Invalid credentials \(ip=<HOST>, provider.*\): (No such user|Invalid credentials)$
would work then.

May I suggest an improvement of the duplicate Invalid credentials sequence?

2017-07-24 10:52:00,471  WARNING  74032185fe0d49f0  indico.auth               No such user (ip=2001:1234::dead:beef, provider=cern-ldap)
2017-07-24 10:52:03,364  WARNING  5d2142643aa742a1  indico.auth               Invalid credentials (ip=2001:1234::dead:beef, provider=cern-ldap)

Or replace the first Invalid credentials with a more generic Authentication failure?

2017-07-24 10:52:12,991  WARNING  b890a2ab5be644f1  indico.auth               Authentication failure (ip=2001:1234::dead:beef, provider=dummy): Invalid credentials

[ThiefMaster]

indico.auth - I changed this after posting my initial comment.

Only match on this part of the message (and the prefix): Invalid credentials (ip=XXX, provider=XXX):.
The part after the : should be considered free-form.

Also, your regex is too general and will most likely allow for people to ban arbitrary IPs since they just need to trigger a warning containing indico.auth x Invalid credentials ...!

[DirkHoffmann]

Today (9:45am) Adrian wrote:

Only match on this part of the message (and the prefix): Invalid credentials
(ip=XXX, provider=XXX): - the last part should be considered free-form.

We prefer to include the line end marker, in order to avoid code injection. (Yes, there can also be malicious attempts to ban another site, or the one of the admin.)

Also, your regex is too general and will most likely allow for people to ban > arbitrary IPs since they just need to trigger a warning containing > indico.auth x Invalid credentials ...!

Well, indico.auth x Invalid credentials would exactly fail (to match). But you got the point. We need to make the regex injection proof. Indeed, the beginning of the line is expected to contain a standard date/timestamp by fail2ban. Thus I expect the regex to be rather failsafe in the proposed form.

As you still seem to have the choice, it would be great, if you could be inspired from the manual for designing your error logs. I am sure they comply to some logfile standard anyway, and other applications would benefit as well from using the or a standard.

[ThiefMaster]

<timestamp>\s+WARNING\s+[0-9a-f]+\s+indico\.auth\s+Invalid credentials \(ip=<ip>, provider=[^) ]+\): .+$

(replace <timestamp> and <ip> with regexps matching these two segments)

If you match this you should be on the safe side - and you still do not need to include anything for the last segment after the :

[ThiefMaster]

Actually, I would recommend you to edit logging.conf to write auth-related messages into a separate file, using a simpler format. For example, you can get this:

2017-07-24 14:35:34,113 WARNING indico.auth Invalid credentials (ip=2001:1458:202:6::100:6, provider=cern-ldap): No such user

by adding this to logging.conf:

[logger_auth]
level=WARNING
handlers=auth
qualname=indico.auth

[handler_auth]
class=FileHandler
args=('/opt/indico/log/auth.log',)
level=WARNING
formatter=authFormatter

[formatter_authFormatter]
format=%(asctime)s %(levelname)s %(name)s %(message)s
datefmt=

Of course you also need to add auth/auth/ authFormatter to the loggers/handlers/formatters keys lists.

With this format you can then use this regex:

^<timestamp> WARNING indico\.auth Invalid credentials \(ip=<ip>, provider=[^) ]+\): .+$

[culpinnis]

Hi everyone,

I am using the following configuration with fail2ban at the moment:

In /opt/indico/etc/logging.yaml

# This is the default indico logging configuration.
# It uses the standard Python dictConfig syntax, but provides a few enhancements:
# - The `filename` of a `logging.FileHandler` is relative to the configured LOG_DIR
# - The SMTP connection data of a `logging.handlers.SMTPHandler` is taken from the
#   indico config (this is disabled if a custom `mailhost` is set).
#   `fromaddr`, `toaddrs` and `subject` are also set to useful default values, but
#   these values may be overridden.
# - The `indico.core.logger.FormattedSubjectSMTPHandler` handler allows formatting
#   the email subject using a standard python format string (e.g. `%(message)s`).
#   Besides that it behaves exactly like the normal `SMTPHandler`, including the
#   Indico-specific default values mentioned above.  It also handles unicode
#   properly so it is not recommended to use the default SMTPHandler as that one
#   may silently drop some log messages which contain non-ascii characters.
# - All log formaters may use `%(request_id)s` to get a random ID unique to each
#   request which makes it easier to grep for all other log entries that belong to
#   a specific request.
# - Setting `append_request_info` to `true` in a formatter will append various
#   information about the current request to the log message.  This will result in
#   a multi-line message and is only recommended for errors; usually in a handler
#   which sends emails.

version: 1

root:
  level: INFO
  handlers: [other]

loggers:
  indico:
    handlers: [indico, email]
  celery:
    handlers: [celery, email, stderr]
  indico.auth:
    handlers: [auth]
    qualname: indico.auth

handlers:
  indico:
    class: logging.FileHandler
    filename: indico.log
    filters: [indico]
    formatter: default
  celery:
    class: logging.FileHandler
    filename: celery.log
    filters: [celery]
    formatter: simple
  other:
    class: logging.FileHandler
    filename: other.log
    filters: [other]
    formatter: simple
    level: WARNING
  stderr:
    class: logging.StreamHandler
    formatter: default
  email:
    class: indico.core.logger.FormattedSubjectSMTPHandler
    formatter: email
    level: ERROR
  auth:
    class: logging.FileHandler
    filename: auth.log
    filters: [indico]
    formatter: simple_auth
    level: WARNING
    #qualname: indico.auth


formatters:
  default:
    format: '%(asctime)s  %(levelname)-7s  %(request_id)s  %(name)-25s %(message)s'
  simple:
    format: '%(asctime)s  %(levelname)-7s  %(name)-25s %(message)s'
  simple_auth:
    format: '%(asctime)s  %(levelname)s  %(name)s %(message)s'
    datefmt: ''
  email:
    append_request_info: true
    format: "%(asctime)s  %(request_id)s  %(name)s - %(levelname)s %(filename)s:%(lineno)d -- %(message)s\n\n"

filters:
  indico:
    name: indico
  celery:
    name: celery
  other:
    (): indico.core.logger.BlacklistFilter
    names: [indico, celery]

In /etc/fail2ban/jails.local:

[indico_pw]
enabled = true
port = http,https
filter = indico_pw
logpath = /opt/indico/log/auth.log
maxretry=3
findtime  = 10m

[indico_nouser]
enabled = true
port = http,https
filter = indico_nouser
logpath = /opt/indico/log/auth.log
maxretry=4
findtime  = 5m
bantime = 1h

And following in /etc/fail2ban/filter.d/indico_pw.conf and indico_nouser.conf

# Fail2Ban configuration file
# Author: Chris Ulpinnis
# $Revision: 0.0.1 $
[Definition]
failregex = ^  WARNING  indico\.auth Invalid credentials \(ip=<HOST>, provider=(indico|ldap)\): Invalid credentials$

# Fail2Ban configuration file
# Author: Chris Ulpinnis
# $Revision: 0.0.1 $
[Definition]
failregex = ^  WARNING  indico\.auth Invalid credentials \(ip=<HOST>, provider=(indico|ldap)\): No such user$

The idea is to block requests especially long that try to figure out valid usernames while the check for wrong passwords should prevent that users lock their ldap account (maybe an additional LDAP rule would be enough).

[ThiefMaster]

3.0 will support rate limiting for failed logins out of the box (see #4817) - and unlike fail2ban it will only block logins, so it's much more user-friendly.