Traefik-auth filter requires username but should not #2693

youtous · 2020-04-17T13:36:39Z

Hi,

First of all, I would like to thank you for developing this software since many years 💯

Environment:

Fail2Ban version : Fail2Ban v0.10.2
OS, including release name/version: Debian 10 buster / Linux 4.19.0-8-amd64
Fail2Ban installed via OS/distribution mechanisms
You have not applied any additional foreign patches to the codebase
Some customizations were done to the configuration (provide details below is so)

The issue:

Commit 7cdabdd introduced a new behavior on fail detection of 401 Unauthorized HTTP responses. An username must be present in order to match Failregex.
Unfortunately, traefik does not always provide tested username.

This could lead in brute force attacks not detected.

Steps to reproduce

Setup traefik using a similar docker-compose.yml as provided in traefik-auth.conf filter.
Go to http://traefik.localhost.dv and enter wrong credentials.
cat ./tmp/logs/access.log | grep 401
Test output from logs fail2ban-regex '172.18.0.1 - - [17/Apr/2020:12:49:13 +0000] "GET / HTTP/1.1" 401 17 "-" "curl/7.69.1" 5 "Auth for frontend-Host-traefik-localhost-dv-0" "/" 0ms' /etc/fail2ban/filter.d/traefik-auth.conf

traefik-auth.conf filter failed : Lines: 1 lines, 0 ignored, 0 matched, 1 missed.

Expected behavior

Matching.

Observed behavior

Not matching.

Any additional information

Tested on traefik 1.7 and 2.2, latest.
Can be solved by reverting commit 7cdabdd.

Configuration, dump and another helpful excerpts

Show details

docker-compose.traefik.v1.yml, start it with docker-compose -f docker-compose.traefik.v1.yml up

version: "3.7"

services:
  traefik:
    image: traefik:v1.7
    command: >
      --docker
      --docker.watch
      --docker.exposedbydefault=false
      --loglevel=INFO
      --accesslog
      --accessLog.filePath=/var/log/access.log
      --entryPoints='Name:http Address::80'
      --entryPoints='Name:https Address::443 TLS'
      --api
    ports:
      - 80:80
      - 443:443
    labels:
      - "traefik.enable=true"
      - "traefik.port=8080"
      - "traefik.frontend.rule=Host:traefik.localhost.dv" # adjust the tld
      - "traefik.frontend.auth.basic.users=test:$$apr1$$H6uskkkW$$IgXLP6ewTrSuBkTrqE8wj/"
    volumes:
      - "./tmp/logs/:/var/log"
      - "/var/run/docker.sock:/var/run/docker.sock"

docker-compose.traefik.v2.yml, start it with docker-compose -f docker-compose.traefik.v2.yml up

version: "3.3"

services:

  traefik:
    image: "traefik:latest"
    command:
      - "--log.level=INFO"
      - "--accesslog"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  whoami:
    image: "containous/whoami"
    container_name: "simple-service"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`traefik.localhost.dv`)"
      - "traefik.http.routers.whoami.entrypoints=web"
      - "traefik.http.routers.whoami.middlewares=test-auth@docker"
      - "traefik.http.middlewares.test-auth.basicauth.users=test:$$apr1$$H6uskkkW$$IgXLP6ewTrSuBkTrqE8wj/,test2:$$apr1$$d9hr9HBB$$4HxwgUir3HP4EsggP/QNo0"

Traefik v1 test command: fail2ban-regex '172.18.0.1 - - [17/Apr/2020:12:49:13 +0000] "GET / HTTP/1.1" 401 17 "-" "curl/7.69.1" 5 "Auth for frontend-Host-traefik-localhost-dv-0" "/" 0ms' /etc/fail2ban/filter.d/traefik-auth.conf
Traefik v2 test command: fail2ban-regex '172.18.0.1 - - [17/Apr/2020:13:15:41 +0000] "GET / HTTP/1.1" 401 17 "-" "-" 2 "whoami@docker" "-" 0ms' /etc/fail2ban/filter.d/traefik-alt.conf

The text was updated successfully, but these errors were encountered:

sebres · 2020-04-20T12:18:38Z

Unfortunately, traefik does not always provide tested username.

Sure it does not, but it is the case where a server makes to say a "handshake" (so it requires authentication, so send to user 401 response with auth-methods and authentication doesn't take place (normally occurred with next request. This is a standard behavior of the web-servers (resp. of http protocol).
If you'd enable an empty usernames, fail2ban could catch many false positives on legitimate users initiating the session.

Also note #2286 for implementation details or a discussion about.

This could lead in brute force attacks not detected.

How so if no authentication is possible without to provide a user name?

If you nevertheless want ban such attempts, you can extend the filter to find this as a failure.
For instance:

[traefik-auth]
failregex = %(known/failregex)s
            ^<HOST> \- \S+ \[\] \"(?:GET|POST|HEAD) [^\"]+\" 401\b
enabled = true

youtous · 2020-04-20T13:21:36Z

According to the MDN doc, a 401 response is sent when invalid (or no) credentials are provided by the client (browsers behavior).

If the credentials are valid, the server check if the client is authorized to access the resource (200 if YES, 403/407 otherwise)
If the credentials are wrong : the loop continues: server respond with a 401 response asking client credentials and so on... <-- this is the tested case

Please notice 401 responses does not embed any username: it's only a response sent by the server to the client asking for credentials. (It's used as a handshake by browsers, but it's not an handshake)

The 401 (Unauthorized) status code indicates that the request has not
been applied because it lacks valid authentication credentials for
the target resource.
RFC7235 3.1. 401 Unauthorized

How so if no authentication is possible without to provide a user name?

The username is provided by the client, but not logged by the server in the 401 response.

Test case:

curl https://joe:wrongpassword@consul.heaven-pascal.youtous.dv/imabadguy --insecure

Log result:

192.168.100.1 - - [20/Apr/2020:12:39:58 +0000] "GET / HTTP/2.0" 401 17 "-" "curl/7.69.1" 54 "Auth for frontend-webservice-traefik-consul-consul-leader-webservice" "/" 0ms

In this example, the client sent wrong credentials and traefik did not logged any username.

curl https://youtous:goodpassword@consul.heaven-pascal.youtous.dv/imagoodguy --insecure

Log result:

192.168.100.1 - youtous [20/Apr/2020:12:41:26 +0000] "GET /imagoodguy HTTP/2.0" 404 0 "-" "curl/7.69.1" 56 "webservice-traefik-consul-consul-leader-webservice" "http://10.0.1.2:8500" 2ms

In this example, traefik logged the username when the client is successfully authenticated.

I did not dig into the traefik loggging source code, but it seems traefik is only logging username when the client is fully authenticated. This behavior breaks the current filter and does not block bruteforce attacks.

If you'd enable an empty usernames, fail2ban could catch many false positives on legitimate users initiating the session.

We could adjust maxrety.

sebres · 2020-04-20T13:41:50Z

In this example, the client sent wrong credentials and traefik did not logged any username.

Hmm... Is it backend related issue?.. Does traefik using some backend in your configuration for an authentication?

curl https://joe:wrongpassword@consul.heaven-pascal.youtous.dv/imabadguy --insecure
it seems traefik is only logging username when the client is fully authenticated.

Is user joe available? Can you also try it with an existing user?

@crazy-max can you also confirm this behavior? And how it works at all on your environment?
Is it depending on traefik version?

sebres · 2020-04-20T14:01:39Z

We could adjust maxrety.

As for maxrety, unfortunately it may be a bit worse in case of many http authentication mechanisms (depending on authentication middleware or application context) - in certain cases a web server not necessarily needs a user name after successful authentication process (e. g, if Persistent-Auth: true the client connection is authenticated until it gets closed), and so it has to repeat the authentication phase, so would send 401 again (e. g. keepalive connection closed due timeout etc).

youtous · 2020-04-20T14:17:16Z

Hmm... Is it backend related issue?.. Does traefik using some backend in your configuration for an authentication?

Basic auth middleware provided by traefik (https://docs.traefik.io/middlewares/basicauth/).

Click on "Show details" in my first post, there is a complete configuration for v1 and v2.

Is it depending on traefik version?

Same behavior on both versions.

Is user joe available? Can you also try it with an existing user?

Here is a complete example:

docker-compose.yml run it using docker-compose up

version: "3.3"

services:

  traefik:
    image: "traefik:latest"
    command:
      - "--log.level=INFO"
      - "--accesslog"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
    ports:
      - "80:80"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  whoami:
    image: "containous/whoami"
    container_name: "simple-service"
    labels:
      - "traefik.enable=true"
      - "traefik.port=80"
      - "traefik.http.routers.whoami.rule=Host(`traefik.localhost.dv`)"
      - "traefik.http.routers.whoami.entrypoints=web"
      - "traefik.http.routers.whoami.middlewares=test-auth@docker"
      - "traefik.http.middlewares.test-auth.basicauth.users=test:$$apr1$$H6uskkkW$$IgXLP6ewTrSuBkTrqE8wj/,test2:$$apr1$$d9hr9HBB$$4HxwgUir3HP4EsggP/QNo0"

Existing user, bad password:

$ curl http://test:testdsds@traefik.localhost.dv/user-exists-badpass
> 172.26.0.1 - - [20/Apr/2020:14:09:03 +0000] "GET /user-exists-badpass HTTP/1.1" 401 17 "-" "-" 5 "whoami@docker" "-" 1ms

Existing user, good password:

$ curl http://test:test@traefik.localhost.dv/user-exists-goodpass
> 172.26.0.1 - test [20/Apr/2020:14:07:08 +0000] "GET /user-exists-goodpass HTTP/1.1" 200 396 "-" "-" 3 "whoami@docker" "http://172.26.0.2:80" 9ms

Unknown user

$ curl http://noexist:badpass@traefik.localhost.dv/no-exists
> 172.26.0.1 - - [20/Apr/2020:14:14:49 +0000] "GET /no-exists HTTP/1.1" 401 17 "-" "-" 1 "whoami@docker" "-" 0ms

so it has to repeat the authentication phase, so would send 401 again (e. g. keepalive connection closed due timeout etc).

That's sad :( I think #2102 could help to solve this use case.

sebres · 2020-04-20T14:24:19Z

OK, let's wait for @crazy-max's answer

crazy-max · 2020-04-23T08:23:53Z

@sebres Looks like this is Traefik-related behavior indeed.

@youtous Can you open an issue on Traefik repo about this?

…mal`, `ddos`, `aggressive`) to handle the match of username differently: - `normal`: matches 401 with supplied username only - `ddos`: matches 401 without supplied username only - `aggressive`: matches 401 and any variant (with and without username) closes gh-2693

sebres · 2020-04-23T11:15:27Z

OK, we'll consider this as traefik issue now.

Anyway in-between I extended the filter (6b90ca8) with parameter mode, which can be used to workaround that unless traefik/traefik#6722 is not fixed - set mode to aggressive to match any username (and possibly increase maxrety).

sebres · 2020-04-23T11:16:44Z

Additionally it matches any request method now (e. g. TRACE, PUT, etc pp).

crazy-max · 2020-06-18T14:18:52Z

@youtous Looks like it's ifxed on Traefik side traefik/traefik#6827

youtous changed the title ~~Traefik filter~~ Traefik-auth filter requires username but should not Apr 17, 2020

youtous mentioned this issue Apr 17, 2020

Fix Traefik-auth filter requires username but should not #2694

Closed

7 tasks

sebres added the closed-as-incorrect label Apr 20, 2020

sebres closed this as completed Apr 20, 2020

sebres reopened this Apr 20, 2020

sebres removed the closed-as-incorrect label Apr 20, 2020

youtous mentioned this issue Apr 23, 2020

Basic auth does not log username on failure traefik/traefik#6722

Closed

sebres added the 3rd party issue label Apr 23, 2020

sebres closed this as completed Apr 23, 2020

crazy-max added a commit to crazy-max/docker-fail2ban that referenced this issue Apr 23, 2020

Update Traefik jail example (fail2ban/fail2ban#2693)

f84cea6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Traefik-auth filter requires username but should not #2693

Traefik-auth filter requires username but should not #2693

youtous commented Apr 17, 2020 •

edited

sebres commented Apr 20, 2020

youtous commented Apr 20, 2020

sebres commented Apr 20, 2020

sebres commented Apr 20, 2020

youtous commented Apr 20, 2020 •

edited

sebres commented Apr 20, 2020

crazy-max commented Apr 23, 2020

sebres commented Apr 23, 2020

sebres commented Apr 23, 2020

crazy-max commented Jun 18, 2020

Traefik-auth filter requires username but should not #2693

Traefik-auth filter requires username but should not #2693

Comments

youtous commented Apr 17, 2020 • edited

Environment:

The issue:

Steps to reproduce

Expected behavior

Observed behavior

Any additional information

Configuration, dump and another helpful excerpts

sebres commented Apr 20, 2020

youtous commented Apr 20, 2020

sebres commented Apr 20, 2020

sebres commented Apr 20, 2020

youtous commented Apr 20, 2020 • edited

sebres commented Apr 20, 2020

crazy-max commented Apr 23, 2020

sebres commented Apr 23, 2020

sebres commented Apr 23, 2020

crazy-max commented Jun 18, 2020

youtous commented Apr 17, 2020 •

edited

youtous commented Apr 20, 2020 •

edited