Portainer malfunctions when treafik has password enabled

[Jannik-Lappe]

---

BUG REPORT INFORMATION

What did you do?

three tests with a docker-compose.yml with traefik and portainer (docker-compose.yml below):

• first test without a password: everything was OK. Portainer worked properly.
• second test with a basic password: Portainer didn't work properly (described below)
• third test with a digest password: same results as test 2

What did you expect to see?

• password input before accessing a side behind traefik (e.g. portainer)
• after passing: normal portainer behavior (accessing local containers,...)

What did you see instead?

• password input before accessing a side behind traefik (e.g. portainer)
• after passing:
  • portainer can't connect to local docker endpoint
  • portainer don't accept configured admin-password

used application versions

docker: 17.12.0-ce
docker-compose: 1.18.0
traefik: 1.5.1
portainer: 1.16.1

What is your environment & configuration (arguments, toml, provider, platform, ...)?

I'm using traefik and portainer with docker on a virtual server from strato with an ubuntu 16.04.3 server installation.

docker-compose.yml:

version: '3.3'

volumes:
  traefik: {}
  portainer: {}

networks:
  traefik-net:
    external:
      name: "$PROXY_NET"

services:
  traefik:
    image: traefik:latest
    restart: always
    networks:
      traefik-net:
    ports:
      - "80:80"
      - "443:443"
    labels:
      traefik.enable: "true"
      traefik.backend: "traefik"
      traefik.port: "8080"
      traefik.frontend.rule: "Host:traefik.$DOMAIN"
      traefik.docker.network: "$PROXY_NET"
      traefik.frontend.entryPoints: "http"
      traefik.frontend.passHostHeader: "true"
      traefik.backend.loadbalancer.stickiness: "true"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - traefik:/etc/traefik
      - ./traefik.toml:/traefik.toml
    command: |
      --web
      --docker.domain=$DOMAIN
      --docker
      --docker.watch
      --docker.exposedbydefault=false

  portainer:
    image: portainer/portainer:latest
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - portainer:/data
    command: |
      --data /data
      --admin-password $PORTAINER_ADMIN_PASSWORD
    networks:
      traefik-net:
    labels:
      traefik.enable: "true"
      traefik.docker.network: "$PROXY_NET"
      traefik.backend: "portainer"
      traefik.port: "9000"
      traefik.frontend.rule: "Host:portainer.$DOMAIN"
      traefik.frontend.entryPoints: "http"
      traefik.frontend.passHostHeader: "true"
      traefik.backend.loadbalancer.stickiness: "true"

traefik.toml:

debug = true
defaultentrypoints = ["http"]

[entryPoints]
  [entryPoints.http]
    address = ":80"
    [entryPoints.http.auth.basic]
      users = [
        "test:$2y$05$e0MzgLzaalyrrCMMoBubHeloYv/lREBZlzWjC9SOQ3sYrSwjeyw5u"
      ]

debug logs

I replaced my domain with {mydomain}.
User is test:test

portainer log

2018/01/31 07:22:55 Creating admin user with password hash $2y$05$e0MzgLzaalyrrCMMoBubHeloYv/lREBZlzWjC9SOQ3sYrSwjeyw5u
2018/01/31 07:22:55 Starting Portainer 1.16.1 on :9000

traefik log

time="2018-01-31T07:22:55Z" level=info msg="Using TOML configuration file //traefik.toml"
time="2018-01-31T07:22:55Z" level=info msg="Traefik version v1.5.1 built on 2018-01-29_02:14:02PM"
time="2018-01-31T07:22:55Z" level=info msg="
Stats collection is disabled.
Help us improve Traefik by turning this feature on :)
More details on: https://docs.traefik.io/basics/#collected-data
"
time="2018-01-31T07:22:55Z" level=debug msg="Global configuration loaded {"LifeCycle":{"RequestAcceptGraceTimeout":0,"GraceTimeOut":0},"GraceTimeOut":0,"Debug":true,"CheckNewVersion":true,"SendAnonymousUsage":false,"AccessLogsFile":"","AccessLog":null,"TraefikLogsFile":"","TraefikLog":null,"LogLevel":"DEBUG","EntryPoints":{"http":{"Network":"","Address":":80","TLS":null,"Redirect":null,"Auth":{"Basic":{"Users":["test:$2y$05$e0MzgLzaalyrrCMMoBubHeloYv/lREBZlzWjC9SOQ3sYrSwjeyw5u"],"UsersFile":""},"Digest":null,"Forward":null,"HeaderField":""},"WhitelistSourceRange":null,"Compress":false,"ProxyProtocol":null,"ForwardedHeaders":{"Insecure":true,"TrustedIPs":null}},"traefik":{"Network":"","Address":":8080","TLS":null,"Redirect":null,"Auth":null,"WhitelistSourceRange":null,"Compress":false,"ProxyProtocol":null,"ForwardedHeaders":{"Insecure":true,"TrustedIPs":null}}},"Cluster":null,"Constraints":[],"ACME":null,"DefaultEntryPoints":["http"],"ProvidersThrottleDuration":2000000000,"MaxIdleConnsPerHost":200,"IdleTimeout":0,"InsecureSkipVerify":false,"RootCAs":null,"Retry":null,"HealthCheck":{"Interval":30000000000},"RespondingTimeouts":null,"ForwardingTimeouts":null,"Web":{"Address":":8080","CertFile":"","KeyFile":"","ReadOnly":false,"Statistics":null,"Metrics":null,"Path":"/","Auth":null,"Debug":false},"Docker":{"Watch":true,"Filename":"","Constraints":null,"Trace":false,"DebugLogGeneratedTemplate":false,"Endpoint":"unix:///var/run/docker.sock","Domain":"{mydomain}","TLS":null,"ExposedByDefault":false,"UseBindPortIP":false,"SwarmMode":false},"File":null,"Marathon":null,"Consul":null,"ConsulCatalog":null,"Etcd":null,"Zookeeper":null,"Boltdb":null,"Kubernetes":null,"Mesos":null,"Eureka":null,"ECS":null,"Rancher":null,"DynamoDB":null,"ServiceFabric":null,"Rest":null,"API":{"EntryPoint":"traefik","Dashboard":true,"Debug":true,"CurrentConfigurations":null,"Statistics":null},"Metrics":null,"Ping":{"EntryPoint":"traefik"}}"
time="2018-01-31T07:22:55Z" level=info msg="Preparing server http &{Network: Address::80 TLS:<nil> Redirect:<nil> Auth:0xc420613e60 WhitelistSourceRange:[] Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc420572060} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2018-01-31T07:22:55Z" level=info msg="Preparing server traefik &{Network: Address::8080 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc420572080} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2018-01-31T07:22:55Z" level=info msg="Starting server on :80"
time="2018-01-31T07:22:55Z" level=info msg="Starting server on :8080"
time="2018-01-31T07:22:55Z" level=info msg="Starting provider *docker.Provider {"Watch":true,"Filename":"","Constraints":null,"Trace":false,"DebugLogGeneratedTemplate":false,"Endpoint":"unix:///var/run/docker.sock","Domain":"{mydomain}","TLS":null,"ExposedByDefault":false,"UseBindPortIP":false,"SwarmMode":false}"
time="2018-01-31T07:22:55Z" level=debug msg="Provider connection established with docker 17.12.0-ce (API 1.35)"
time="2018-01-31T07:22:55Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels"
time="2018-01-31T07:22:55Z" level=debug msg="Could not load traefik.frontend.auth.basic labels"
time="2018-01-31T07:22:55Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels"
time="2018-01-31T07:22:55Z" level=debug msg="Could not load traefik.frontend.auth.basic labels"
time="2018-01-31T07:22:55Z" level=debug msg="Configuration received from provider docker: {"backends":{"backend-portainer":{"servers":{"server-proxy_portainer_1":{"url":"http://172.18.0.2:9000","weight":0}},"loadBalancer":{"method":"wrr","stickiness":{}}},"backend-traefik":{"servers":{"server-proxy_traefik_1":{"url":"http://172.18.0.3:8080","weight":0}},"loadBalancer":{"method":"wrr","stickiness":{}}}},"frontends":{"frontend-Host-portainer-{mydomain}-1":{"entryPoints":["http"],"backend":"backend-portainer","routes":{"route-frontend-Host-portainer-{mydomain}-1":{"rule":"Host:portainer.{mydomain}"}},"passHostHeader":true,"priority":0,"basicAuth":[]},"frontend-Host-traefik-{mydomain}-0":{"entryPoints":["http"],"backend":"backend-traefik","routes":{"route-frontend-Host-traefik-{mydomain}-0":{"rule":"Host:traefik.{mydomain}"}},"passHostHeader":true,"priority":0,"basicAuth":[]}}}"
time="2018-01-31T07:22:55Z" level=debug msg="Creating frontend frontend-Host-portainer-{mydomain}-1"
time="2018-01-31T07:22:55Z" level=debug msg="Wiring frontend frontend-Host-portainer-{mydomain}-1 to entryPoint http"
time="2018-01-31T07:22:55Z" level=debug msg="Creating route route-frontend-Host-portainer-{mydomain}-1 Host:portainer.{mydomain}"
time="2018-01-31T07:22:55Z" level=debug msg="Creating backend backend-portainer"
time="2018-01-31T07:22:55Z" level=debug msg="Creating load-balancer wrr"
time="2018-01-31T07:22:55Z" level=debug msg="Sticky session with cookie _9c5c4"
time="2018-01-31T07:22:55Z" level=debug msg="Creating server server-proxy_portainer_1 at http://172.18.0.2:9000 with weight 0"
time="2018-01-31T07:22:55Z" level=debug msg="Creating frontend frontend-Host-traefik-{mydomain}-0"
time="2018-01-31T07:22:55Z" level=debug msg="Wiring frontend frontend-Host-traefik-{mydomain}-0 to entryPoint http"
time="2018-01-31T07:22:55Z" level=debug msg="Creating route route-frontend-Host-traefik-{mydomain}-0 Host:traefik.{mydomain}"
time="2018-01-31T07:22:55Z" level=debug msg="Creating backend backend-traefik"
time="2018-01-31T07:22:55Z" level=debug msg="Creating load-balancer wrr"
time="2018-01-31T07:22:55Z" level=debug msg="Sticky session with cookie _0e3a4"
time="2018-01-31T07:22:55Z" level=debug msg="Creating server server-proxy_traefik_1 at http://172.18.0.3:8080 with weight 0"
time="2018-01-31T07:22:55Z" level=info msg="Server configuration reloaded on :80"
time="2018-01-31T07:22:55Z" level=info msg="Server configuration reloaded on :8080"
time="2018-01-31T07:23:29Z" level=debug msg="Basic auth failed..."
time="2018-01-31T07:23:35Z" level=debug msg="Basic auth success..."
time="2018-01-31T07:23:35Z" level=info msg="vulcand/oxy/forward/http: Round trip: http://172.18.0.6:80, code: 200, Length: 3317, duration: 3.479522122s"
time="2018-01-31T07:23:40Z" level=debug msg="Basic auth failed..."

[deviantony]

Hi @Jannik-Lappe

Thanks for reporting this.

Please ensure that the Portainer container is removed and another one is created before trying each scenario.

Also, browser cache issues might appear between these scenarios, could you retry using a private navigation session for each one?

Regarding the fact that Portainer do not accept your admin password during authentification, please ensure that you use double quotes and escape each $ chars, see this comment as an example: #1506 (comment)

[Jannik-Lappe]

Hi @deviantony

Thanks for your reply.

Short description of my test arrangement:

• complete removal after a test (Containers, network, volumes,...)
• different project names for each test (t1_, t2_,...)
• two different test systems
• new private sessions on different browsers each test

During my tests without a traefik-password I had no troubles accessing portainer with the admin-password.

Sometimes during my test, I additionally started a blank webserver to test the traefik-password. There was no problem accessing the webserver with the preset password.

[deviantony]

Thanks for the update, unfortunately I do not have the ability to re-create this setup at the moment. Will tag this issue as help wanted as someone in our community might be able to help us here.

[thojkooi]

Portainer uses the Authorization header for passing the JWT.

portainer/api/http/security/bouncer.go

Line 122 in 9239125

tokens, ok := r.Header["Authorization"]

When configuring Traefik to use basic authentication, it will also use the Authorization header, but it will pass the username and password to traefik. I'm not sure if Traefik forwards the Authorization to Portainer, but if it does, Portainer will not accept it because it's not a (valid) JWT.
When you enter the username / password into Portainer and retrieve a JWT, that will be passed through Traefik in the Authorization header. Which will deny the request, as it no longer matches the basic auth user.

I'd say this is expected behaviour with this set-up.

To get this to work, you would need to have Portainer use a different header for passing the JWT. This is currently not supported.

[Jannik-Lappe]

Thank you for your answer.
Then I need to look for another way to secure my server.