traefik:latest fails to auto-detect Docker containers

[dikarel]

Details

• Latest version of Traefik (traefik:latest) fails to auto-detect Docker containers
• Reverting to an older version (traefik:reblochon) fixed it
• Trying to access via domain names that used to work gives back a 404 page
• Debug logs

Specs

• Machine: Ubuntu Xenial
• Docker version 1.12.3, build 6b644ec
• Containers not hosted using swarm mode
• Docker daemon has swarm mode activated

---

[dikarel]

Need to do more investigation to pinpoint what the exact issue is -- will improve description as I discover more info. I'm guessing it has something to do with the new Docker Swarm Mode feature.

[emilevauge]

@dikarel Any DEBUG logs?

[emilevauge]

/cc @vdemeester

[tbouvet]

Same problem with legacy swarm (not swarm mode).
Containers are not auto-detected. I need to stop/start traefik to update configuration.

docker -v

Docker version 1.12.3, build 6b644ec

docker exec -it swarm-agent-master /swarm -v

swarm version 1.2.5 (27968ed)

[dikarel]

Debug logs

Note: if-gitlab is mistakenly listed as disabled

time="2016-11-25T16:13:18Z" level=info msg="Traefik version v1.1.0 built on 2016-11-18_09:20:46AM"
time="2016-11-25T16:13:18Z" level=info msg="Using TOML configuration file /etc/traefik/traefik.toml"
time="2016-11-25T16:13:18Z" level=debug msg="Global configuration loaded {\"GraceTimeOut\":10,\"Debug\":true,\"CheckNewVersion\":true,\"AccessLogsFile\":\"\",\"TraefikLogsFile\":\"\",\"LogLevel\":\"DEBUG\",\"EntryPoints\":{\"http\":{\"Network\":\"\",\"Address\":\":80\",\"TLS\":null,\"Redirect\":{\"EntryPoint\":\"https\",\"Regex\":\"\",\"Replacement\":\"\"},\"Auth\":null,\"Compress\":false},\"https\":{\"Network\":\"\",\"Address\":\":443\",\"TLS\":{\"MinVersion\":\"\",\"CipherSuites\":null,\"Certificates\":null,\"ClientCAFiles\":null},\"Redirect\":null,\"Auth\":null,\"Compress\":false}},\"Cluster\":null,\"Constraints\":[],\"ACME\":{\"Email\":\"darius.karel@company1.co.id\",\"Domains\":[{\"Main\":\"company1.io\",\"SANs\":[\"www.company1.io\",\"gitlab.company1.io\",\"registry.company1.io\",\"npm.company1.io\",\"bower.company1.io\",\"support.company1.io\"]},{\"Main\":\"company2.com\",\"SANs\":[\"api.company2.com\",\"www.company2.com\",\"beta.company2.com\"]}],\"Storage\":\"\",\"StorageFile\":\"/etc/traefik/acme.json\",\"OnDemand\":false,\"OnHostRule\":false,\"CAServer\":\"\",\"EntryPoint\":\"https\"},\"DefaultEntryPoints\":[],\"ProvidersThrottleDuration\":2000000000,\"MaxIdleConnsPerHost\":200,\"InsecureSkipVerify\":false,\"Retry\":{\"Attempts\":3},\"Docker\":{\"Watch\":true,\"Filename\":\"\",\"Constraints\":null,\"Endpoint\":\"unix:///var/run/docker.sock\",\"Domain\":\"company1.io\",\"TLS\":null,\"ExposedByDefault\":false,\"UseBindPortIP\":false,\"SwarmMode\":false},\"File\":{\"Watch\":true,\"Filename\":\"/etc/traefik/traefik.toml\",\"Constraints\":null},\"Web\":{\"Address\":\":8080\",\"CertFile\":\"\",\"KeyFile\":\"\",\"ReadOnly\":true,\"Auth\":null},\"Marathon\":null,\"Consul\":null,\"ConsulCatalog\":null,\"Etcd\":null,\"Zookeeper\":null,\"Boltdb\":null,\"Kubernetes\":null,\"Mesos\":null}"
time="2016-11-25T16:13:18Z" level=info msg="Preparing server http &{Network: Address::80 TLS:<nil> Redirect:0xc4203f1aa0 Auth:<nil> Compress:false}"
time="2016-11-25T16:13:18Z" level=info msg="Preparing server https &{Network: Address::443 TLS:0xc420306f60 Redirect:<nil> Auth:<nil> Compress:false}"
time="2016-11-25T16:13:18Z" level=info msg="Starting server on :80"
time="2016-11-25T16:13:19Z" level=warning msg="ACME.StorageFile is deprecated, use ACME.Storage instead"
time="2016-11-25T16:13:19Z" level=info msg="Loading ACME Account..."
time="2016-11-25T16:13:19Z" level=info msg="Loaded ACME config from store /etc/traefik/acme.json"
time="2016-11-25T16:13:19Z" level=debug msg="Building ACME client..."
time="2016-11-25T16:13:19Z" level=debug msg=AgreeToTOS...
time="2016-11-25T16:13:20Z" level=info msg="Retrieving ACME certificates..."
time="2016-11-25T16:13:20Z" level=info msg="Retrieved ACME certificates"
time="2016-11-25T16:13:20Z" level=debug msg="Testing certificate renew..."
time="2016-11-25T16:13:20Z" level=info msg="Starting server on :443"
time="2016-11-25T16:13:20Z" level=info msg="Starting provider *provider.Docker {\"Watch\":true,\"Filename\":\"\",\"Constraints\":null,\"Endpoint\":\"unix:///var/run/docker.sock\",\"Domain\":\"company1.io\",\"TLS\":null,\"ExposedByDefault\":false,\"UseBindPortIP\":false,\"SwarmMode\":false}"
time="2016-11-25T16:13:20Z" level=info msg="Starting provider *provider.File {\"Watch\":true,\"Filename\":\"/etc/traefik/traefik.toml\",\"Constraints\":null}"
time="2016-11-25T16:13:20Z" level=info msg="Starting provider *main.WebProvider {\"Address\":\":8080\",\"CertFile\":\"\",\"KeyFile\":\"\",\"ReadOnly\":true,\"Auth\":null}"
time="2016-11-25T16:13:20Z" level=debug msg="Docker connection established with docker 1.12.3 (API 1.24)"
time="2016-11-25T16:13:20Z" level=debug msg="Load balancer method '<nil>' for backend company2: Invalid method, using default. Using default wrr."
time="2016-11-25T16:13:20Z" level=debug msg="Load balancer method '<nil>' for backend gi-api: Invalid method, using default. Using default wrr."
time="2016-11-25T16:13:20Z" level=debug msg="Configuration received from provider file: {\"backends\":{\"company2\":{\"servers\":{\"server1\":{\"url\":\"http://192.168.0.78:3000\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"gi-api\":{\"servers\":{\"server1\":{\"url\":\"http://192.168.0.78:3001\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}}},\"frontends\":{\"company2\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"company2\",\"routes\":{\"test1\":{\"rule\":\"Host:www.company2.com,company2.com,beta.company2.com\"}},\"priority\":10},\"gi-api\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"gi-api\",\"routes\":{\"test1\":{\"rule\":\"Host:api.company2.com\"}},\"priority\":9}}}"
time="2016-11-25T16:13:20Z" level=debug msg="Last file config received more than 2s, OK"
time="2016-11-25T16:13:20Z" level=debug msg="Creating frontend company2"
time="2016-11-25T16:13:20Z" level=debug msg="Wiring frontend company2 to entryPoint http"
time="2016-11-25T16:13:20Z" level=debug msg="Creating route test1 Host:www.company2.com,company2.com,beta.company2.com"
time="2016-11-25T16:13:20Z" level=debug msg="Creating entryPoint redirect http -> https : ^(?:https?:\\/\\/)?([\\da-z\\.-]+)(?::\\d+)?(.*)$ -> https://$1:443$2"
time="2016-11-25T16:13:20Z" level=debug msg="Wiring frontend company2 to entryPoint https"
time="2016-11-25T16:13:20Z" level=debug msg="Creating route test1 Host:www.company2.com,company2.com,beta.company2.com"
time="2016-11-25T16:13:20Z" level=debug msg="Creating backend company2"
time="2016-11-25T16:13:20Z" level=debug msg="Creating load-balancer wrr"
time="2016-11-25T16:13:20Z" level=debug msg="Creating server server1 at http://192.168.0.78:3000 with weight 1"
time="2016-11-25T16:13:20Z" level=debug msg="Creating retries max attempts 3"
time="2016-11-25T16:13:20Z" level=debug msg="Creating frontend gi-api"
time="2016-11-25T16:13:20Z" level=debug msg="Wiring frontend gi-api to entryPoint http"
time="2016-11-25T16:13:20Z" level=debug msg="Creating route test1 Host:api.company2.com"
time="2016-11-25T16:13:20Z" level=debug msg="Wiring frontend gi-api to entryPoint https"
time="2016-11-25T16:13:20Z" level=debug msg="Creating route test1 Host:api.company2.com"
time="2016-11-25T16:13:20Z" level=debug msg="Creating backend gi-api"
time="2016-11-25T16:13:20Z" level=debug msg="Creating load-balancer wrr"
time="2016-11-25T16:13:20Z" level=debug msg="Creating server server1 at http://192.168.0.78:3001 with weight 1"
time="2016-11-25T16:13:20Z" level=debug msg="Creating retries max attempts 3"
time="2016-11-25T16:13:20Z" level=info msg="Server configuration reloaded on :443"
time="2016-11-25T16:13:20Z" level=info msg="Server configuration reloaded on :80"
time="2016-11-25T16:13:20Z" level=debug msg="Filtering disabled container /hopeful_bartik"
time="2016-11-25T16:13:20Z" level=debug msg="Filtering container without port and no traefik.port label /gi-website"
time="2016-11-25T16:13:20Z" level=debug msg="Filtering disabled container /if-gitlab"
time="2016-11-25T16:13:20Z" level=debug msg="Filtering disabled container /if-bind.1.2n6ak2dw0lgbe4togal18gnr4"
time="2016-11-25T16:13:20Z" level=debug msg="Filtering disabled container /if-squid.1.43xk5sn3oo8bp2sn4b0cm1aof"
time="2016-11-25T16:13:20Z" level=debug msg="Filtering container without port and no traefik.port label /gi-couch"
time="2016-11-25T16:13:20Z" level=debug msg="Filtering container without port and no traefik.port label /gi-mongo"
time="2016-11-25T16:13:20Z" level=debug msg="Configuration received from provider docker: {}"
time="2016-11-25T16:13:20Z" level=debug msg="Last docker config received less than 2s, waiting..."
time="2016-11-25T16:13:20Z" level=debug msg="Challenge GetCertificate gitlab.company1.io"
time="2016-11-25T16:13:20Z" level=debug msg="ACME got domain cert gitlab.company1.io"
time="2016-11-25T16:13:22Z" level=debug msg="Waited for docker config, OK"
time="2016-11-25T16:13:22Z" level=debug msg="Creating frontend company2"
time="2016-11-25T16:13:22Z" level=debug msg="Wiring frontend company2 to entryPoint http"
time="2016-11-25T16:13:22Z" level=debug msg="Creating route test1 Host:www.company2.com,company2.com,beta.company2.com"
time="2016-11-25T16:13:22Z" level=debug msg="Creating entryPoint redirect http -> https : ^(?:https?:\\/\\/)?([\\da-z\\.-]+)(?::\\d+)?(.*)$ -> https://$1:443$2"
time="2016-11-25T16:13:22Z" level=debug msg="Wiring frontend company2 to entryPoint https"
time="2016-11-25T16:13:22Z" level=debug msg="Creating route test1 Host:www.company2.com,company2.com,beta.company2.com"
time="2016-11-25T16:13:22Z" level=debug msg="Creating backend company2"
time="2016-11-25T16:13:22Z" level=debug msg="Creating load-balancer wrr"
time="2016-11-25T16:13:22Z" level=debug msg="Creating server server1 at http://192.168.0.78:3000 with weight 1"
time="2016-11-25T16:13:22Z" level=debug msg="Creating retries max attempts 3"
time="2016-11-25T16:13:22Z" level=debug msg="Creating frontend gi-api"
time="2016-11-25T16:13:22Z" level=debug msg="Wiring frontend gi-api to entryPoint http"
time="2016-11-25T16:13:22Z" level=debug msg="Creating route test1 Host:api.company2.com"
time="2016-11-25T16:13:22Z" level=debug msg="Wiring frontend gi-api to entryPoint https"
time="2016-11-25T16:13:22Z" level=debug msg="Creating route test1 Host:api.company2.com"
time="2016-11-25T16:13:22Z" level=debug msg="Creating backend gi-api"
time="2016-11-25T16:13:22Z" level=debug msg="Creating load-balancer wrr"
time="2016-11-25T16:13:22Z" level=debug msg="Creating server server1 at http://192.168.0.78:3001 with weight 1"
time="2016-11-25T16:13:22Z" level=debug msg="Creating retries max attempts 3"
time="2016-11-25T16:13:22Z" level=info msg="Server configuration reloaded on :443"
time="2016-11-25T16:13:22Z" level=info msg="Server configuration reloaded on :80"
time="2016-11-25T16:13:26Z" level=debug msg="Challenge GetCertificate gitlab.company1.io"
time="2016-11-25T16:13:26Z" level=debug msg="ACME got domain cert gitlab.company1.io"

GET /api/providers

Note: None of the Docker containers showed up

{
  "docker": {},
  "file": {
    "backends": {
      "company2": {
        "servers": {
          "server1": {
            "url": "http://192.168.0.78:3000",
            "weight": 1
          }
        },
        "loadBalancer": {
          "method": "wrr"
        }
      },
      "gi-api": {
        "servers": {
          "server1": {
            "url": "http://192.168.0.78:3001",
            "weight": 1
          }
        },
        "loadBalancer": {
          "method": "wrr"
        }
      }
    },
    "frontends": {
      "company2": {
        "entryPoints": [
          "http",
          "https"
        ],
        "backend": "company2",
        "routes": {
          "test1": {
            "rule": "Host:www.company2.com,company2.com,beta.company2.com"
          }
        },
        "priority": 10
      },
      "gi-api": {
        "entryPoints": [
          "http",
          "https"
        ],
        "backend": "gi-api",
        "routes": {
          "test1": {
            "rule": "Host:api.company2.com"
          }
        },
        "priority": 9
      }
    }
  }
}

[dikarel]

I've spent some time digging through the source and I think I may have found the problem

What probably happened

1. A couple of months ago I read the docs regarding exposedByDefault which was on Traefik's website; I did not know at the time that this feature hasn't been officially released
2. I set exposedByDefault to false (which did nothing at the time); I assumed this was correct configuration because my setup worked at the time
3. An update to Traefik was released a couple of days ago, which was picked up by my server
4. This time round, the exposedByDefault feature worked and promptly disabled all my containers

I don't have server access atm to test this hypothesis. Will confirm and send another update.

[dikarel]

This turned out to be the case.

I had exposedByDefault set to false in my config which is why my setup suddenly stopped working with traefik:latest. I fixed it by setting traefik.enable to true on my containers.

Reading through the docs, it's not clear how exposedByDefault interplays with the traefik.enable label. #904 is an attempt to address this.

[andypost]

Looks that still does not work

[ldez]

@andypost could you give your configuration and your containers labels?

[Timmmm]

I can't get Traefik to detect my docker container. I have:

$ docker inspect <my container>
      ...
        "Labels": {
            "traefik.backend": "godbolt",
            "traefik.enable": "true",
            "traefik.port": "10240"
        }
      ...

$ cat traefik.toml
...
[docker]
endpoint = "unix:///var/run/docker.sock"
watch = true
exposedByDefault = true

Yet when I run it I get this output:

INFO[2018-06-06T15:21:37+01:00] Starting provider *docker.Provider {"Watch":true,"Filename":"","Constraints":null,"Trace":false,"TemplateVersion":2,"DebugLogGeneratedTemplate":false,"Endpoint":"unix:///var/run/docker.sock","Domain":"","TLS":null,"ExposedByDefault":true,"UseBindPortIP":false,"SwarmMode":false} 
DEBU[2018-06-06T15:21:37+01:00] Provider connection established with docker 18.03.1-ce (API 1.37) 
DEBU[2018-06-06T15:21:37+01:00] Configuration received from provider docker: {} 

And the dashboard says there are 0 docker backends.