Traefik 1.7 does not serve provided TLS certificate when combined with LetsEncrypt and cluster mode

[JamesGuthrie]

Do you want to request a feature or report a bug?

Bug

What did you do?

I configured traefik with a provided certificate for mydomain.com, and to issue a wildcard cert for *.mydomain.com with LetsEncrypt.

What did you expect to see?

I expect to receive the provided certificate when I navigate to mydomain.com, and the LetsEncrypt cert when I navigate to foo.mydomain.com.

What did you see instead?

When I navigate to mydomain.com I receive an SSL error, when I navigate to foo.mydomain.com I get the LetsEncrypt wildcard cert. When I configure traefik without LetsEncrypt, it works.

Output of traefik version: (What version of Traefik are you using?)

Version:      v1.7.0
Codename:     maroilles
Go version:   go1.11
Built:        2018-09-24_09:57:21AM
OS/Arch:      linux/amd64

Configuration

Using cluster mode with etcd, docker swarm config is as follows:

    - "storeconfig"
    - "--accesslog=true"
    - "--accessLog.format=json"
    - "--traefikLog=true"
    - "--traefikLog.format=json"
    - "--defaultentrypoints=http,https"
    - "--entrypoints=Name:http Address::80 Redirect.EntryPoint:https"
    - "--entrypoints=Name:https Address::443 TLS:/run/secrets/mydomain.cert.pem,/run/secrets/mydomain.key.pem TLS.MinVersion:VersionTLS11 TLS.CipherSuites:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA"
    - "--api"
    - "--docker"
    - "--docker.swarmmode=true"
    - "--docker.domain=mydomain.com"
    - "--docker.watch=true"
    - "--acme=true"
    - "--acme.acmelogging=true"
    - "--acme.email=admin@mydomain.com"
    - "--acme.entrypoint=https"
    - "--acme.storage=/traefik/acme/account"
    - "--acme.dnschallenge=true"
    - "--acme.httpchallenge=false"
    - "--acme.dnschallenge.provider=route53"
    - "--acme.domains=*.mydomain.com"
    - "--acme.onhostrule=true"
    - "--etcd"
    - "--etcd.endpoint=etcd:2379"
    - "--etcd.watch=true"
    - "--etcd.prefix=/traefik"
    - "--etcd.useAPIV3=true"

Additional debugging information

I was able to track this down in the code, and come up with two potential fixes, but I'm not sure if either are the "correct" fix.

Fix 1:
Comment out this line of code. Apparently the TLSConfig is captured during CreateClusterConfig here and here. When the Certificates array is later accessed here, it's empty.

Fix 2:
Apply the following change:

diff --git a/acme/acme.go b/acme/acme.go
index d9c90137..c3f98f95 100644
--- a/acme/acme.go
+++ b/acme/acme.go
@@ -131,6 +131,7 @@ func (a *ACME) CreateClusterConfig(leadership *cluster.Leadership, tlsConfig *tl
 
 	tlsConfig.GetCertificate = a.getCertificate
 	a.TLSConfig = tlsConfig
+	a.TLSConfig.BuildNameToCertificate()
 
 	listener := func(object cluster.Object) error {
 		account := object.(*Account)

This way, when the Certificates entry of the TLSConfig is emptied, the NameToCertificate entry is still populated.

[juliens]

Thanks for reporting this issue.

Could you provide us your logs in debug mode, and your entire configuration? (do you use cluster mode?)

[JamesGuthrie]

@juliens I've updated the issue with our config (yes, I'm using cluster mode with etcd and docker swarm providers). I can/could provide a subset of the logs, is there anything that you would like to see in particular? I did not find the debug logs particularly helpful in determining the underlying issue, so I instrumented the code with additional logs in order to try and get to the bottom of the issue.

[stoinov]

I can confirm the same issue manifests in a very basic configuration without swarm setup. Here is the relevant part of my config:

    [[entryPoints.https.tls.certificates]]
      certFile = "tls/cert.pem"
      keyFile = "tls/key.pem"
[acme]
email = "user@domain.com"
storage = "tls/acme.json"
entryPoint = "https"
acmeLogging = true

This used to work before the 1.7 update. DEBUG logs do not return any error messages - just normal initialization.

[loopway]

I have a similar cluster config and can confirm that traefik 1.7.0 does not even serve the default self signed certificate if SNI does not match the letsencrypt configured domains. works with v1.6.6.

[traefiker]

Closed by #3946.