Error in passthrough with TCP routers. Generating wrong certificate

[xmonader]

I've a machine with 1 public IP (traefik tcp proxy) that will serve multiple caddy applications over TLS

Do you want to request a feature or report a bug?

Bug

Did you try using a 1.7.x configuration for the version 2.0?

• Yes
• No

What did you do?

Setup caddy servers

• first one is running on 172.18.0.35:443
• second one is running on 172.18.0.44:443

e.g configuration

https://first.mybot.testbots.grid.tf {
    bind 0.0.0.0
    tls thabeta@codescalers.com {
        ca https://acme-staging-v02.api.letsencrypt.org/directory
    }
...

Setup Traefik

[global]
  debug = true
  logLevel = "DEBUG"
[entryPoints]
  [entryPoints.tcp]
  address = "0.0.0.0:443"

[api]

[providers.file]

[tcp]
[tcp.services]
    [tcp.services.caddy1.LoadBalancer]
      [[tcp.services.caddy1.LoadBalancer.servers]]
        address = "172.18.0.35:443"
    [tcp.services.caddy2.LoadBalancer]
      [[tcp.services.caddy2.LoadBalancer.servers]]
        address = "172.18.0.44:443"

[tcp.routers]
    [tcp.routers.caddy1]
      entrypoints = ["tcp"]
      rule = "HostSNI(`first.mybot.testbots.grid.tf`)"
      service = "caddy1"
    [tcp.routers.caddy1.tls]
       passthrough=true


    [tcp.routers.caddy2]
      entrypoints = ["tcp"]
      rule = "HostSNI(`second.mybot.testbots.grid.tf`)"
      service = "caddy2"
    [tcp.routers.caddy2.tls]
       passthrough=true

What did you expect to see?

In caddy side it should get the certificate without any problems

Activating privacy features... 2019/05/27 12:14:42 [INFO] [second.mybot.testbots.grid.tf] acme: Obtaining bundled SAN certificat
e
2019/05/27 12:14:43 [INFO] [second.mybot.testbots.grid.tf] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/lott
z4tjNgcvXEc37rGraEfp4c_QvlggZfXxii0ea3Y
2019/05/27 12:14:43 [INFO] [second.mybot.testbots.grid.tf] acme: use tls-alpn-01 solver
2019/05/27 12:14:43 [INFO] [second.mybot.testbots.grid.tf] acme: Trying to solve TLS-ALPN-01
2019/05/27 12:14:50 [INFO] [second.mybot.testbots.grid.tf] The server validated our request
2019/05/27 12:14:50 [INFO] [second.mybot.testbots.grid.tf] acme: Validations succeeded; requesting certificates
2019/05/27 12:14:51 [INFO] [second.mybot.testbots.grid.tf] Server responded with a certificate.

What did you see instead?

Got this very weird output

first certificate had names "2994d0b30ee2a6c063d0807f04c5da8e.e5f74b8b9ea9099ff69e5191f302b630.traefik.default, traefik default

Serving HTTPS on port 443
https://second.mybot.testbots.grid.tf


Serving HTTP on port 80
http://second.mybot.testbots.grid.tf

WARNING: File descriptor limit 1024 is too low for production servers. At least 8192 is recommended. Fix with `ulimit -n 8192`.
^Croot@caddy2:/opt/www_incubaid# caddy
Activating privacy features...

Your sites will be served over HTTPS automatically using Let's Encrypt.
By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
  https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Do you agree to the terms? (y/n): y
2019/05/27 11:21:24 [INFO] acme: Registering account for thabeta@codescalers.com
2019/05/27 11:21:24 [INFO] [second.mybot.testbots.grid.tf] acme: Obtaining bundled SAN certificate
2019/05/27 11:21:25 [INFO] [second.mybot.testbots.grid.tf] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/ONUm
dpY6JhXWfWvfJIN2-qUFAiuL5VHUsxEwveIduAg
2019/05/27 11:21:25 [INFO] [second.mybot.testbots.grid.tf] acme: use tls-alpn-01 solver
2019/05/27 11:21:25 [INFO] [second.mybot.testbots.grid.tf] acme: Trying to solve TLS-ALPN-01
2019/05/27 11:21:31 [second.mybot.testbots.grid.tf] failed to obtain certificate: acme: Error -> One or more domains had a probl
em:
[second.mybot.testbots.grid.tf] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Incorrect validation certificate
for tls-alpn-01 challenge. Requested second.mybot.testbots.grid.tf from 37.59.44.168:443. Received 1 certificate(s), first certi
ficate had names "2994d0b30ee2a6c063d0807f04c5da8e.e5f74b8b9ea9099ff69e5191f302b630.traefik.default, traefik default cert", url:

Output of traefik version: (What version of Traefik are you using?)

root@taefik:~# traefik version
Version:      2.0.0-alpha4
Codename:     faisselle
Go version:   go1.12.4
Built:        2019-04-17T11:38:21Z
OS/Arch:      linux/amd64

What is your environment & configuration (arguments, toml, provider, platform, ...)?

Ubuntu 18.04 containers and here's the traefik toml config file

[global]
  debug = true
  logLevel = "DEBUG"
[entryPoints]
  [entryPoints.tcp]
  address = "0.0.0.0:443"

[api]

[providers.file]

[tcp]
[tcp.services]
    [tcp.services.caddy1.LoadBalancer]
      [[tcp.services.caddy1.LoadBalancer.servers]]
        address = "172.18.0.35:443"
    [tcp.services.caddy2.LoadBalancer]
      [[tcp.services.caddy2.LoadBalancer.servers]]
        address = "172.18.0.44:443"

[tcp.routers]
    [tcp.routers.caddy1]
      entrypoints = ["tcp"]
      rule = "HostSNI(`first.mybot.testbots.grid.tf`)"
      service = "caddy1"
    [tcp.routers.caddy1.tls]
       passthrough=true

    [tcp.routers.caddy2]
      entrypoints = ["tcp"]
      rule = "HostSNI(`second.mybot.testbots.grid.tf`)"
      service = "caddy2"
    [tcp.routers.caddy2.tls]
       passthrough=true

If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)

Traefik logs

root@taefik:~# traefik -c config.toml
INFO[2019-05-27T12:14:28Z] Using TOML configuration file /root/config.toml 
INFO[2019-05-27T12:14:28Z] Traefik version 2.0.0-alpha4 built on 2019-04-17T11:38:21Z 
DEBU[2019-05-27T12:14:28Z] Static configuration loaded {"Global":{"Debug":true,"CheckNewVersion":false,"SendAnonymousUsage":null},"ServersTransport":{"InsecureSkipVerify":false,"RootCAs":null,"MaxIdleConnsPerHost":200,"ForwardingTimeouts":null},"EntryPoints":{"tcp":{"Address":"0.0.0.0:443","Transport":{"LifeCycle":{"RequestAcceptGraceTimeout":0,"GraceTimeOut":10000000000},"RespondingTimeouts":{"ReadTimeout":0,"WriteTimeout":0,"IdleTimeout":180000000000}},"ProxyProtocol":null,"ForwardedHeaders":{"Insecure":false,"TrustedIPs":null}},"traefik":{"Address":":8080","Transport":{"LifeCycle":{"RequestAcceptGraceTimeout":0,"GraceTimeOut":10000000000},"RespondingTimeouts":{"ReadTimeout":0,"WriteTimeout":0,"IdleTimeout":180000000000}},"ProxyProtocol":null,"ForwardedHeaders":{"Insecure":false,"TrustedIPs":null}}},"Providers":{"ProvidersThrottleDuration":2000000000,"Docker":null,"File":{"Directory":"","Watch":true,"Filename":"","DebugLogGeneratedTemplate":false,"TraefikFile":"/root/config.toml"},"Marathon":null,"Kubernetes":null,"KubernetesCRD":null,"Rest":null,"Rancher":null},"API":{"EntryPoint":"traefik","Dashboard":true,"Statistics":null,"Middlewares":null},"Metrics":null,"Ping":null,"Log":null,"AccessLog":null,"Tracing":null,"HostResolver":null,"ACME":null} 
ERRO[2019-05-27T12:14:28Z] 
You haven't specify the sendAnonymousUsage option, it will be enable by default. 
INFO[2019-05-27T12:14:28Z] 
Stats collection is enabled.
Many thanks for contributing to Traefik's improvement by allowing us to receive anonymous information from your configuration.
Help us improve Traefik by leaving this feature on :)
More details on: https://docs.traefik.io/basics/#collected-data 
DEBU[2019-05-27T12:14:28Z] No default certificate, generate one         
INFO[2019-05-27T12:14:28Z] Starting provider aggregator.ProviderAggregator {} 
DEBU[2019-05-27T12:14:28Z] Start TCP Server                              entryPointName=tcp
INFO[2019-05-27T12:14:28Z] Starting provider *file.Provider {"Directory":"","Watch":true,"Filename":"","DebugLogGeneratedTemplate":false,"TraefikFile":"/root/config.toml"} 
DEBU[2019-05-27T12:14:28Z] Start TCP Server                              entryPointName=traefik
DEBU[2019-05-27T12:14:28Z] Configuration received from provider file: {"HTTP":{},"TCP":{"routers":{"caddy1":{"entryPoints":["tcp"],"service":"caddy1","rule":"HostSNI(`first.mybot.testbots.grid.tf`)","tls":{"passthrough":true}},"caddy2":{"entryPoints":["tcp"],"service":"caddy2","rule":"HostSNI(`second.mybot.testbots.grid.tf`)","tls":{"passthrough":true}}},"services":{"caddy1":{"loadbalancer":{"servers":[{"address":"172.18.0.35:443","weight":0}]}},"caddy2":{"loadbalancer":{"servers":[{"address":"172.18.0.44:443","weight":0}]}}}},"TLSOptions":{},"TLSStores":{}}  providerName=file
DEBU[2019-05-27T12:14:28Z] No default certificate, generate one         
DEBU[2019-05-27T12:14:28Z] Add route second.mybot.testbots.grid.tf on TCP 
DEBU[2019-05-27T12:14:28Z] Add route first.mybot.testbots.grid.tf on TCP 
DEBU[2019-05-27T11:14:02Z] Add route first.mybot.testbots.grid.tf on TCP                                     
DEBU[2019-05-27T11:14:12Z] Handling connection from 45.244.3.73:37776                                        
DEBU[2019-05-27T11:14:12Z] Load balancer: going back to the first available server                           
DEBU[2019-05-27T11:14:12Z] Handling connection from 45.244.3.73:37778                                        
DEBU[2019-05-27T11:14:15Z] Load balancer: going back to the first available server                           
DEBU[2019-05-27T11:14:15Z] Handling connection from 45.244.3.73:37780                                        
DEBU[2019-05-27T11:15:33Z] Configuration received from provider file: {"HTTP":{},"TCP":{},"TLSOptions":{},"TL
SStores":{}}  providerName=file                                                                              
DEBU[2019-05-27T11:15:33Z] Configuration received from provider file: {"HTTP":{},"TCP":{"routers":{"caddy1":{
"entryPoints":["tcp"],"service":"caddy1","rule":"HostSNI(`first.mybot.testbots.grid.tf`)","tls":{"passthrough
":true}},"caddy2":{"entryPoints":["tcp"],"service":"caddy2","rule":"HostSNI(`second.mybot.testbots.grid.tf`)"
,"tls":{"passthrough":true}}},"services":{"caddy1":{"loadbalancer":{"servers":[{"address":"172.18.0.35:443","
weight":0}]}},"caddy2":{"loadbalancer":{"servers":[{"address":"172.18.0.44:443","weight":0}]}}}},"TLSOptions"
:{},"TLSStores":{}}  providerName=file                                                                       
INFO[2019-05-27T11:15:33Z] Skipping same configuration for provider file  providerName=file                  
DEBU[2019-05-27T11:15:33Z] Configuration received from provider file: {"HTTP":{},"TCP":{"routers":{"caddy1":{
"entryPoints":["tcp"],"service":"caddy1","rule":"HostSNI(`first.mybot.testbots.grid.tf`)","tls":{"passthrough
":true}},"caddy2":{"entryPoints":["tcp"],"service":"caddy2","rule":"HostSNI(`second.mybot.testbots.grid.tf`)"
,"tls":{"passthrough":true}}},"services":{"caddy1":{"loadbalancer":{"servers":[{"address":"172.18.0.35:443","
weight":0}]}},"caddy2":{"loadbalancer":{"servers":[{"address":"172.18.0.44:443","weight":0}]}}}},"TLSOptions"
:{},"TLSStores":{}}  providerName=file                                                                       
INFO[2019-05-27T11:15:33Z] Skipping same configuration for provider file  providerName=file                  
DEBU[2019-05-27T11:15:33Z] Configuration received from provider file: {"HTTP":{},"TCP":{"routers":{"caddy1":{
"entryPoints":["tcp"],"service":"caddy1","rule":"HostSNI(`first.mybot.testbots.grid.tf`)","tls":{"passthrough
":true}},"caddy2":{"entryPoints":["tcp"],"service":"caddy2","rule":"HostSNI(`second.mybot.testbots.grid.tf`)"
,"tls":{"passthrough":true}}},"services":{"caddy1":{"loadbalancer":{"servers":[{"address":"172.18.0.35:443","
weight":0}]}},"caddy2":{"loadbalancer":{"servers":[{"address":"172.18.0.44:443","weight":0}]}}}},"TLSOptions"
:{},"TLSStores":{}}  providerName=file                                                                       
INFO[2019-05-27T11:15:33Z] Skipping same configuration for provider file  providerName=file                  
DEBU[2019-05-27T11:15:33Z] Configuration received from provider file: {"HTTP":{},"TCP":{"routers":{"caddy1":{
"entryPoints":["tcp"],"service":"caddy1","rule":"HostSNI(`first.mybot.testbots.grid.tf`)","tls":{"passthrough
":true}},"caddy2":{"entryPoints":["tcp"],"service":"caddy2","rule":"HostSNI(`second.mybot.testbots.grid.tf`)"
,"tls":{"passthrough":true}}},"services":{"caddy1":{"loadbalancer":{"servers":[{"address":"172.18.0.35:443","
weight":0}]}},"caddy2":{"loadbalancer":{"servers":[{"address":"172.18.0.44:443","weight":0}]}}}},"TLSOptions"
:{},"TLSStores":{}}  providerName=file                                                                       
INFO[2019-05-27T11:15:33Z] Skipping same configuration for provider file  providerName=file                  
DEBU[2019-05-27T11:15:33Z] No default certificate, generate one                                              
DEBU[2019-05-27T11:19:38Z] Serving default certificate for request: "first.mybot.testbots.grid.tf"           
DEBU[2019-05-27T11:19:38Z] Serving default certificate for request: "first.mybot.testbots.grid.tf"  

INFO[2019-05-27T11:15:33Z] Skipping same configuration for provider file  providerName=file                  
DEBU[2019-05-27T11:15:33Z] No default certificate, generate one                                              
DEBU[2019-05-27T11:19:38Z] Serving default certificate for request: "first.mybot.testbots.grid.tf"           
DEBU[2019-05-27T11:19:38Z] Serving default certificate for request: "first.mybot.testbots.grid.tf"           
2019/05/27 11:19:38 server.go:3012: http: TLS handshake error from 45.244.3.73:37850: remote error: tls: ille
gal parameter
DEBU[2019-05-27T11:19:38Z] Serving default certificate for request: "first.mybot.testbots.grid.tf"
2019/05/27 11:19:38 server.go:3012: http: TLS handshake error from 45.244.3.73:37852: remote error: tls: ille
gal parameter
2019/05/27 11:19:38 server.go:3012: http: TLS handshake error from 45.244.3.73:37854: remote error: tls: unkn
own certificate
DEBU[2019-05-27T11:19:41Z] Serving default certificate for request: "first.mybot.testbots.grid.tf"
DEBU[2019-05-27T11:19:42Z] Serving default certificate for request: "first.mybot.testbots.grid.tf"
2019/05/27 11:19:42 server.go:3012: http: TLS handshake error from 45.244.3.73:37860: remote error: tls: ille
gal parameter
DEBU[2019-05-27T11:19:42Z] Serving default certificate for request: "first.mybot.testbots.grid.tf"
DEBU[2019-05-27T11:19:42Z] Serving default certificate for request: "first.mybot.testbots.grid.tf"
2019/05/27 11:19:42 server.go:3012: http: TLS handshake error from 45.244.3.73:37864: remote error: tls: unkn
own certificate
2019/05/27 11:19:42 server.go:3012: http: TLS handshake error from 45.244.3.73:37862: remote error: tls: ille
gal parameter
DEBU[2019-05-27T11:19:42Z] Serving default certificate for request: "first.mybot.testbots.grid.tf"
DEBU[2019-05-27T11:19:45Z] Serving default certificate for request: "first.mybot.testbots.grid.tf"
2019/05/27 11:19:45 server.go:3012: http: TLS handshake error from 45.244.3.73:37868: remote error: tls: ille
gal parameter
DEBU[2019-05-27T11:19:45Z] Serving default certificate for request: "first.mybot.testbots.grid.tf"
2019/05/27 11:19:45 server.go:3012: http: TLS handshake error from 45.244.3.73:37870: remote error: tls: unkn
own certificate
DEBU[2019-05-27T11:19:45Z] Serving default certificate for request: "first.mybot.testbots.grid.tf"
DEBU[2019-05-27T11:19:46Z] Serving default certificate for request: "first.mybot.testbots.grid.tf"
2019/05/27 11:19:46 server.go:3012: http: TLS handshake error from 45.244.3.73:37874: remote error: tls: ille
gal parameter
DEBU[2019-05-27T11:19:46Z] Serving default certificate for request: "first.mybot.testbots.grid.tf"
DEBU[2019-05-27T11:19:46Z] Serving default certificate for request: "first.mybot.testbots.grid.tf"
2019/05/27 11:19:46 server.go:3012: http: TLS handshake error from 45.244.3.73:37876: remote error: tls: ille
gal parameter

Caddy let's encrypt suspicious output

 root@caddy2:/opt/www_incubaid# caddy                                                                                            
Activating privacy features... 2019/05/27 12:12:14 [INFO] [second.mybot.testbots.grid.tf] acme: Obtaining bundled SAN certificat
e                                                                                                                               
2019/05/27 12:12:15 [INFO] [second.mybot.testbots.grid.tf] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/DaYl
FsX5ptnjJFNmEYE-IfnFtt7lXVd0Z3ZbOEabv4w                                                                                         
2019/05/27 12:12:15 [INFO] [second.mybot.testbots.grid.tf] acme: use tls-alpn-01 solver                                         
2019/05/27 12:12:15 [INFO] [second.mybot.testbots.grid.tf] acme: Trying to solve TLS-ALPN-01                                    
2019/05/27 12:12:22 [second.mybot.testbots.grid.tf] failed to obtain certificate: acme: Error -> One or more domains had a probl
em:                                                                                                                             
[second.mybot.testbots.grid.tf] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Incorrect validation certificate
for tls-alpn-01 challenge. Requested second.mybot.testbots.grid.tf from 37.59.44.168:443. Received 1 certificate(s), first certi
ficate had names "2994d0b30ee2a6c063d0807f04c5da8e.e5f74b8b9ea9099ff69e5191f302b630.traefik.default, traefik default cert", url:
                                              

Randomly running it worked somehow

Activating privacy features... 2019/05/27 12:14:42 [INFO] [second.mybot.testbots.grid.tf] acme: Obtaining bundled SAN certificat
e
2019/05/27 12:14:43 [INFO] [second.mybot.testbots.grid.tf] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/lott
z4tjNgcvXEc37rGraEfp4c_QvlggZfXxii0ea3Y
2019/05/27 12:14:43 [INFO] [second.mybot.testbots.grid.tf] acme: use tls-alpn-01 solver
2019/05/27 12:14:43 [INFO] [second.mybot.testbots.grid.tf] acme: Trying to solve TLS-ALPN-01
2019/05/27 12:14:50 [INFO] [second.mybot.testbots.grid.tf] The server validated our request
2019/05/27 12:14:50 [INFO] [second.mybot.testbots.grid.tf] acme: Validations succeeded; requesting certificates
2019/05/27 12:14:51 [INFO] [second.mybot.testbots.grid.tf] Server responded with a certificate.

[dduportal]

This behavior also appears with a Traefik as backend instead of caddy:

version: '3'

services:
  edge:
    image: traefik:v2.0.0-alpha4
    command:
      - "--configfile=/traefik.toml"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - "./traefik.toml:/traefik.toml"

  backend1:
    image: traefik:v1.7.11
    command:
      - "--entryPoints=Name:https Address::443 TLS"
      - "--defaultEntryPoints=https"
      - "--acme.entryPoint=https"
      - "--acme.email=damien@containo.us"
      - "--acme.storage=acme.json"
      - "--acme.tlsChallenge=true"
      - "--acme.onHostRule=true"
      - "--acme.acmeLogging=true"
      - "--acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--docker.exposedByDefault=false"
      - "--api"
    labels:
      - "traefik.enable=true"
      - "traefik.frontend.rule=Host:YYYYYYYY"
      - "traefik.port=8080"
    ports:
      - "8443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

  backend2:
    image: traefik:v1.7.11
    command:
      - "--entryPoints=Name:https Address::443 TLS"
      - "--defaultEntryPoints=https"
      - "--acme.entryPoint=https"
      - "--acme.email=damien@containo.us"
      - "--acme.storage=acme.json"
      - "--acme.tlsChallenge=true"
      - "--acme.onHostRule=true"
      - "--acme.acmeLogging=true"
      - "--acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--docker.exposedByDefault=false"
      - "--api"
    labels:
      - "traefik.enable=true"
      - "traefik.frontend.rule=Host:XXXXXXXXX"
      - "traefik.port=8080"
    ports:
      - "8444:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

and traefik.toml:

[global]
  debug = true
  logLevel = "DEBUG"
[entryPoints]
  [entryPoints.tcp]
  address = "0.0.0.0:443"

[api]

[providers.file]

[tcp]
[tcp.services]
    [tcp.services.caddy1.LoadBalancer]
      [[tcp.services.caddy1.LoadBalancer.servers]]
        address = "172.17.0.1:8443"
    [tcp.services.caddy2.LoadBalancer]
      [[tcp.services.caddy2.LoadBalancer.servers]]
        address = "172.17.0.1:8444"

[tcp.routers]
    [tcp.routers.caddy1]
      entrypoints = ["tcp"]
      rule = "HostSNI(`YYYYYY`)"
      service = "caddy1"
    [tcp.routers.caddy1.tls]
       passthrough=true


    [tcp.routers.caddy2]
      entrypoints = ["tcp"]
      rule = "HostSNI(`XXXXXXXX`)"
      service = "caddy2"
    [tcp.routers.caddy2.tls]
       passthrough=true

However, after a while, the LE certificates are generated, even if we have the same error.

We are trying to dig deeper.

[dduportal]

Hi @xmonader ,

I was not able to reproduce the behavior at all.

• Trying with Caddy, I had this elements:

docker-compose.yml:

version: '3'

services:
  edge:
    image: traefik:v2.0.0-alpha4
    command:
      - "--configfile=/traefik.toml"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - "./traefik.toml:/traefik.toml"

  backend1:
    image: abiosoft/caddy
    ports:
      - "8443:443"
    environment:
      - "ACME_AGREE=true"
    volumes:
      - ./Caddyfile-1:/etc/Caddyfile

  backend2:
    image: abiosoft/caddy
    ports:
      - "8444:443"
    environment:
      - "ACME_AGREE=true"
    volumes:
      - ./Caddyfile-2:/etc/Caddyfile

Caddyfile-1

https://XXXX {
    bind 0.0.0.0
    tls damien@noip.org {
        ca https://acme-staging-v02.api.letsencrypt.org/directory
    }
}

Caddyfile-2

https://YYYY {
    bind 0.0.0.0
    tls damien@noip.org {
        ca https://acme-staging-v02.api.letsencrypt.org/directory
    }
}

traefik.toml

[global]
  debug = true
  logLevel = "DEBUG"
[entryPoints]
  [entryPoints.tcp]
  address = "0.0.0.0:443"

[api]

[providers.file]

[tcp]
[tcp.services]
    [tcp.services.caddy1.LoadBalancer]
      [[tcp.services.caddy1.LoadBalancer.servers]]
        address = "172.17.0.1:8443"
    [tcp.services.caddy2.LoadBalancer]
      [[tcp.services.caddy2.LoadBalancer.servers]]
        address = "172.17.0.1:8444"

[tcp.routers]
    [tcp.routers.caddy1]
      entrypoints = ["tcp"]
      rule = "HostSNI(`XXXX`)"
      service = "caddy1"
    [tcp.routers.caddy1.tls]
       passthrough=true


    [tcp.routers.caddy2]
      entrypoints = ["tcp"]
      rule = "HostSNI(`YYYY`)"
      service = "caddy2"
    [tcp.routers.caddy2.tls]
       passthrough=true

• With my previous reproduction example, I realize that the error message appeared when the Traefik backends where not started yet. But it retries and finally get the certificate. Also, if I start first the Traefik v2, and then the backends, it works all the time in 2-3s max, as well as with Caddy.

We would need more informations from your side to help us reproducing and understand better what is happening: topology of your infrastructure, logs from Caddy, full reproduction case we could run on our own etc...

Aside, is there a particular reason to no use Traefik's Let's Encrypt capability, so it would simplify your setup (and use Caddy only a webserver)?

[xmonader]

I'll try to reproduce and get more logs tomorrow,

for my use case I need to use traefik on a public IP as TCP proxy and forward the TLS traffic to some secure applications based on the SNI and they do the certificate generation, TLS termination not traefik.

I'm using caddy as an example of a secure application to simplify the setup and check if it works with traefik, because i already tested the flow against HAProxy and it worked.

for the infrasturture :

• I've a container on OVH with coredns to resolve my subdomain testbots.grid.tf to traefik public IP
• Machine with 3 containers with the previous configurations in the issue.
  • traefik v2 acting as gateway tcp proxy with public IP
  • caddy container for domain first.mybot.testbots.grid.tf
  • caddy container for domain second.mybot.testbots.grid.tf
  • All containers are part of a private zerotier network

[xmonader]

I tried multiple times, but can't reproduce it, but it's worrying for me to use it with such a random error.

[dduportal]

Hi @xmonader , thanks for your feedbacks. We retried ~50 times each setup (with caddy and with Traefik), without any problem: certificates where always generated. Sometime the "403" message from LE appeared with no consequence on the certificate generation.

We were able to have issue for certificate generation when only the backend where started: it means your issue appeared during a time window where Caddy contacted Let's Encrypt while Traefik v2 did not had its configuration fully loaded, hence the default certificate.

If you ensure that Traefik v2 is started AND configured before any backend start, then you will face no error.

As you were not able to reproduce the issue, we're gonna close this issue assuming the technical issue is no more. If you happen to reproduce and can provide us how to reproduce, feel free to re-open it.

Thanks for your interest!