TLS certificate selection breaks using caddy behind ingress controller

[segfault16]

1. Which version of Caddy are you using (caddy -version)?

1.0.0, built from source

2. What are you trying to do?

I'm trying to run caddy with TLS behind a traefik ingress controller.
The problem that I'm facing is that a TLS handshake cannot be performed because caddy can't find the certificate. Traefik is forwarding the request based on the IP address of caddy, so the host information isn't available for the handshake.
To my understanding this affects potentially every IP-based LoadBalancer/Proxy/IngressController that is in front of caddy. No problems whatsoever prior to version 1.0.0.

To reproduce:

1. generate a valid certificate for localhost (e.g. with minica): minica localhost
2. Run caddy with Caddyfile below (./caddy -log /dev/stdout)
3. curl -ks https://localhost:9443 succeeds
4. curl -ks https://127.0.0.1:9443 fails

3. What is your Caddyfile?

:9443 {
    tls localhost/cert.pem localhost/key.pem
    status 200 /
}

4. How did you run Caddy (give the full command and describe the execution environment)?

./caddy -log /dev/stdout

5. Please paste any relevant HTTP request(s) here.

curl -v -ks https://localhost:9443/

*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=localhost
*  start date: Jun 28 11:04:36 2019 GMT
*  expire date: Jun 28 12:04:36 2109 GMT
*  issuer: CN=minica root ca 55bcd2
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f8bef002400)
> GET / HTTP/2
> Host: localhost:9443
> User-Agent: curl/7.54.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200 
< server: Caddy
< content-length: 0
< date: Mon, 01 Jul 2019 13:43:51 GMT
< 
* Connection #0 to host localhost left intact

curl -v -ks https://127.0.01:9443/

*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 9443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS alert, Server hello (2):
* error:14004438:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert internal error
* stopped the pause stream!
* Closing connection 0

Caddy log:

2019/07/01 15:44:21 http: TLS handshake error from 127.0.0.1:60064: no certificate available for ''

6. What did you expect to see?

Successful connection on localhost as well as 127.0.0.1 with curl -k option, SSL Verification error from curl when omitting -k option for 127.0.0.1.

7. What did you see instead (give full error messages and/or log)?

8. Why is this a bug, and how do you think this should be fixed?

9. What are you doing to work around the problem in the meantime?

No workaround known.

10. Please link to any related issues, pull requests, and/or discussion.

Bonus: What do you use Caddy for? Why did you choose Caddy?

[mholt]

It's because there's no SNI information from the client; without it, Caddy doesn't know which certificate should be served up. Before 1.0, we chose a random certificate, but this angered certain users.

This isn't a bug in Caddy, then. To avoid information leakage, either the IP address used to connect must match that on the certificate, or the client must send the proper SNI value to get the certificate.

More info: #2339

[segfault16]

For further readers: Caddy's parameter -default-sni might come in handy.

[mholt]

Indeed, setting a default ServerName when none is provided is one way to resolve the issue.

[malteneuss]

Just wanted to add that curl has a handy, relatively new --connect-to option. The general usage is:

curl --connect-to <virtualhost>:443:<realhost>:443 https://<virtualhost> -i -k

Curl will set Host header and TLS SNI in the request to <virtualhost> as desired by Caddy, but will make the actual request against <realhost>, e.g. your ingress controller.

curl --connect-to localhost:443:ingress.com:443 https://localhost -i -k

This can be extended to work with Caddys default auto_https redirects by repeating that option for http and https:

curl --connect-to <virtualhost>:80:<realhost>:80  --connect-to <virtualhost>:443:<realhost>:443 http://<virtualhost> -i -k -L