Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Traefik should not serve TRAEFIK DEFAULT CERT when there is a matching custom cert #5849

Closed
bithavoc opened this issue Nov 15, 2019 · 13 comments
Labels
area/tls contributor/waiting-for-feedback kind/bug/possible a possible bug that needs analysis before it is confirmed or fixed. status/5-frozen-due-to-age

Comments

@bithavoc
Copy link

bithavoc commented Nov 15, 2019

Do you want to request a feature or report a bug?

Bug

What did you do?

I have a deployment for my workload served by an ingress with a custom Let's Encrypt certificate I added manually to the kubernetes cluster. The website works fine in Chrome most of the time, however, some users reports that Firefox sometimes does not work.

What did you expect to see?

Traefik serves ONLY ONE certificate matching the host of the ingress path all the time.

What did you see instead?

Traefik serves TWO certificates, one matching my host of the ingress path and also a non SNI certificate with Subject TRAEFIK DEFAULT CERT.

This one was hard to catch because I guess most of the time browsers such as Firefox, Safari and Chrome latest version are able to figure out what certificate to pick from the ones Traefik serves via TLS and ignore the unmatching non SNI default cert, however, the same browsers some time stutter and pick the wrong one which is why some users sometimes see a page flagged as non-secure.

Output of traefik version: (What version of Traefik are you using?)

1.7.19

What is your environment & configuration (arguments, toml, provider, platform, ...)?

- args:                                                               
  - --api                                                             
  - --kubernetes                                                      
  - --logLevel=DEBUG                                                  
  - --defaultentrypoints=http,https                                   
  - --entrypoints=Name:https Address::443 TLS                         
  - --entrypoints=Name:http Address::80
image: traefik:1.7.19

If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)

time="2019-11-15T21:41:35Z" level=warning msg="Skipping addition of certificate for domain(s) \"*.widemeshstaging.net\", to EntryPoint https, as it already exists for this Entrypoint."
time="2019-11-15T21:41:35Z" level=debug msg="No entryPoint is defined to add the certificate MIIFQjCCBCqgAwIBAgITAPqioz+AQZpoAUMUfkEWbcB/szANBg, it will be added to the default entryPoints: http, https"
time="2019-11-15T21:41:35Z" level=warning msg="Skipping addition of certificate for domain(s) \"*.widemeshstaging.net\", to EntryPoint http, as it already exists for this Entrypoint."
time="2019-11-15T21:41:35Z" level=warning msg="Skipping addition of certificate for domain(s) \"*.widemeshstaging.net\", to EntryPoint https, as it already exists for this Entrypoint."
time="2019-11-15T21:41:35Z" level=debug msg="No entryPoint is defined to add the certificate MIIFQjCCBCqgAwIBAgITAPqioz+AQZpoAUMUfkEWbcB/szANBg, it will be added to the default entryPoints: http, https"
time="2019-11-15T21:41:35Z" level=warning msg="Skipping addition of certificate for domain(s) \"*.widemeshstaging.net\", to EntryPoint http, as it already exists for this Entrypoint."
time="2019-11-15T21:41:35Z" level=warning msg="Skipping addition of certificate for domain(s) \"*.widemeshstaging.net\", to EntryPoint https, as it already exists for this Entrypoint."
time="2019-11-15T21:41:35Z" level=debug msg="No entryPoint is defined to add the certificate MIIFUjCCBDqgAwIBAgITAPojprhTW7+YB7vIFammRaQQRDANBg, it will be added to the default entryPoints: http, https"
time="2019-11-15T21:41:35Z" level=debug msg="Adding certificate for domain(s) platrio.ohshitreactnative.com"
time="2019-11-15T21:41:35Z" level=debug msg="Adding certificate for domain(s) platrio.ohshitreactnative.com"
time="2019-11-15T21:41:35Z" level=debug msg="No entryPoint is defined to add the certificate MIIFQjCCBCqgAwIBAgITAPqioz+AQZpoAUMUfkEWbcB/szANBg, it will be added to the default entryPoints: http, https"
time="2019-11-15T21:41:35Z" level=warning msg="Skipping addition of certificate for domain(s) \"*.widemeshstaging.net\", to EntryPoint http, as it already exists for this Entrypoint."
time="2019-11-15T21:41:35Z" level=warning msg="Skipping addition of certificate for domain(s) \"*.widemeshstaging.net\", to EntryPoint https, as it already exists for this Entrypoint."
time="2019-11-15T21:41:35Z" level=info msg="Server configuration reloaded on :8080"
time="2019-11-15T21:41:35Z" level=info msg="Server configuration reloaded on :443"
time="2019-11-15T21:41:35Z" level=debug msg="Certificates not added to non-TLS entryPoint http."
time="2019-11-15T21:41:35Z" level=info msg="Server configuration reloaded on :80"
time="2019-11-15T21:41:51Z" level=debug msg="Received Kubernetes event kind *v1.Endpoints"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:51Z" level=error msg="Error creating global backend for ingress env-2f095024-cdf5-4ba7-bf8a-64e7c10cefb8/ep-1844688a-01e3-4354-b812-e881350a5f7a: duplicate frontend: global-default-frontend"
time="2019-11-15T21:41:51Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints"
time="2019-11-15T21:41:51Z" level=debug msg="Received Kubernetes event kind *v1.Endpoints"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:51Z" level=error msg="Error creating global backend for ingress env-2f095024-cdf5-4ba7-bf8a-64e7c10cefb8/ep-1844688a-01e3-4354-b812-e881350a5f7a: duplicate frontend: global-default-frontend"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:51Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:51Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints"
time="2019-11-15T21:41:52Z" level=debug msg="Received Kubernetes event kind *v1.Endpoints"
time="2019-11-15T21:41:52Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:52Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:52Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:52Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:52Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:52Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:52Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:52Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:52Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:52Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:52Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:52Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:52Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:52Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:52Z" level=error msg="Error creating global backend for ingress env-2f095024-cdf5-4ba7-bf8a-64e7c10cefb8/ep-1844688a-01e3-4354-b812-e881350a5f7a: duplicate frontend: global-default-frontend"
time="2019-11-15T21:41:52Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints"
time="2019-11-15T21:41:53Z" level=debug msg="Received Kubernetes event kind *v1.Endpoints"
time="2019-11-15T21:41:53Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:53Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:53Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:53Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:53Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:53Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:53Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:53Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:53Z" level=error msg="Error creating global backend for ingress env-2f095024-cdf5-4ba7-bf8a-64e7c10cefb8/ep-1844688a-01e3-4354-b812-e881350a5f7a: duplicate frontend: global-default-frontend"
time="2019-11-15T21:41:53Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:53Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:53Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:53Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:53Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web01"
time="2019-11-15T21:41:53Z" level=error msg="Service not found for env-9f57598b-e71d-487a-9a9e-7a522d1babe3/frontend-web6"
time="2019-11-15T21:41:53Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints"

Here's a report from SSL Checker reporting that secondary certificate, check Certificate #2 the one that says non-SNI:

SSL Server Test: sample-custom-dc2.widemeshstaging.net (Powered by Qualys SSL Labs).pdf

For comparison, here's a SSL checker report but using HAPROXY Controller serving the exact same ingresses:
HAPROXY SSL Server Test: sample-custom-dc2.widemeshstaging.net (Powered by Qualys SSL Labs).pdf

As you can see, there is no default cert being served.

@bithavoc
Copy link
Author

bithavoc commented Nov 16, 2019

More details

This is HAPROXY Controller serving the exact same ingresses:
HAPROXY SSL Server Test: sample-custom-dc2.widemeshstaging.net (Powered by Qualys SSL Labs).pdf

As you can see, there is no default cert being served in addition to the matching server_name host(only one cert) which is the correct behavior.

@Blackclaws
Copy link

Can confirm the same is happening when using traefik from docker-compose directly with ACME. Its getting the letsencrypt certificate fine and serving it but traefik keeps serving the default cert for requests not specifying a hostname. I would expect traefik to simply fail hard if the hostname is not known when using SNI not serve a default cert.

I would also not expect traefik to serve its default certificate while loading the ACME certificates from a store.

@dduportal dduportal added area/tls kind/bug/possible a possible bug that needs analysis before it is confirmed or fixed. and removed status/0-needs-triage labels Nov 18, 2019
@bithavoc
Copy link
Author

I think there's a chance Traefik might be returning the certificates in the wrong order randomly, so in some requests it sometimes returns the matching SNI certificate first and then the default while some other times it returns the default certificate first and then the matching certificate SNI second.

In any case, it should not serve the default certificate if there is a matching certificate.

@dduportal
Copy link
Contributor

Hi @bithavoc , could you provide a reproduction case (let's say with a script using curl and/or openssl that underlines this behavior, without any caching risk from web browser) ?

@dtomcej
Copy link
Contributor

dtomcej commented Nov 27, 2019

@bithavoc,
SSL Labs tests SNI and Non-SNI connection attempts to your server.

By default, if a non-SNI request is sent to Traefik, and it cannot find a matching certificate (with an IP SAN), it will return the default certificate, which is usually self signed.

If you are required to pass this sort of SSL test, you may need to either:

  1. Configure a default certificate to serve when no match can be found:
    https://docs.traefik.io/v1.7/configuration/entrypoints/#default-certificate

  2. Configure Strict SNI checking so that no connection can be made without a matching certificate:
    https://docs.traefik.io/v1.7/configuration/entrypoints/#strict-sni-checking

Please let us know if that resolves your issue.

@bithavoc
Copy link
Author

@dtomcej I shouldn't need Strict SNI checking since there is a matching certificate for the domain, should I?

@bithavoc
Copy link
Author

beware that that URL I first posted is already using Haproxy, not Traefik. I can restore the traefik environment so you can try again though, lmk what you want to do.

@dtomcej
Copy link
Contributor

dtomcej commented Nov 27, 2019

@bithavoc,

As I mentioned earlier: SSL Labs tests SNI and Non-SNI connection attempts to your server. It will attempt to connect via the domain name AND the IP address, which is why you get the non-match due to the IP address connections.

If your certificate is for example.com it is NOT a match for 1.1.1.1 which your domain could resolve to. That is where the strict SNI matching may be required.

@bithavoc
Copy link
Author

I switched to ha proxy briefly, will be trying the strict tls option soon. Don't close yet.

@adamdecaf
Copy link

adamdecaf commented Jan 17, 2020

I ran into this in my traefik setup as well. Specifying tls.domains on each router seems to have solved the issue by prioritizing the custom certificate instead of the default certificate. (commit)

The comment above about this being sporadic got me looking through the code and I see a couple map[string]Certificate for loops, which are iterated randomly in Go. That could be a cause of this happening when no domain is specified which excludes the default certificate.

It would be nice to have an option to disable the DEFAULT CERTIFICATE and error/warn in cases where no certificate is usable for a route. There's no reason (in production) to serve the default.

I didn't try strict SNI checking, but my problem seems solved without it. I've got a LB and some requests without hostnames in my setup that I didn't want to change to fix this issue.

@traefiker
Copy link
Contributor

Hi! I'm Træfiker 🤖 the bot in charge of tidying up the issues.

I have to close this one because of its lack of activity 😞

Feel free to re-open it or join our Community Forum.

@traefik traefik locked and limited conversation to collaborators Apr 28, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area/tls contributor/waiting-for-feedback kind/bug/possible a possible bug that needs analysis before it is confirmed or fixed. status/5-frozen-due-to-age
Projects
None yet
Development

No branches or pull requests

7 participants