bug: `TLSStore` with Wildcard Certificate and `sniStrict: true` does not work

[georglauterbach]

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

I am not sure if this repository or traefik/traefik-helm-chart is the correct repo to open this in, but I am under the impression this might be something in Traefik itself, not in Kubernetes (CRDs).

---

I am using Cloudflare (CF) Origin Certificates (Traefik can use them to talk to the CF proxies). This is nice because I do not need certificate resolvers like Let'sEncrypt. I was upgrading from v2.10.7 to 3.0.2 and first completely uninstalled all the old CRDs and Traefik, then installed the new version.

All of this worked nicely and I encountered no issues. Here are the relevant definitions:

---
apiVersion: v1
kind: Secret

metadata:
  name: cloudflare-origin-certificate

type: kubernetes.io/tls
immutable: true

data:
  tls.key: A VALID BASE64 KEY VERIFIED WITH OPENSSL
  tls.crt: A VALID BASE64 CERTIFICATE VERIFIED WITH OPENSSL

I am using Helm to deploy Traefik, and inside the values.yaml, I use

tlsStore:
  default:
    defaultCertificate:
      secretName: cloudflare-origin-certificate

All of this lives in the ingress namespace.

I also use TLSOptions:

tlsOptions:
  default:
    minVersion: VersionTLS12
    maxVersion: VersionTLS13
    sniStrict: true
    curvePreferences: [ secp521r1, secp384r1, secp256r1, x25519 ]
    alpnProtocols: [h2, h3]

Assume my domain is mydomain.test. The CF Origin Certificate is valid for *.mydomain.test and mydomain.test.

What did you see instead?

When trying to open subdomain.mydomain.test, what I am now seeing is this, though:

It seems Traefik does not use the certificates from my TLSStore anymore. Looking at the debug logs:

TLS: strict SNI enabled - No certificate found for domain: "subdomain.mydomain.test", closing connection
2024-06-16T15:29:05+02:00 DBG log/log.go:245 > http: TLS handshake error from 162.158.110.154:36198: tls: no certificates configured

I am also seeing

2024-06-16T15:15:33+02:00 DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default

which is unexpected because both the Secret and the TLSStore exist:

$ kubectl describe secrets -n ingress
Name:         cloudflare-origin-certificate-com
Namespace:    ingress
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/tls

Data
====
tls.key:  X bytes
tls.crt:  Y bytes

$ kubectl describe tlsstores.traefik.io -n ingress
Name:         default
Namespace:    ingress
Labels:       app.kubernetes.io/instance=traefik-ingress
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=traefik
              helm.sh/chart=traefik-28.3.0
Annotations:  <none>
API Version:  traefik.io/v1alpha1
Kind:         TLSStore
Metadata:
  Creation Timestamp:  2024-06-16T10:20:37Z
  Generation:          9
  Resource Version:    16866321
  UID:                 288e3946-70d5-41bb-94d3-83771c65c6a8
Spec:
  Default Certificate:
    Secret Name:  cloudflare-origin-certificate
Events:           <none>

What bugs me is the log message

TLS: strict SNI enabled - No certificate found for domain: "subdomain.mydomain.test", closing connection

I used sniStrict: true before as well, though; hence I think SNI should not be the problem here. But, when using sniStrict: false, everything starts to work. Am I misunderstanding something here? I had assumed that with a wildcard certificate, I could use sniStrict: true (like before).

---

related: traefik/traefik-helm-chart#851

What version of Traefik are you using?

Version:      3.0.2
Codename:     beaufort
Go version:   go1.22.4
Built:        2024-06-10T14:38:51Z
OS/Arch:      linux/amd64

What is your environment & configuration?

See previous fields for relevant configuration snippets. Here is the IngressRoute:

---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute

metadata:
  name: subdomain

spec:
  entryPoints: [ websecure ]

  routes:
    - kind: Rule
      match: Host(`subdomain.mydomain.test`)

      services:
        - name: someservice
          namespace: some-namespace-not-ingress
          port: http
          scheme: http

  tls:
    store:
      name: default
      namespace: ingress

I tried countless different configurations now, omitting the tls section in the IngressRoute, etc.

If applicable, please paste the log output in DEBUG level

2024-06-16T15:15:33+02:00 DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
2024-06-16T15:15:58+02:00 DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:216 > TLS: strict SNI enabled - No certificate found for domain: "subdomain.mydomain.test", closing connection
2024-06-16T15:15:58+02:00 DBG log/log.go:245 > http: TLS handshake error from 172.70.175.27:17586: tls: no certificates configured

[georglauterbach]

@mmatur any progress on this one?

[mmatur]

Hi @georglauterbach,

Unfortunately I didn't had the time to look at it before. Just starting to reproduce the issue on my side and Iw will come back to you.

[mmatur]

Hi @geoffgarside,

I gave it a try and I have the same behaviour between Traefik v2.10.7 and Traefik v3.0.1.

• Test 1
  • sniStrict set to true

---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: subdomain
spec:
  entryPoints: [ websecure ]
  routes:
    - kind: Rule
      match: Host(`subdomain.mydomain.test`)
      services:
        - name: someservice
          namespace: some-namespace-not-ingress
          port: http
          scheme: http
  tls:
    store:
      name: default
      namespace: ingress

Result: Connection closed

• Test 2
  • sniStrict set to true

---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: subdomain
spec:
  entryPoints: [ websecure ]
  routes:
    - kind: Rule
      match: Host(`subdomain.mydomain.test`)
      services:
        - name: someservice
          namespace: some-namespace-not-ingress
          port: http
          scheme: http
  tls:
    secretName: cloudflare-origin-certificate # define a secret name to load the certificate in traefik because the when sniStrict is set to true, traefik will verify certificates loadded but not default certificate.
    store:
      name: default
      namespace: ingress

Result: It works

• Test 3
  • sniStrict set to false

---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: subdomain
spec:
  entryPoints: [ websecure ]
  routes:
    - kind: Rule
      match: Host(`subdomain.mydomain.test`)
      services:
        - name: someservice
          namespace: some-namespace-not-ingress
          port: http
          scheme: http
  tls:
    store:
      name: default
      namespace: ingress

Result: It works

---

With sniStrict option set to true, Traefik will check certificates that have been loaded and will not check the default certificate.

Doc reference:

With strict SNI checking enabled, Traefik won't allow connections from clients that do not specify a server_name extension or don't match any of the configured certificates. The default certificate is irrelevant on that matter.

It the same behaviour in v2.10

This behaviour has been introduced in Traefik v1.7 and has never been changed since the merge of the PR.

I unassigned myself to have the issue popping during the next triage and be able to discuss with other maintainers about it.

[Benly-walter]

I faced similar issues after doing an upgrade from v2.9.0 to v2.11.2 in helm.
With DEBUG enabled for my traefik deployment running on k8s, I could see that the default secret was no longer loaded using the TLSStore definition.

As per docs, the Kubernetes CRDs API Group traefik.containo.us has been deprecated in v2.10, and its support will end starting with Traefik v3. The API Group traefik.io has to be used instead.

Although the new traefik.io groups were applied correctly, I noted that the kubectl commands strangely still kept returning the old CRD in API group traefik.containo.us

kubectl get tlsstore -n traefik -o yaml

To check if the issue could be fixed, I rolled back Traefik to v2.9.0, but to my surprise, it still did not work
I double checked that the latest CRDs mentioned in the helm chart were already applied.

Looking at the TLSstores CRD definition properties, it also included an array object to pass a list of secret names as certificates.

apiVersion: traefik.io/v1alpha1
kind: TLSStore
metadata:
  name: default
  namespace: traefik
spec:
  certificates:
    - secretName: wildcard-tls-cert
  defaultCertificate:
    secretName: wildcard-tls-cert

After upgrading Traefik back to v2.11.2 and adding this change, the default certificate was loaded without any issues and the SNI errors were no longer logged.
I will plan the upgrade to V3 for later when more information about this issue is available.
FYI, there are multiple clusters which are running okay without any issues after the upgrade. We faced this issue only for just one cluster.

[kevinpollet]

Hello @Benly-walter and thanks for your interest in Traefik,

The issue you are facing does not seem to be related to the original one, which is only about the behavior of using the sniStrict option with the defaultCertificate.

Please could you open a new issue?

[Benly-walter]

Hello @Benly-walter and thanks for your interest in Traefik,

The issue you are facing does not seem to be related to the original one, which is only about the behavior of using the sniStrict option with the defaultCertificate.

Please could you open a new issue?

Sorry I am confused, it is the same issue.
I also use sniStrict: true with wildcard certificates which stopped working after the upgrade.
Without the default certificate loading, my API was broken and traefik logs indicated SNI errors

[kevinpollet]

Hello @Benly-walter,

At first glance, it looks like a configuration issue during the migration. It is not the same issue, as you also use the wildcard cert as a certificate in the configured TLStore. This issue concerns the non-usage of the default certificate when the sniStrict option is enabled.

Feel free to ask for help in the community forum. It's quite active, so you're likely to find an answer to your question quickly.

If you believe I am mistaken, please open a new issue.

[georglauterbach]

@kevinpollet do you have a rough estimation of when I can estimate this issue to be tackled? I'd like to turn sniStrict back on without adding the certificate explicitly to every ingress :)