Traefik provides default TLS certificate instead of one from a secret

[ostapkostyk]

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

In kubernetes cluster I configured IngressRoute to secure communication with simple nginx server using tls certificate already issued.

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: test-nginx
  namespace: test
  annotations: 
    kubernetes.io/ingress.class: traefik-external
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`10.1.0.71`)
      kind: Rule
      services:
        - name: test-nginx
          port: 80
  tls:
    secretName: admin-local-tls

The secret exists in the same namespace, here is description from kubectl:

$ kubectl get secret admin-local-tls -n test -o yaml
apiVersion: v1
data:
  ca.crt: LS0t....
  tls.crt: LS0tL.....
  tls.key: LS0tLS1CRUdJT.....
kind: Secret
metadata:
  annotations:
    cert-manager.io/alt-names: 10.1.0.71
    cert-manager.io/certificate-name: admin-local
    cert-manager.io/common-name: 10.1.0.70
    cert-manager.io/ip-sans: 10.1.0.71
    cert-manager.io/issuer-group: ""
    cert-manager.io/issuer-kind: ClusterIssuer
    cert-manager.io/issuer-name: vault-cluster-issuer
    cert-manager.io/uri-sans: ""
  creationTimestamp: "2024-06-06T09:18:10Z"
  labels:
    controller.cert-manager.io/fao: "true"
  name: admin-local-tls
  namespace: test
  resourceVersion: "11426168"
  uid: 210f0e72-90ea-46a8-9a96-267d2e07a59d
type: kubernetes.io/tls

Note that CN and SANs are valid, secret type is correct, certificate is present.

Routing itself works well if I remove tls part and use web entryPoint.

Here is certificate info (from the secret):

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            33:17:74:6f:rest omitted
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = D......
        Validity
            Not Before: Jun  6 09:24:30 2024 GMT
            Not After : Sep  4 09:25:00 2024 GMT
        Subject: CN = 10.1.0.70
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:ab:c8:rest omitted
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Key Agreement
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Key Identifier: 
                E2:00:E8:B6:rest omitted
            X509v3 Authority Key Identifier: 
                46:CD:B6:41:rest omitted
            Authority Information Access: 
                CA Issuers - URI:http://omitted
            X509v3 Subject Alternative Name: 
                DNS:10.1.0.71, IP Address:10.1.0.71
            X509v3 CRL Distribution Points: 
                Full Name:
                  URI:http://omitted
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        9d:a7:c6:rest omitted

What did you see instead?

Traefik serves it's default TLS certificate.

No entries in traefik logs (Warn level)

What version of Traefik are you using?

3.0.1

What is your environment & configuration?

globalArguments:
  - "--global.sendanonymoususage=false"
  - "--global.checknewversion=false"

additionalArguments:
  - "--serversTransport.insecureSkipVerify=true"
  - "--log.level=WARN"

deployment:
  enabled: true
  replicas: 3
  annotations: {}
  podAnnotations: {}
  additionalContainers: []
  initContainers: []

ingressClass:
  enabled: true
  isDefaultClass: true

ports:
  web:
    port: 8000
    expose:
      default: true
    exposedPort: 80
    protocol: TCP
  websecure:
    tls:
      enabled: true
      
ingressRoute:
  dashboard:
    enabled: false

providers:
  kubernetesCRD:
    enabled: true
    ingressClass: traefik-external
    allowExternalNameServices: true
  kubernetesIngress:
    enabled: true
    ingressClass: traefik-external
    allowExternalNameServices: true
    publishedService:
      enabled: false

rbac:
  enabled: true

service:
  enabled: true
  type: LoadBalancer
  annotations: {}
  labels: {}
  spec:
    loadBalancerIP: 10.1.0.71
  loadBalancerSourceRanges: []

nodeSelector: 
  node-role: agent
resources:
  requests:
    cpu: "100m"
    memory: "128Mi"
  limits:
    cpu: "300m"
    memory: "256Mi"

Add more configuration information here.

If applicable, please paste the log output in DEBUG level

2024-06-07T13:40:11Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:228 > Serving default certificate for request: ""
2024-06-07T13:40:11Z DBG log/log.go:245 > http: TLS handshake error from 10.42.5.0:60126: remote error: tls: unknown certificate

[ostapkostyk]

Update.

Seems there is routing problem when using websecure endpoint.

I tried to test a certificate itself. Therefore I switched from IngressRoute to IngressRouteTCP to be able to use TLS passthrough. But it appeared that routing doesn't work when I try to use IP-address: HostSNI(`10.1.0.71`).

If I use HostSNI(`*`) and TLS passthrough, then communication works, my certificate is OK. Here is working manifest:

apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
  name: test-nginx-ingressroute
  namespace: test
  annotations: 
    kubernetes.io/ingress.class: traefik-external
spec:
  entryPoints:
    - web
    - websecure
  routes:
    - match: HostSNI(`*`)
      services:
        - name: test-nginx
          port: 443
  tls:
    passthrough: true

So, the problem still persists. As I understand, Traefik doesn't route https traffic correctly with IngressRoute

[ostapkostyk]

Ah, I can't use IP-address with HostSNI. But at least I see that the certificate is fine and routing works for http traffic and using HostSNI(`*`).
IngressRoute, however, can't match the certificate for some reason, even though, the IP address in the routing configuration matches IP SANs in the certificate.

[nmengin]

Hello @ostapkostyk,

Thanks for reaching out!
To keep the repository focused, we ask that all questions be asked in the community forum. It is pretty active, so you might find that your question has already been answered there. If not, you can ask and get help from other community members pretty quickly.

[ostapkostyk]

Hello nmengin,

Sorry if I didn't understand something or missed something, but my intestion wasn't to post a question here, rather to point that Traefik doesn't work correctly.
I asked in community, got no answer, then I investigated further and now, to the best of my knowledge, I assume that Traefik just doesn't work correctly.

If you see that this is not a bug but rather my configuration is wrong, could you point me where I'm wrong?

To my understanding, Traefik doesn't mach TLS certificate in case I use IP SANs in the certificate and Host rule in IngressRoute.

[rtribotte]

Hello @ostapkostyk,

Let's reopen the issue, maybe we were mistaken at first when understanding that it was a question about TLS configuration.

So, the problem still persists. As I understand, Traefik doesn't route https traffic correctly with IngressRoute

The problem you are facing is not related to HTTP, the TLS termination happens before the request to be sent, and the IngressRoute is not in cause.

Serving default certificate for request: ""

This error log states that the default certificate is served because of an empty server name in the client hello.
The client doesn't provide the serverName and no matching certificate can be found.

What is the use case you are trying to achieve? Could you please elaborate?
As of now, the certificate lookup is based only on the serverName, but looking for a certificate matching the local address IP would not fit in your case, if the IP in the Host matcher of the IngressRoute is the service one.

[DavidN-Work]

I thought I was the only one experiencing this issue. I've been burnt out from trying different configurations, and I keep coming to the same conclusion.

In my case, I can successfully issue a certificate, but only for the Traefik Dashboard using secretName through a certificate in the Traefik namespace, pulled from a production Let's Encrypt cluster issuer.

However, all other services deployed with an Ingress Route don't get issued a certificate, despite having a valid and successfully issued certificate for the hostname and namespace from ACME.

In the Traefik Dashboard, I can see that my test Nginx server has a web secure entry point with a TLS certificate from Let's Encrypt.

But when I run openssl s_client -connect nginx.local.lab.tech:443, it indicates that the certificate is self-signed and not from Let's Encrypt.

I tested this with both staging and production ACME servers.

My Environment:

• RKE2 with IPVS
• MetalLB
• Cert Manager using Let's Encrypt with DNS-01 challenge from Cloudflare
• Traefik 3.0.1

[ostapkostyk]

Hello @rtribotte,

As of now, the certificate lookup is based only on the serverName

Thanks, this explains the behavior quite clearly.

My usecase:
We have a k3s cluster on bare metal. I'd like to expose some services only within local network (administration services). For not technical reasons there is no DNS service in the network and I can't set it up (no access). Therefore certificates are issued with IP-addresses as IP SANs, but unfortunately Traefik can't match them.

Technical configuration
I use Vault from HashiCorp as PKI, Cert-Manager to manage certificates. Just for testing I deployed simple nginx web-server and try to route traffic with Traefik's IngressRoute.

What I tried:

• route traffic without TLS: works
• route traffic with TLS from Traefik (without TLS between Traefik and nginx): doesn't work due to default certificate from Traefik
• Encrypt traffic from nginx and use tls passthough mode in Traefik: doesn't work for individual service and I don't want to set this flag globally
• use default TLSStore from Traefik: it works and currently is my workaround. Just can't manage certificates individually for each service (don't need it now)

The IP address in the IngressRoute for matching is from the host network, it's not an IP from kubernetes environment. Actually, this IP comes from MetalLB load-balancer.

Thank you for your interest to this issue! If you need manifests, I'll be happy to provide them on request.

[traefiker]

Hi! I'm Træfiker 🤖 the bot in charge of tidying up the issues.I have to close this one because of its lack of activity 😞Feel free to re-open it or join our Community Forum.