Improved error logging for HTTPRoute to HTTPS backend

[Javex]

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you expect to see?

Summary

When a user incorrectly configures a HTTPRoute with a backendRef to a service that speaks HTTPS, Traefik's access log will log an error 500 and return only a generic "Internal Server Error". It would be good if Traefik could log a more specific error indicating that the issue is the HTTP vs HTTPS discrepancy, i.e. that a TLS connection is expected.

The only log message looks like this:

10.244.0.1 - - [27/Dec/2024:02:44:55 +0000] "GET /metrics HTTP/2.0" 500 21 "-" "-" 2 "httproute-kubernetes-dashboard-dashboard-gw-traefik-traefik-gateway-ep-websecure-1-6aff3b64d344407bb5f7@kubernetesgateway" "https://10.244.0.25:8443" 2ms

Details

I have deployed Traefik as a Kubernetes Gateway API provider and want to use it as the single HTTPS ingress point for my cluster. While building my cluster, I deployed Kubernetes Dashboard as a service to prove out the concept.

First, I installed Traefik using Helm, following these instructions, i.e. using Helm, with the following values.yaml:

gateway:
  enabled: true
  listeners:
    web:
      port: 8000
      hostname: ""
      protocol: HTTP
      namespacePolicy:
    websecure:
      port: 8443
      protocol: HTTPS
      namespacePolicy: All
      certificateRefs:
        - name: traefik-cert-tls
          kind: Secret
      mode: Terminate

providers:
  kubernetesIngress:
    enabled: false
  kubernetesGateway:
    enabled: true

logs:
  access:
    enabled: true

helm list reports chart version traefik-33.2.1 and app version v3.2.2.

I then installed Kubernetes Dashboard:

helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/
helm upgrade --install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --create-namespace --namespace kubernetes-dashboard

And finally I added a HTTPRoute:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: dashboard
  namespace: kubernetes-dashboard
spec:
  parentRefs:
    - name: traefik-gateway
      namespace: traefik
  hostnames:
    - k8s-dashboard.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: kubernetes-dashboard-kong-proxy
          port: 443

When all of this is deployed, running curl https://k8s-dashboard.example.com/ produces "Internal Server Error" and the access log has a message similar to the above.

The reason for this, I think, is because kubernetes-dashboard-kong-proxy expects a TLS connection, but Traefik doesn't know that and would initiate a HTTP connection without TLS. The fix for this is relatively easy, I bypassed kong-proxy entirely and copied its setup into a Traefik-native HTTPRoute:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: dashboard
  namespace: kubernetes-dashboard
spec:
  parentRefs:
    - name: traefik-gateway
      namespace: traefik
  hostnames:
    - k8s-dashboard.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api/v1/login
        - path:
            type: PathPrefix
            value: /api/v1/csrftoken/login
        - path:
            type: PathPrefix
            value: /api/v1/me
      backendRefs:
        - name: kubernetes-dashboard-auth
          port: 8000
    - matches:
        - path:
            type: PathPrefix
            value: /api
        - path:
            type: PathPrefix
            value: /metrics
      backendRefs:
        - name: kubernetes-dashboard-api
          port: 8000
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: kubernetes-dashboard-web
          port: 8000

Now the dashboard works as expected.

This issue is really just a feature request to log a more expressive error. Note that there are also no messages in the regular Traefik log at "INFO" level, i.e. the log line above is the only line that appears when running kubectl logs -n traefik services/traefik -f

Additional notes:

• The HTTPRouteRule backendRefs field says to return a 500 when the backend ref is invalid. HTTPBackendRef further elaborates, calling out the status: False condition field. In the above configuration this is True, making the issue harder to debug as the 500 expectation from the spec isn't met
• I'm aware of BackendTLSPolicy and its experimental status. Of course it would be nice if I could just set a TLS backend (some searching revealed there might be a label, too?), but my goal with this issue isn't to ask for that feature, just for a better error message in logs to pinpoint the problem more easily.

[rtribotte]

Hello @Javex,

Thanks for reporting this!

Did you enabled the logs at DEBUG level, their should be a DEBUG log explaining the error.
Does the kubernetes-dashboard-kong-proxy service require a particular SNI?

[Javex]

Hi @rtribotte,

Thank you for taking a look at this report! You're right, I am only running with INFO log level. I've switched to DEBUG and I can now see an error message:

2024-12-31T01:58:11Z DBG github.com/traefik/traefik/v3/pkg/proxy/httputil/proxy.go:113 > 500 Internal Server Error error="tls: failed to verify certificate: x509: cannot validate certificate for 10.244.0.46 because it doesn't contain any IP SANs"

I also had a similar issue yesterday when I forgot to set the correct port on a forward auth middleware, and adding debug logging produces a helpful message here, too:

2024-12-31T02:04:45Z DBG github.com/traefik/traefik/v3/pkg/middlewares/auth/forward.go:152 > Error calling http://authentik-outpost.authentik-outpost.svc.cluster.local/outpost.goauthentik.io/auth/traefik. Cause: Get "http://authentik-outpost.authentik-outpost.svc.cluster.local/outpost.goauthentik.io/auth/traefik": dial tcp 10.111.16.242:80: connect: connection refused middlewareName=monitoring-forward-auth@kubernetescrd middlewareType=ForwardAuth

It looks like in both cases, the log level is DEBUG, but to me this feels like it's worth of a warning level. It's pretty easy to make these kinds of misconfiguration and those log messages provide really valuable insight. Having to turn on debug logging seems like too verbose a log level for such a "serious" issue. What do you think?

[kevinpollet]

Hello @Javex,

It looks like this issue is a duplicate of #11292, therefore I am closing it.

Duplicate of #11292