Kubernetes provider logs errors during object loading

[dtomcej]

Do you want to request a feature or report a bug?

Bug

What did you do?

Create an ingress with a TLS cert in a secret

What did you expect to see?

Nice, clean, happy logs.

What did you see instead?

time="2018-07-24T15:42:43Z" level=error msg="Error configuring TLS for ingress web/api: secret web/traefik-cert does not exist"
time="2018-07-24T15:42:43Z" level=error msg="Error configuring TLS for ingress web/web: secret web/traefik-cert does not exist"
time="2018-07-24T15:42:43Z" level=error msg="Error configuring TLS for ingress web/store: secret web/traefik-cert does not exist"
time="2018-07-24T15:42:43Z" level=error msg="Error configuring TLS for ingress web/services: secret web/traefik-cert does not exist"
time="2018-07-24T15:42:43Z" level=debug msg="Creating entry point redirect http -> https"
time="2018-07-24T15:42:44Z" level=debug msg="Received Kubernetes event kind *v1.Secret"
time="2018-07-24T15:42:44Z" level=debug msg="Received Kubernetes event kind *v1.Secret"
time="2018-07-24T15:42:44Z" level=debug msg="Received Kubernetes event kind *v1.Secret"

If applicable, please paste the log output in DEBUG level (--logLevel=DEBUG switch)

This is the cause of the error logs in #3402, as the secrets do exist, they are just not sync'd yet.

[micw]

This is absolutely bad user experience which costed me (and others) hours to find the issue. I had some missing certs, so I got a invalid cert error in browser and saw in the logs that the cert is missing. After I fixed this, I still got the error in the logs. I diged through all the other (open and closed) bugs with this error message, altered my traefik installation, reinstalled traefik, manually checked the service role, reinstalled the cert - all the time getting this wrong error.
I was about to give up and switch to nginx ingres when I saw by chance that the cert is actually loaded and the browser error disappeared.

[ekjuanrejon]

Am having the same issue with aks 1.14.8

Is there an update on this issue?

[dtomcej]

Although this was an issue in v1, we are using a different client and a different provider process in v2.

Note that this issue did not affect performance, it was a logging anomaly.

For this reason, we are going to be closing this issue. If the issue occurs in v2, we will open a new issue, and have debugging data in that new issue.

[wailua]

time="2019-11-19T17:39:28Z" level=error msg="Error configuring TLS: secret traefik/kui-ingressgateway-certs does not exist" providerName=kubernetescrd ingress=https-dashboard namespace=traefik

False error still exists in Traefik 2.05. It's not a timing issue. The secret existed in my kubernetes namespace long before the traefik pod started. Besides that all works fine. It's really just a false log message.

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: https-dashboard
  namespace: traefik
spec:
  entryPoints:
  - https
  routes:
  - match: Host(`traefik.example.com`) && PathPrefix(`/`)
    kind: Rule
    services:
    - name: dashboard
      port: 8080
    middlewares:
    - name: kui-forward-auth-dashboard
  tls:
    secretName: kui-ingressgateway-certs

Setting in Traefik config:

...
    entryPoints:
      https:
        address: ":443"
    certificatesResolvers:
      default: {}

I'm using let's encrypt to create the secret

[Serrvosky]

Hello guys,

I'm trying to apply secure dashboard in a Kubernetes scenario using Traefik v2.1, but something it's not ok and I think it's the same error.
This is my config:

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: traefik-auth
spec:
  basicAuth:
    secret: traefikusers
---
apiVersion: v1
kind: Secret
metadata:
  name: traefikusers
  namespace: default
data:
  users: |2
    dGVzdDokYXByMSRINnVza2trVyRJZ1hMUDZld1RyU3VCa1RycUU4d2ov
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
spec:
  routes:
  - match: PathPrefix(`/api`) || PathPrefix(`/dashboard`))
    kind: Rule
    services:
    - name: api@internal
      kind: TraefikService

But I'm getting this log:

time="2019-12-27T15:31:49Z" level=info msg="Configuration loaded from file: /etc/traefik/traefik.toml"
time="2019-12-27T15:31:50Z" level=error msg="Error while reading basic auth middleware: failed to load auth credentials: secret 'default/traefikusers' not found" providerName=kubernetescrd middlewareName=default-traefik-auth
time="2019-12-27T15:31:50Z" level=error msg="Error while reading basic auth middleware: failed to load auth credentials: secret 'default/traefikusers' not found" middlewareName=default-traefik-auth providerName=kubernetescrd

kubectl auth can-i get secrets --namespace=default --as=system:serviceaccount:default:traefik-ingress-controller
yes

Name:         traefik-ingress-controller
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"rbac.authorization.k8s.io/v1beta1","kind":"ClusterRole","metadata":{"annotations":{},"name":"traefik-ingress-controller"},"...
PolicyRule:
  Resources                             Non-Resource URLs  Resource Names  Verbs
  ---------                             -----------------  --------------  -----
  endpoints                             []                 []              [get list watch]
  secrets                               []                 []              [get list watch]
  services                              []                 []              [get list watch]
  ingresses.extensions                  []                 []              [get list watch]
  ingressroutes.traefik.containo.us     []                 []              [get list watch]
  ingressroutetcps.traefik.containo.us  []                 []              [get list watch]
  middlewares.traefik.containo.us       []                 []              [get list watch]
  tlsoptions.traefik.containo.us        []                 []              [get list watch]
  traefikservices.traefik.containo.us   []                 []              [get list watch]
  secrets.v1                            []                 []              [get list watch]
  ingresses.extensions/status           []                 []              [update]

When i try to see if the secret is there with kubectl it works, but looks like traefik doesn't find it, and I guess it's because traefik is trying to find secret default/traefikusers instead of secret traefikusers

Anyone with this error too?
Thanks

[bigpigeon]

I found default tls setting source code
https://github.com/containous/traefik/blob/master/pkg/server/router/tcp/router.go#L97
if you want to use custom tls ,must define tlsOption kind
https://github.com/containous/traefik/blob/master/pkg/server/router/tcp/router.go#L110-L153
and then I try to use tlsOption to setting it,but still not work and not log

  tls:
    secretName: traefik-tls
    options:
      name: mytlsoption
      namespace: kube-system
---
apiVersion: traefik.containo.us/v1alpha1
kind: TLSOption
metadata:
  name: mytlsoption
  namespace: kube-system

spec:
  minVersion: VersionTLS12                     
  maxVersion: VersionTLS13                      
  curvePreferences:                             
    - CurveP521
    - CurveP384
  cipherSuites:                                
    - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    - TLS_RSA_WITH_AES_256_GCM_SHA384
  clientAuth:                                  
    secretNames:                                
      - traefik-tls
    clientAuthType: VerifyClientCertIfGiven     
  sniStrict: true                               

I give up
reading the source code experience is terrible. There is a lot of redundant and unclear code.

[ldez]

We improved the doc:

• https://docs.traefik.io/v2.1/routing/providers/kubernetes-crd/#kind-ingressroute
• https://docs.traefik.io/v2.1/routing/providers/kubernetes-crd/#kind-tlsoption

[micw]

@ldez I cannot see how the updated doc is related to the fact that traefik logs a false error message. Maybe you should add to https://docs.traefik.io/v2.1/providers/kubernetes-ingress/ something like "if you see secret not found errors, ignore it, it's a false error" if you don't intent to fix this.

[bradjones1]

This is still an issue in master, AFAIK. In my case, re: secrets and also CRD Middlewares.

[tgruenert]

Traefik version 2.1.4 built on 2020-02-06T17:10:06Z
gives the same false error message. Good to find this ticket to find out the "reason". Please don´t close this until this is fixed!

[traefiker]

Closed by #6364.