Traefik tries to reach out to k8s Secrets even when basic auth isn't used

[timoreimann]

Do you want to request a feature or report a bug?

Bug

What did you do?

Deploy Traefik on Kubernetes without basic authentication enabled through the ingress.kubernetes.io/auth-type annotation and RBAC not permitting access to Kubernetes Secrets.

What did you expect to see?

Traefik running fine without complaints.

What did you see instead?

Traefik continuously logging about being unable to access Secrets.

Output of traefik version: (What version of Traefik are you using?)

latest master

If applicable, please paste the log output in debug mode (--debug switch)

Here's the relevant portion from the logs:

E0625 21:28:59.568275       1 reflector.go:300] github.com/containous/traefik/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to watch *v1.Secret: the server does not allow access to the requested resource (get secrets)

Traefik should only start to query for Secrets when basic authentication requires it.

A fix to the issue may impact #1707.

[timoreimann]

@ldez @dtomcej @errm I wasn't sure if we want to consider the issue as a (confirmed) bug or enhancement, so I left out the relevant label.

[dtomcej]

I guess as in #1785 the question is how we want to handle RBAC permissions. If we assume that traefik should have full access, but might not, then we need to handle failure gracefully.

If we assume that traefik has no permissions, then we need to query the API for the permissions it does have, then operate on a limited scope based on that.

[timoreimann]

I think the simplest approach would be to expect the Traefik user/administrator to know whether RBAC access to Secrets need to be granted (i.e., basic auth is going to be used) and to do so accordingly. Traefik's job would then be to get/watch for Secrets when we run into an Ingress object containing basic auth annotations only.

So I guess that matches your "Traefik should have full (Secrets) access" with the conditional "if we realize there's a need to access Secrets".

Note that a more sophisticated (but also more complex) approach that involves adapting the RBAC permissions dynamically is being discussed in #1626.

[timoreimann]

I hope to get this fixed by Traefik 1.5.

To the least, the issue will be constrained to specific namespaces that Traefik operates in as the latest master has been switched over to a watch-per-namespace model, obsoleting the need to grant unnecessary global RBAC access to secrets.

[erkolson]

I think it is problematic to grant get permission to every secret, especially in the kube-system namespace. The RBAC constraints should be limited to a specific secret resource or pattern for the secret name, for example:

rules:
  - apiGroups:
      - ""
    resources:
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - secrets
    verbs:
      - get
    resourceNames:
      - "traefik-*"

Permitting only secrets in a specific name pattern seems a reasonable limitation on a user's secret usage.

[timoreimann]

@erkolson I absolutely agree with you -- Traefik should be able to get along with a per-resource RBAC secrets access level just fine.

My currently planned approach to solve the more general issue should naturally bring this capability along. It's good though to have the requirement mentioned explicitly so that we can check it off once the feature lands.

[dtomcej]

Now that we have secret support for ingress TLS, is this still an issue? We need to be able to watch secrets in namespaces that we watch for ingresses in?

[timoreimann]

@dtomcej

We need to be able to watch secrets in namespaces that we watch for ingresses in?

I think we still do because by definition (convention?), we expect secrets to be located in the same namespace as the Ingress object requiring access.

Not sure if support for Ingress TLS will actually help us here. Isn't the converse true that with it, we have yet another secrets usage triggering the behavior described in this issue?

What I think we should do is check if we can reproduce the problem described if we (a) watch all namespaces and (b) watch individual namespaces only. If we don't see the error log message anymore in both cases, we could close the issue.

[dtomcej]

Ah I think I understand what your point is.

If we don't have any ingresses, don't start a watch for secrets.

If we have an ingress that needs to watch for secrets (such as TLS or other annotations), start a watch in that namespace?

[timoreimann]

Yep, that's basically it. There are a few challenges along the way:

• make sure we don't start to listen for secrets more than once per namespace.
• initially syncing the secrets informer store takes some time, so we need to deterministically reprocess Ingresses for which the secrets aren't available in the local cache yet. This can be done in at least two ways: to put affected Ingresses into a work queue (a feature available in recent client-go versions ,which we are running on by now) causing them to be reinjected into the event machinery for retry automatically; or to leverage the event from the successful secrets store population as a trigger to retry. In each case, we need to make sure that missing secrets due to API errors and pending store populations can be distinguished for proper handling (the former is a true error, the latter an indicator to retry later).
• do all of this not only on a namespace level, but secrets resource level to allow users very fine-grained control over their RBAC permissions.
• stop watching secrets namespaces / resources if we find them unused for longer periods of time.

Not all of these items need to be implemented at the same time. Especially the latter two could be done in follow-up PRs.