Kubernetes Auth Method documentation not compatible with Kubernetes v1.21+

[tsaarni]

Describe the bug

Kubernetes v1.21 introduced new bound service account tokens with following new properties:

1. Tokens are bound to a pod and they get revoked when pod is deleted.
2. Tokens will expire.
3. Tokens are audience-limited.

The Kubernetes Auth Method documentation does not cover the use of the new tokens. This results in failed authentication.

To Reproduce

Pre-condition:

Deploy Vault inside Kubernetes. Configure Kubernetes Auth Method according to examples at https://www.vaultproject.io/docs/auth/kubernetes. Set JWT value to token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)".

(1) Authentication fails because of invalidated token

1. Delete the pod that runs Vault and wait for it to be recreated.
2. Try to authenticate using Kubernetes Auth Method
3. Vault returns 403 Forbidden with { "errors": [ "permission denied" ] } in response body.

Vault prints following error:

2021-10-18T12:09:01.075Z [ERROR] auth.kubernetes.auth_kubernetes_726dbb64: login unauthorized due to: lookup failed: service account unauthorized; this could mean it has been deleted or recreated with a new token

Kubernetes API server prints following error

E1018 12:09:01.074341       1 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, Token has been invalidated]"

(2) Authentication fails because of expired token

1. Configure Kubernetes cluster with a short expiration time by setting API server parameter service-account-max-token-expiration to 1h. Wait for 1 hour for the token to expire .
2. Try to authenticate using Kubernetes Auth Method.
3. Vault returns 403 Forbidden with { "errors": [ "permission denied" ] } in response body.

Vault prints following error:

2021-10-18T11:47:23.756Z [ERROR] auth.kubernetes.auth_kubernetes_97f02f17: login unauthorized due to: lookup failed: service account unauthorized; this could mean it has been deleted or recreated with a new token

Kubernetes API server prints following error

E1018 11:47:23.755730       1 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, Token has expired.]"

Proposed fix

Include following PR

• Do not store local service account token and CA to config. vault-plugin-auth-kubernetes#122

Update documentation, taking following points into account:

• It is recommended NOT to store Kubernetes service account token persistently into config when running Vault inside Kubernetes. Instead, the recommendation is to use the local token and CA files.
• When running Vault outside of Kubernetes, user needs to use legacy token which is not bound to any pod and does not expire. User should be instructed how to access legacy token via Secret directly. User should be WARNED that the legacy tokens could be removed in later Kubernetes release completely.

I also think following point would be beneficial

• Recommend disable_iss_validation=True. The benefit is that user does not need to find the correct iss from their specific cluster (or change it if it changes later). Kubernetes Token Review API will validate the token, so I do not see negative side in recommending that.

[hsimon-hashicorp]

Hi @tsaarni! We have a ticket we're tracking internally to update the documentation. Also, we're taking a look at the linked PR as well. Thanks for your report! :)

[tsaarni]

xref hashicorp/vault-plugin-auth-kubernetes#125

[hsimon-hashicorp]

@taoism4504 Another documentation update request. I'll have our engineers peek at the code PR.

[tsaarni]

I'll have our engineers peek at the code PR.

Thank you @hsimon-hashicorp. Waiting eagerly to fix the problem and getting Vault working again on current Kubernetes versions without inconvenient workarounds!

[hsimon-hashicorp]

As commented in the linked PR, issuer validation is now disabled by default in Vault 1.9: hashicorp/vault-plugin-auth-kubernetes#125 (comment)
Would you consider this to resolve this issue? Please let me know if you have any other concerns. Thanks! :)

[tsaarni]

I would have liked to keep this open for the documentation changes that would accompany hashicorp/vault-plugin-auth-kubernetes#122, assuming the PR accepted. Deprecating the issuer validation is a good improvement, but we still lack proper Kubernetes 1.21+ support.