[SUPPORT] Kubernetes ServiceAccount no longer auto-generates secret

[M-gre]

Summary

As of 1.24 Kubernetes no longer auto-generates secrets for ServiceAccounts. I'm having trouble working around this limitation.

Detailed Description

I'm trying to connect my Vault instance as described in the wiki. Due to the mentioned change in 1.24 of kubernetes the tutorial provided in the wiki is no longer accurate. When creating the ServiceAccount and exporting the secrets there is no return value. I have tried to work around this issue with the suggested workaround but kubernetes is not generating a secret for me to export.

Expected Behavior

After creating a ServiceAccount and creating a service account token secret I should be able to export a secret for further use.

Current Behavior

The secret for the service account does not get populated.

Steps to Reproduce

1. kubectl -n kube-system create serviceaccount vault-auth
serviceaccount/vault-auth created

kubectl -n kube-system apply -f - <<EOF
> apiVersion: v1
> kind: Secret
> metadata:
>   name: vault-auth-secret
>   namespace: kube-system
>   annotations:
>     kubernetes.io/service-account.name: "vault-auth"
> type: kubernetes.io/service-account-token
> data:
> EOF

secret/vault-auth-secret created
3.

kubectl -n kube-system get sa vault-auth
NAME         SECRETS   AGE
vault-auth   0         14s

Context (Environment)

This is my first try connecting Vault to Kubernetes. I wanted to follow along the tutorial to get the connection working.
I'm running my cluster in minikube (not sure if this is relevant).

[romdalf]

Thanks for reporting.

Trousseau is marked as supporting Kubernetes versions up to 1.22.
I see three things that needs to happen based on your experience:

• create manually the token for the ServiceAccount, test, and update the documentation accordingly
• check if there are a different approach to leverage the Vault Kubernetes Auth with Kubernetes version 1.24+
• update our end-2-end to include newer version of Kubernetes earlier in the QA

Let me crack these this Thursday.

[romdalf]

• create manually the token for the ServiceAccount, test, and update the documentation accordingly

Hey @M-gre, it might be related to minikube - here is my steps with rancher-desktop:

kubectl get nodes                                      
NAME                   STATUS   ROLES                  AGE   VERSION
lima-rancher-desktop   Ready    control-plane,master   42s   v1.24.3+k3s1

then create the vault-auth ServiceAccount:

kubectl -n kube-system create serviceaccount vault-auth       
serviceaccount/vault-auth created

then create the vault-auth Secret:

kubectl apply -f vault-auth-secret.yml 
secret/vault-auth-secret created

with as YAML manifest:

---
apiVersion: v1
kind: Secret
metadata:
  namespace: kube-system
  name: vault-auth
  annotations:
    kubernetes.io/service-account.name: "vault-auth"
type: kubernetes.io/service-account-token

then apply RBAC:

kubectl apply -f vault-auth.yml 
clusterrolebinding.rbac.authorization.k8s.io/role-tokenreview-binding created

with as YAML manifest:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
   name: role-tokenreview-binding
   namespace: kube-system
roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-auth
  namespace: kube-system

finally check the token generation by Kubernetes:

kubectl describe secrets vault-auth -n kube-system
Name:         vault-auth
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: vault-auth
              kubernetes.io/service-account.uid: aa9fe853-29a7-4f0d-a247-0e6df10925f7

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     566 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6Illybldqd2xaRTZNRl95RXpoZzFiTFVRWlVWLUpjZkdraHlSTDVxTmZfX1EifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJ2YXVsdC1hdXRoLXNlY3JldCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJ2YXVsdC1hdXRoIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYWE5ZmU4NTMtMjlhNy00ZjBkLWEyNDctMGU2ZGYxMDkyNWY3Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOnZhdWx0LWF1dGgifQ.iD7-LOvvKRuNxVkr7P-bMd3x4c9g65LMhRDNoj8oAgnMq08_RkZrEXKWRCXiQ2h7ucNlop6DC17Hnp5cppbbcnSPDwOUV9iN2Y4I9yKmVOW0pv_f4Co52HmToswMElEaBj3l9LPzPiHG-QlTGPDoXX5j3mcj7Rd53UO7ep08CkvUZGFxInSNvuTkJScOda1_WQxRXLyB3AMW-hPfVcejsCI-6Lnea31YFFJ_WqO_mL5LTG92c4eOk5bF9i7sFkbLU8GvEWNzXytTSLYZ6J2J_sL1W8Eq8vfkwHhMu9dtEKcHoe3U0MRjP6ZVvsTSlcrQYIEUXmRlMLlofZzwkfSNpw

and the linked with the ServiceAccount:

kubectl -n kube-system describe sa vault-auth

Name:                vault-auth
Namespace:           kube-system
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              vault-auth
Events:              <none>

Notes:

• the ServiceAccount vault-auth is mapping the token from the Secret vault-auth
• interesting enough it doesn't link the Secret with the ServiceAccount

This might be an expected outcome that needs to be confirmed. Obviously, it changes the deployment as the export commands depends on that field.

[blomsoft]

I found an explanation here, this is the default behaviour in Kubenetes v1.24: https://itnext.io/big-change-in-k8s-1-24-about-serviceaccounts-and-their-secrets-4b909a4af4e0, https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#no-really-you-must-read-this-before-you-upgrade and https://kubernetes.io/docs/concepts/configuration/secret/#service-account-token-secrets:

"The LegacyServiceAccountTokenNoAutoGeneration feature gate is beta, and enabled by default. When enabled, Secret API objects containing service account tokens are no longer auto-generated for every ServiceAccount. Use the TokenRequest API to acquire service account tokens, or if a non-expiring token is required, create a Secret API object for the token controller to populate with a service account token by following this guide. (kubernetes/kubernetes#108309, @zshihang)"

[romdalf]

There is a bit more than that from a deployment perspective as the created token Secret is not showing up as linked to the ServiceAccount.
So we need to change the retrieval command which will be addressed by retrofitting the Helm charts to v1.

[romdalf]

The documentation has been updated accordingly: https://docs.trousseau.io/trousseau/deployment/