Helm upgrade causes service account to be recreated (recreating service token)

[yasn77]

Helm upgrade recreates service account, which in turn recreates the secret token
for the service account. This breaks current deployments. Below are symptoms:

Before Upgrade:

kubectl -n kong get serviceaccount -l app.kubernetes.io/name=kong-app -o jsonpath='{.items[].secrets}'
[map[name:kong-tst-kong-app-token-k7qv2]]

kubectl -n kong get pod kong-tst-kong-app-cc9787b65-fwzrd -o json | jq '.spec.containers[] | select(.name == "ingress-controller").volumeMounts'
[
  {
    "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
    "name": "kong-tst-kong-app-token-k7qv2",
    "readOnly": true
  }
]

(Note: Token is the same)

After Upgrade:

kubectl -n kong get serviceaccount -l app.kubernetes.io/name=kong-app -o jsonpath='{.items[].secrets}'
[map[name:kong-tst-kong-app-token-rrzfh]]

kubectl -n kong get pod kong-tst-kong-app-cc9787b65-fwzrd -o json | jq '.spec.containers[] | select(.name == "ingress-controller").volumeMounts'
[
  {
    "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
    "name": "kong-tst-kong-app-token-k7qv2",
    "readOnly": true
  }
]

(Note: Tokens are different)

From the logs (after upgrade):

Tiller:

[tiller] 2020/01/31 17:01:46 deleting pre-upgrade hook kong-tst-kong-app for release kong-tst due to "before-hook-creation" policy
[kube] 2020/01/31 17:01:46 Starting delete for "kong-tst-kong-app" ServiceAccount
[kube] 2020/01/31 17:01:46 Waiting for 60 seconds for delete to be completed
[kube] 2020/01/31 17:01:48 building resources from manifest
[kube] 2020/01/31 17:01:48 creating 1 resource(s)
[kube] 2020/01/31 17:01:48 Watching for changes to ServiceAccount kong-tst-kong-app with timeout of 5m0s
[kube] 2020/01/31 17:01:48 Add/Modify event for kong-tst-kong-app: ADDED

From IngresController:

E0131 18:09:30.368424       1 reflector.go:125] pkg/mod/k8s.io/client-go@v0.0.0-20190819141724-e14f31a72a77/tools/cache/reflector.go:98: Failed to list *v1beta1.Ingress: Unauthorized
E0131 18:09:31.206518       1 reflector.go:125] pkg/mod/k8s.io/client-go@v0.0.0-20190819141724-e14f31a72a77/tools/cache/reflector.go:98: Failed to list *v1.KongCredential: Unauthorized
E0131 18:09:31.371432       1 reflector.go:125] pkg/mod/k8s.io/client-go@v0.0.0-20190819141724-e14f31a72a77/tools/cache/reflector.go:98: Failed to list *v1beta1.Ingress: Unauthorized
E0131 18:09:32.214002       1 reflector.go:125] pkg/mod/k8s.io/client-go@v0.0.0-20190819141724-e14f31a72a77/tools/cache/reflector.go:98: Failed to list *v1.KongCredential: Unauthorized
E0131 18:09:32.379662       1 reflector.go:125] pkg/mod/k8s.io/client-go@v0.0.0-20190819141724-e14f31a72a77/tools/cache/reflector.go:98: Failed to list *v1beta1.Ingress: Unauthorized

This is due to a pre hook that creates the service account and then deletes
(then is created again).

The fix is to remove something introduced by helm/charts@bf12a71 I understand why it was introduced, but it breaks upgrades. My suggestion is to instead manually add the service account and reference it in the values file.

PR incoming....

[yasn77]

PR created (#31)