Can not authenticate with service-account default token

[erimatnor]

I have a pod that uses the default service account token in the pod to speak to the API server. However, sometimes this token cannot be used to authenticate with the API server and the logs give this error:

E0302 12:32:50.114892       7 handlers.go:37] Unable to authenticate the request due to an error: crypto/rsa: verification error

The same thing happens when manually reading the token from the service account secret with kubectl and using the token to curl the API server using the token.

It appears that the token signature is invalid and that the token is either bad or the API server has changed its signing key and not updated the token.

[erimatnor]

Update: deleting the default token secret and letting Kubernetes create a new secret solved the problem. Still, it is a bit concerning that Kubernetes can allow invalid token secrets to stick around.

[apatil]

@erimatnor, how did you delete the default token secret?

[rondinelisaad]

To get the default toker secret execute:
sudo kubectl get secrets --all-namespaces
the result will be something like this:
NAMESPACE NAME TYPE DATA AGE
calico-system default-token-jon84 kubernetes.io/service-account-token 3 5m
default default-token-cqe56 kubernetes.io/service-account-token 3 4m
kube-system default-token-6uce0 kubernetes.io/service-account-token 3 4m

To delete for example the kube-system namespace to regenerate, execute:
sudo kubectl delete secret --namespace=kube-system default-token-6uce0

It ins't necessary to restart kubelet service, the token will be create.

[lacion]

this hit me today @rondinelisaad steps fixed the issue, but why would the token get corrupted|invalidated?

[liggitt]

if you have changed the service account token signing key, existing tokens in etcd will no longer validate

[alexbrand]

@liggitt Which key is used to sign the service accounts? When running multiple API Servers, does the TLS private key have to be the same for all of them?

We have seen in one of our HA clusters that requests to one of the API servers constantly fails with the verification error when using the service account from within a pod.

[liggitt]

The service account public key has to be the same for all. In an HA setup that means you need to explicitly give it to each apiserver (recommended) or make all the apiservers use the same serving cert/private TLS key (not recommended)

[liggitt]

The service account token private key given to the controller manager is used to sign the tokens.

[alexbrand]

@liggitt thanks!

[bigglesandginger]

@liggitt what do you mean "...explicitly give it to each apiserver...", how? I have not seen any documentation about this mention this seemingly huge flaw in HA deployments.

[liggitt]

distribute the public key you want the api servers to use to verify service account tokens as part of distributing the configuration/options.

[bigglesandginger]

I misinterpreted the CoreOS instructions about the api-server and controller-manager configurations. The CLI parameter --service-account-private-key-file= must point at the same key for all api-servers and controller-managers. Now I understand. Thanks.

[EamonZhang]

@bigglesandginger @liggitt

1.--service-account-private-key-file=?

2.what is the use of

 apiserver.crt
 apiserver.key

generate by --admission-control=ServiceAccount

3. when i config the params in kube-apiservice

 --client-ca-file=
--tls-cert-file=
--tls-private-key-file=

then apiserver.crt ,apiserver.key file will not generate.

[liggitt]

@EamonZhang --service-account-private-key-file provided to the controller manager is used to sign service account tokens. The corresponding public key must be provided to the api server with --service-account-key-file, which uses it to verify tokens.

As a convenience, you can provide a private key to both, and the public key portion of it will be used by the api server to verify token signatures.

As a further convenience, the api server's private key for it's serving certificate is used to verify service account tokens if you don't specify --service-account-key-file

--tls-cert-file and --tls-private-key-file are used to provide the serving cert and key to the api server. If you don't specify these, the api server will make a self-signed cert/key-pair and store it at apiserver.crt/apiserver.key

[EamonZhang]

@liggitt

I kown clearly now. Thx

[deitch]

@liggitt : so --service-account-key-file is the certificate, not the CA signing certficate?

That does make it more complicated to deploy. With the other components (client auth, e.g.), each master server has its own public/private keypair. As long as the cert is signed by the same CA, all is good.

With this, it looks like it must be the same key (not signed by the same CA), that means that we need to worry about private key distribution. Is that correct?

[liggitt]

It is a private key, not a certificate. Only the controller manager needs the private key, which it uses to sign the tokens.

The masters only need the public key portion in order to verify the tokens signed by the controller manager.

[deitch]

That was my question. Why, with everything else, is it proper CA structure, but here it is private key? It means I need to distribute the keys to all master nodes in the cluster. I already have to distribute a CA key (so the masters can sign their server certs), would be good to have it follow the same philosophy - as long as your token is a cert signed by the CA, you are good.

[liggitt]

You can't use a certificate as token auth.

[deitch]

That's what I don't get. Why not? Everything else uses certs and validates using a CA. This even has the (api) infra in place with --service-account-private-key-file and --service-account-key-file which accept RSA private key and private key or signed cert.

Why does the controller not follow the same pattern and instead of issuing a token, issue a cert, and have the API server validate it with a CA?

[bamb00]

Hi, I have a similar issue with the kube-dashboard getting the CrashLoopBackOff error after the pod has been up for 100+ days. I ran the "kubectl delete secret default-token-9shsv --namespace=kube-system" but it didn't seem to resolve the kube-dashboard CrashLoopBackOff.

 # kubectl get po -n kube-system
 kube-apiserver-devtest-stage5-master-01            1/1       Running            0          23h
 kube-controller-manager-devtest-stage5-master-01   1/1       Running            0          112d
 kube-dns-2425271678-5bzs5                          3/3       Running            0          112d
 kube-proxy-n9962                                   1/1       Running            0          112d
 kube-scheduler-devtest-stage5-master-01            1/1       Running            0          112d
 kubernetes-dashboard-3902617266-wf9r6              0/1       CrashLoopBackOff   5          22h
 weave-net-lcx0z                                    2/2       Running            0          112d

 # kubectl logs kube-apiserver-devtest-stage5-master-01 -n kube-system
 E0223 01:09:35.375994       1 authentication.go:58] Unable to authenticate the request due to an error: 
 [invalid bearer token, [invalid bearer token, [Token has been invalidated, invalid bearer token]]]
 E0223 01:09:36.126745       1 authentication.go:58] Unable to authenticate the request due to an error: 
 [invalid bearer token, [invalid bearer token, [Token has been invalidated, invalid bearer token]]]

 # kubectl logs kube-proxy-n9962 -n kube-system
 E0223 03:11:49.132778       1 reflector.go:201] 
 k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:72: Failed to 
 list *api.Service: the server has asked for the client to provide credentials (get services)
 E0223 03:11:50.134702       1 reflector.go:201] 
 k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:72: Failed to 
 list *api.Endpoints: the server has asked for the client to provide credentials (get endpoints)

[rayburgemeestre]

Did you try to delete the pod? It could still have the old token mounted maybe

[valentin-krasontovitsch]

@liggitt it's interesting that one should use a "pure" key pair for signing and verifying service account tokens rather than a certificate (with expiration date) - I guess it implies that this key should not expire, and thus not be changed.

But what if one wants to change this key anyways, for whatever reason? Let's claim it's just "sane" to rotate keys every now and then.

In that case, it seems like changing the key pair will not automatically regenerate service account tokens. In the wonderful and magical world of kubernetes, where most things seem to happen automatically, this appears like a bug to me.

Could you perhaps shed some light on the rationale behind this / on how this is intended to be handled, or on what the best practice around this particular key pair (for signing service account tokens) is?