Can not authenticate with service-account default token #22351

Closed
erimatnor opened this Issue Mar 2, 2016 · 15 comments

Projects

None yet

9 participants

@erimatnor

I have a pod that uses the default service account token in the pod to speak to the API server. However, sometimes this token cannot be used to authenticate with the API server and the logs give this error:

E0302 12:32:50.114892       7 handlers.go:37] Unable to authenticate the request due to an error: crypto/rsa: verification error

The same thing happens when manually reading the token from the service account secret with kubectl and using the token to curl the API server using the token.

It appears that the token signature is invalid and that the token is either bad or the API server has changed its signing key and not updated the token.

@erimatnor

Update: deleting the default token secret and letting Kubernetes create a new secret solved the problem. Still, it is a bit concerning that Kubernetes can allow invalid token secrets to stick around.

@apatil
Contributor
apatil commented Apr 8, 2016

@erimatnor, how did you delete the default token secret?

@rondinelisaad

To get the default toker secret execute:
sudo kubectl get secrets --all-namespaces
the result will be something like this:
NAMESPACE NAME TYPE DATA AGE
calico-system default-token-jon84 kubernetes.io/service-account-token 3 5m
default default-token-cqe56 kubernetes.io/service-account-token 3 4m
kube-system default-token-6uce0 kubernetes.io/service-account-token 3 4m

To delete for example the kube-system namespace to regenerate, execute:
sudo kubectl delete secret --namespace=kube-system default-token-6uce0

It ins't necessary to restart kubelet service, the token will be create.

@bobrik bobrik referenced this issue in projectcalico/k8s-policy Oct 13, 2016
Open

Error messages do not point to the issue #49

@lacion
lacion commented Nov 2, 2016

this hit me today @rondinelisaad steps fixed the issue, but why would the token get corrupted|invalidated?

@liggitt
Member
liggitt commented Nov 2, 2016

if you have changed the service account token signing key, existing tokens in etcd will no longer validate

@alexbrand
Member

@liggitt Which key is used to sign the service accounts? When running multiple API Servers, does the TLS private key have to be the same for all of them?

We have seen in one of our HA clusters that requests to one of the API servers constantly fails with the verification error when using the service account from within a pod.

@liggitt
Member
liggitt commented Nov 16, 2016

The service account public key has to be the same for all. In an HA setup that means you need to explicitly give it to each apiserver (recommended) or make all the apiservers use the same serving cert/private TLS key (not recommended)

@liggitt
Member
liggitt commented Nov 16, 2016

The service account token private key given to the controller manager is used to sign the tokens.

@alexbrand
Member

@liggitt thanks!

@bigglesandginger
bigglesandginger commented Nov 23, 2016 edited

@liggitt what do you mean "...explicitly give it to each apiserver...", how? I have not seen any documentation about this mention this seemingly huge flaw in HA deployments.

@liggitt
Member
liggitt commented Nov 23, 2016

distribute the public key you want the api servers to use to verify service account tokens as part of distributing the configuration/options.

@bigglesandginger

I misinterpreted the CoreOS instructions about the api-server and controller-manager configurations. The CLI parameter --service-account-private-key-file= must point at the same key for all api-servers and controller-managers. Now I understand. Thanks.

@EamonZhang

@bigglesandginger @liggitt

1.--service-account-private-key-file=?

2.what is the use of

 apiserver.crt
 apiserver.key

generate by --admission-control=ServiceAccount

  1. when i config the params in kube-apiservice
 --client-ca-file=
--tls-cert-file=
--tls-private-key-file=

then apiserver.crt ,apiserver.key file will not generate.

@liggitt
Member
liggitt commented Jan 5, 2017

@EamonZhang --service-account-private-key-file provided to the controller manager is used to sign service account tokens. The corresponding public key must be provided to the api server with --service-account-key-file, which uses it to verify tokens.

As a convenience, you can provide a private key to both, and the public key portion of it will be used by the api server to verify token signatures.

As a further convenience, the api server's private key for it's serving certificate is used to verify service account tokens if you don't specify --service-account-key-file

--tls-cert-file and --tls-private-key-file are used to provide the serving cert and key to the api server. If you don't specify these, the api server will make a self-signed cert/key-pair and store it at apiserver.crt/apiserver.key

@liggitt liggitt closed this Jan 5, 2017
@EamonZhang

@liggitt

I kown clearly now. Thx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment