I have a pod that uses the default service account token in the pod to speak to the API server. However, sometimes this token cannot be used to authenticate with the API server and the logs give this error:
E0302 12:32:50.114892 7 handlers.go:37] Unable to authenticate the request due to an error: crypto/rsa: verification error
The same thing happens when manually reading the token from the service account secret with kubectl and using the token to curl the API server using the token.
It appears that the token signature is invalid and that the token is either bad or the API server has changed its signing key and not updated the token.
Update: deleting the default token secret and letting Kubernetes create a new secret solved the problem. Still, it is a bit concerning that Kubernetes can allow invalid token secrets to stick around.
@erimatnor, how did you delete the default token secret?
To get the default toker secret execute:
sudo kubectl get secrets --all-namespaces
the result will be something like this:
NAMESPACE NAME TYPE DATA AGE
calico-system default-token-jon84 kubernetes.io/service-account-token 3 5m
default default-token-cqe56 kubernetes.io/service-account-token 3 4m
kube-system default-token-6uce0 kubernetes.io/service-account-token 3 4m
To delete for example the kube-system namespace to regenerate, execute:
sudo kubectl delete secret --namespace=kube-system default-token-6uce0
It ins't necessary to restart kubelet service, the token will be create.
this hit me today @rondinelisaad steps fixed the issue, but why would the token get corrupted|invalidated?
if you have changed the service account token signing key, existing tokens in etcd will no longer validate
@liggitt Which key is used to sign the service accounts? When running multiple API Servers, does the TLS private key have to be the same for all of them?
We have seen in one of our HA clusters that requests to one of the API servers constantly fails with the verification error when using the service account from within a pod.
The service account public key has to be the same for all. In an HA setup that means you need to explicitly give it to each apiserver (recommended) or make all the apiservers use the same serving cert/private TLS key (not recommended)
The service account token private key given to the controller manager is used to sign the tokens.
@liggitt what do you mean "...explicitly give it to each apiserver...", how? I have not seen any documentation about this mention this seemingly huge flaw in HA deployments.
distribute the public key you want the api servers to use to verify service account tokens as part of distributing the configuration/options.
I misinterpreted the CoreOS instructions about the api-server and controller-manager configurations. The CLI parameter --service-account-private-key-file= must point at the same key for all api-servers and controller-managers. Now I understand. Thanks.
2.what is the use of
generate by --admission-control=ServiceAccount
then apiserver.crt ,apiserver.key file will not generate.
@EamonZhang --service-account-private-key-file provided to the controller manager is used to sign service account tokens. The corresponding public key must be provided to the api server with --service-account-key-file, which uses it to verify tokens.
As a convenience, you can provide a private key to both, and the public key portion of it will be used by the api server to verify token signatures.
As a further convenience, the api server's private key for it's serving certificate is used to verify service account tokens if you don't specify --service-account-key-file
--tls-cert-file and --tls-private-key-file are used to provide the serving cert and key to the api server. If you don't specify these, the api server will make a self-signed cert/key-pair and store it at apiserver.crt/apiserver.key
I kown clearly now. Thx