New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can not authenticate with service-account default token #22351

Closed
erimatnor opened this Issue Mar 2, 2016 · 22 comments

Comments

Projects
None yet
@erimatnor

I have a pod that uses the default service account token in the pod to speak to the API server. However, sometimes this token cannot be used to authenticate with the API server and the logs give this error:

E0302 12:32:50.114892       7 handlers.go:37] Unable to authenticate the request due to an error: crypto/rsa: verification error

The same thing happens when manually reading the token from the service account secret with kubectl and using the token to curl the API server using the token.

It appears that the token signature is invalid and that the token is either bad or the API server has changed its signing key and not updated the token.

@erimatnor

This comment has been minimized.

Show comment
Hide comment
@erimatnor

erimatnor Mar 2, 2016

Update: deleting the default token secret and letting Kubernetes create a new secret solved the problem. Still, it is a bit concerning that Kubernetes can allow invalid token secrets to stick around.

Update: deleting the default token secret and letting Kubernetes create a new secret solved the problem. Still, it is a bit concerning that Kubernetes can allow invalid token secrets to stick around.

@apatil

This comment has been minimized.

Show comment
Hide comment
@apatil

apatil Apr 8, 2016

Contributor

@erimatnor, how did you delete the default token secret?

Contributor

apatil commented Apr 8, 2016

@erimatnor, how did you delete the default token secret?

@rondinelisaad

This comment has been minimized.

Show comment
Hide comment
@rondinelisaad

rondinelisaad Jul 1, 2016

To get the default toker secret execute:
sudo kubectl get secrets --all-namespaces
the result will be something like this:
NAMESPACE NAME TYPE DATA AGE
calico-system default-token-jon84 kubernetes.io/service-account-token 3 5m
default default-token-cqe56 kubernetes.io/service-account-token 3 4m
kube-system default-token-6uce0 kubernetes.io/service-account-token 3 4m

To delete for example the kube-system namespace to regenerate, execute:
sudo kubectl delete secret --namespace=kube-system default-token-6uce0

It ins't necessary to restart kubelet service, the token will be create.

To get the default toker secret execute:
sudo kubectl get secrets --all-namespaces
the result will be something like this:
NAMESPACE NAME TYPE DATA AGE
calico-system default-token-jon84 kubernetes.io/service-account-token 3 5m
default default-token-cqe56 kubernetes.io/service-account-token 3 4m
kube-system default-token-6uce0 kubernetes.io/service-account-token 3 4m

To delete for example the kube-system namespace to regenerate, execute:
sudo kubectl delete secret --namespace=kube-system default-token-6uce0

It ins't necessary to restart kubelet service, the token will be create.

@lacion

This comment has been minimized.

Show comment
Hide comment
@lacion

lacion Nov 2, 2016

this hit me today @rondinelisaad steps fixed the issue, but why would the token get corrupted|invalidated?

lacion commented Nov 2, 2016

this hit me today @rondinelisaad steps fixed the issue, but why would the token get corrupted|invalidated?

@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Nov 2, 2016

Member

if you have changed the service account token signing key, existing tokens in etcd will no longer validate

Member

liggitt commented Nov 2, 2016

if you have changed the service account token signing key, existing tokens in etcd will no longer validate

@alexbrand

This comment has been minimized.

Show comment
Hide comment
@alexbrand

alexbrand Nov 16, 2016

Member

@liggitt Which key is used to sign the service accounts? When running multiple API Servers, does the TLS private key have to be the same for all of them?

We have seen in one of our HA clusters that requests to one of the API servers constantly fails with the verification error when using the service account from within a pod.

Member

alexbrand commented Nov 16, 2016

@liggitt Which key is used to sign the service accounts? When running multiple API Servers, does the TLS private key have to be the same for all of them?

We have seen in one of our HA clusters that requests to one of the API servers constantly fails with the verification error when using the service account from within a pod.

@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Nov 16, 2016

Member

The service account public key has to be the same for all. In an HA setup that means you need to explicitly give it to each apiserver (recommended) or make all the apiservers use the same serving cert/private TLS key (not recommended)

Member

liggitt commented Nov 16, 2016

The service account public key has to be the same for all. In an HA setup that means you need to explicitly give it to each apiserver (recommended) or make all the apiservers use the same serving cert/private TLS key (not recommended)

@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Nov 16, 2016

Member

The service account token private key given to the controller manager is used to sign the tokens.

Member

liggitt commented Nov 16, 2016

The service account token private key given to the controller manager is used to sign the tokens.

@alexbrand

This comment has been minimized.

Show comment
Hide comment
Member

alexbrand commented Nov 16, 2016

@liggitt thanks!

@bigglesandginger

This comment has been minimized.

Show comment
Hide comment
@bigglesandginger

bigglesandginger Nov 23, 2016

@liggitt what do you mean "...explicitly give it to each apiserver...", how? I have not seen any documentation about this mention this seemingly huge flaw in HA deployments.

bigglesandginger commented Nov 23, 2016

@liggitt what do you mean "...explicitly give it to each apiserver...", how? I have not seen any documentation about this mention this seemingly huge flaw in HA deployments.

@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Nov 23, 2016

Member

distribute the public key you want the api servers to use to verify service account tokens as part of distributing the configuration/options.

Member

liggitt commented Nov 23, 2016

distribute the public key you want the api servers to use to verify service account tokens as part of distributing the configuration/options.

@bigglesandginger

This comment has been minimized.

Show comment
Hide comment
@bigglesandginger

bigglesandginger Nov 23, 2016

I misinterpreted the CoreOS instructions about the api-server and controller-manager configurations. The CLI parameter --service-account-private-key-file= must point at the same key for all api-servers and controller-managers. Now I understand. Thanks.

I misinterpreted the CoreOS instructions about the api-server and controller-manager configurations. The CLI parameter --service-account-private-key-file= must point at the same key for all api-servers and controller-managers. Now I understand. Thanks.

@EamonZhang

This comment has been minimized.

Show comment
Hide comment
@EamonZhang

EamonZhang Jan 5, 2017

@bigglesandginger @liggitt

1.--service-account-private-key-file=?

2.what is the use of

 apiserver.crt
 apiserver.key

generate by --admission-control=ServiceAccount

  1. when i config the params in kube-apiservice
 --client-ca-file=
--tls-cert-file=
--tls-private-key-file=

then apiserver.crt ,apiserver.key file will not generate.

@bigglesandginger @liggitt

1.--service-account-private-key-file=?

2.what is the use of

 apiserver.crt
 apiserver.key

generate by --admission-control=ServiceAccount

  1. when i config the params in kube-apiservice
 --client-ca-file=
--tls-cert-file=
--tls-private-key-file=

then apiserver.crt ,apiserver.key file will not generate.

@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Jan 5, 2017

Member

@EamonZhang --service-account-private-key-file provided to the controller manager is used to sign service account tokens. The corresponding public key must be provided to the api server with --service-account-key-file, which uses it to verify tokens.

As a convenience, you can provide a private key to both, and the public key portion of it will be used by the api server to verify token signatures.

As a further convenience, the api server's private key for it's serving certificate is used to verify service account tokens if you don't specify --service-account-key-file

--tls-cert-file and --tls-private-key-file are used to provide the serving cert and key to the api server. If you don't specify these, the api server will make a self-signed cert/key-pair and store it at apiserver.crt/apiserver.key

Member

liggitt commented Jan 5, 2017

@EamonZhang --service-account-private-key-file provided to the controller manager is used to sign service account tokens. The corresponding public key must be provided to the api server with --service-account-key-file, which uses it to verify tokens.

As a convenience, you can provide a private key to both, and the public key portion of it will be used by the api server to verify token signatures.

As a further convenience, the api server's private key for it's serving certificate is used to verify service account tokens if you don't specify --service-account-key-file

--tls-cert-file and --tls-private-key-file are used to provide the serving cert and key to the api server. If you don't specify these, the api server will make a self-signed cert/key-pair and store it at apiserver.crt/apiserver.key

@liggitt liggitt closed this Jan 5, 2017

@EamonZhang

This comment has been minimized.

Show comment
Hide comment
@EamonZhang

EamonZhang Jan 6, 2017

@liggitt

I kown clearly now. Thx

@liggitt

I kown clearly now. Thx

@deitch

This comment has been minimized.

Show comment
Hide comment
@deitch

deitch Apr 6, 2017

@liggitt : so --service-account-key-file is the certificate, not the CA signing certficate?

That does make it more complicated to deploy. With the other components (client auth, e.g.), each master server has its own public/private keypair. As long as the cert is signed by the same CA, all is good.

With this, it looks like it must be the same key (not signed by the same CA), that means that we need to worry about private key distribution. Is that correct?

deitch commented Apr 6, 2017

@liggitt : so --service-account-key-file is the certificate, not the CA signing certficate?

That does make it more complicated to deploy. With the other components (client auth, e.g.), each master server has its own public/private keypair. As long as the cert is signed by the same CA, all is good.

With this, it looks like it must be the same key (not signed by the same CA), that means that we need to worry about private key distribution. Is that correct?

@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Apr 6, 2017

Member

It is a private key, not a certificate. Only the controller manager needs the private key, which it uses to sign the tokens.

The masters only need the public key portion in order to verify the tokens signed by the controller manager.

Member

liggitt commented Apr 6, 2017

It is a private key, not a certificate. Only the controller manager needs the private key, which it uses to sign the tokens.

The masters only need the public key portion in order to verify the tokens signed by the controller manager.

@deitch

This comment has been minimized.

Show comment
Hide comment
@deitch

deitch Apr 6, 2017

That was my question. Why, with everything else, is it proper CA structure, but here it is private key? It means I need to distribute the keys to all master nodes in the cluster. I already have to distribute a CA key (so the masters can sign their server certs), would be good to have it follow the same philosophy - as long as your token is a cert signed by the CA, you are good.

deitch commented Apr 6, 2017

That was my question. Why, with everything else, is it proper CA structure, but here it is private key? It means I need to distribute the keys to all master nodes in the cluster. I already have to distribute a CA key (so the masters can sign their server certs), would be good to have it follow the same philosophy - as long as your token is a cert signed by the CA, you are good.

@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Apr 6, 2017

Member

You can't use a certificate as token auth.

Member

liggitt commented Apr 6, 2017

You can't use a certificate as token auth.

@deitch

This comment has been minimized.

Show comment
Hide comment
@deitch

deitch Apr 6, 2017

That's what I don't get. Why not? Everything else uses certs and validates using a CA. This even has the (api) infra in place with --service-account-private-key-file and --service-account-key-file which accept RSA private key and private key or signed cert.

Why does the controller not follow the same pattern and instead of issuing a token, issue a cert, and have the API server validate it with a CA?

deitch commented Apr 6, 2017

That's what I don't get. Why not? Everything else uses certs and validates using a CA. This even has the (api) infra in place with --service-account-private-key-file and --service-account-key-file which accept RSA private key and private key or signed cert.

Why does the controller not follow the same pattern and instead of issuing a token, issue a cert, and have the API server validate it with a CA?

@bamb00

This comment has been minimized.

Show comment
Hide comment
@bamb00

bamb00 Feb 24, 2018

Hi, I have a similar issue with the kube-dashboard getting the CrashLoopBackOff error after the pod has been up for 100+ days. I ran the "kubectl delete secret default-token-9shsv --namespace=kube-system" but it didn't seem to resolve the kube-dashboard CrashLoopBackOff.

 # kubectl get po -n kube-system
 kube-apiserver-devtest-stage5-master-01            1/1       Running            0          23h
 kube-controller-manager-devtest-stage5-master-01   1/1       Running            0          112d
 kube-dns-2425271678-5bzs5                          3/3       Running            0          112d
 kube-proxy-n9962                                   1/1       Running            0          112d
 kube-scheduler-devtest-stage5-master-01            1/1       Running            0          112d
 kubernetes-dashboard-3902617266-wf9r6              0/1       CrashLoopBackOff   5          22h
 weave-net-lcx0z                                    2/2       Running            0          112d

 # kubectl logs kube-apiserver-devtest-stage5-master-01 -n kube-system
 E0223 01:09:35.375994       1 authentication.go:58] Unable to authenticate the request due to an error: 
 [invalid bearer token, [invalid bearer token, [Token has been invalidated, invalid bearer token]]]
 E0223 01:09:36.126745       1 authentication.go:58] Unable to authenticate the request due to an error: 
 [invalid bearer token, [invalid bearer token, [Token has been invalidated, invalid bearer token]]]

 # kubectl logs kube-proxy-n9962 -n kube-system
 E0223 03:11:49.132778       1 reflector.go:201] 
 k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:72: Failed to 
 list *api.Service: the server has asked for the client to provide credentials (get services)
 E0223 03:11:50.134702       1 reflector.go:201] 
 k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:72: Failed to 
 list *api.Endpoints: the server has asked for the client to provide credentials (get endpoints)

bamb00 commented Feb 24, 2018

Hi, I have a similar issue with the kube-dashboard getting the CrashLoopBackOff error after the pod has been up for 100+ days. I ran the "kubectl delete secret default-token-9shsv --namespace=kube-system" but it didn't seem to resolve the kube-dashboard CrashLoopBackOff.

 # kubectl get po -n kube-system
 kube-apiserver-devtest-stage5-master-01            1/1       Running            0          23h
 kube-controller-manager-devtest-stage5-master-01   1/1       Running            0          112d
 kube-dns-2425271678-5bzs5                          3/3       Running            0          112d
 kube-proxy-n9962                                   1/1       Running            0          112d
 kube-scheduler-devtest-stage5-master-01            1/1       Running            0          112d
 kubernetes-dashboard-3902617266-wf9r6              0/1       CrashLoopBackOff   5          22h
 weave-net-lcx0z                                    2/2       Running            0          112d

 # kubectl logs kube-apiserver-devtest-stage5-master-01 -n kube-system
 E0223 01:09:35.375994       1 authentication.go:58] Unable to authenticate the request due to an error: 
 [invalid bearer token, [invalid bearer token, [Token has been invalidated, invalid bearer token]]]
 E0223 01:09:36.126745       1 authentication.go:58] Unable to authenticate the request due to an error: 
 [invalid bearer token, [invalid bearer token, [Token has been invalidated, invalid bearer token]]]

 # kubectl logs kube-proxy-n9962 -n kube-system
 E0223 03:11:49.132778       1 reflector.go:201] 
 k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:72: Failed to 
 list *api.Service: the server has asked for the client to provide credentials (get services)
 E0223 03:11:50.134702       1 reflector.go:201] 
 k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:72: Failed to 
 list *api.Endpoints: the server has asked for the client to provide credentials (get endpoints)
@rayburgemeestre

This comment has been minimized.

Show comment
Hide comment
@rayburgemeestre

rayburgemeestre Jul 25, 2018

Did you try to delete the pod? It could still have the old token mounted maybe

Did you try to delete the pod? It could still have the old token mounted maybe

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment