New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Troubleshooting docs for k8s default token expiring #2274

Closed
sjezewski opened this Issue Sep 12, 2017 · 1 comment

Comments

Projects
None yet
3 participants
@sjezewski
Contributor

sjezewski commented Sep 12, 2017

Rough notes below.

Basically ... we'd see an error like 'get nodes' not authorized. To fix it, I delete the default token that the pipelien pod is using, delete the pipeline pod, and it gets restarted w the updated token. Not sure how/why k8s default token would get out of sync like this.


Issue:

on 1.5.0-RC1
(and not on 1.4.8)

when spinning up a local cluster

i get an error causing CLB on pipeline pods

stating they dont have creds to do getnodes

i suspect my DNS mod to kube launch

just now checked on it

17-07-05[13:05:04]:1.5creds:1$kc logs po/pipeline-edges-v1-mdbsv user
unable to retrieve node list from k8s to determine parallelism: Get https://10.0.0.1:443/api/v1/nodes: dial tcp 10.0.0.1:443: getsockopt: no route to host

relaunching and making sure I use default DNS setting

so validated w default DNS:

17-07-05[13:18:16]:opencv:0$kc logs po/pipeline-edges-v1-0lp9v user
unable to retrieve node list from k8s to determine parallelism: the server has asked for the client to provide credentials (get nodes)

digging into hte k8s sys apiserver pod

see some errors:

E0705 20:11:15.141701 1 authentication.go:58] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, crypto/rsa: verification error]]

and i can't quite get to the apiserver start command to see the flags

https://stackoverflow.com/questions/32739816/generated-serviceaccount-token-is-rejected-by-kube-apiserver

maybe i could check syslog on that pod

but i do see the known_tokens.csv

i17-07-06[13:57:14]:pachyderm:130$sudo find / -name known_tokens.csv > knowntokensfind.log
[sudo] password for sjezewski:
find: \u2018/run/user/1000/gvfs\u2019: Permission denied
17-07-06[13:58:22]:pachyderm:1$
17-07-06[13:59:09]:pachyderm:1$
17-07-06[13:59:09]:pachyderm:1$
17-07-06[13:59:09]:pachyderm:1$sudo cat /var/lib/kubelet/pods/299e1fb764cf3615b73f5be7880b6b9a/volumes/kubernetes.io~empty-dir/data/known_tokens.csv
azfIJskskcixUKfwdxkQPd9jiK1CcDCr,admin,admin
zQglrjvcbvUowwjjqIeLP09lv869cDxq,kubelet,kubelet
MAODGtZpsCdlJmAdISWbsPTIIl1W3L0O,kube_proxy,kube_proxy

but getting the default token ... i don't see anything that matches up ...

17-07-06[14:01:30]:pachyderm:0$kc get secret default-token-7qfx3 -o yaml
apiVersion: v1
data:
ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURXVENDQWtHZ0F3SUJBZ0lKQUxkN29CVjZ1YlhlTUEwR0NTcUdTSWIzRFFFQkN3VUFNQ0l4SURBZUJnTlYKQkFNTUZ6RTVNaTR4TmpndU1TNHpNRUF4TkRrek5ESXlOVGt6TUI0WERURTNNRFF5T0RJek16WXpNMW9YRFRJMwpNRFF5TmpJek16WXpNMW93SWpFZ01CNEdBMVVFQXd3WE1Ua3lMakUyT0M0eExqTXdRREUwT1RNME1qSTFPVE13CmdnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUUN5UjdONnRldHRZczRuRTc5dlBQbmQKd081Y216TzJkTUVmUzk5TmZRZ1pUZmp6NHNmWG40ckZSSmlwQzIxS0h1QlFocVlLc3hONlY4V3JLMEszRG1paQo0K3U2NkQxMlZ4dE5CMk81R0FKVXFuR1FJOXl1ekgreTF5NVZFZDU4dUxSWnNOZGNSMjk0eDRJdTNoYlcxbmdOCjZJaW5kUWw3ZWRJcnRaVDRnWU8zVGxBT2FLQUVKbEpnSzc0QjFCZXd1Mm5aeFl0S0F4Y293K3NkbW45dFVydkcKSDM0US9qQ2tEaUkyTFl3TVdrV0t1amZGK3F5bTFwSFJ6UDRyVVlCNDJSVDRRUU0rQUNRV2FyMWtEL3hPS0hYaApLWG1SUDZldkR0YlFzVm82MldxOWlaQm9BVmRJc2pWaFZmS3N5Z3htZzNPczJRRVZDWXQ5aWcvY0ZUVWhWelMxCkFnTUJBQUdqZ1pFd2dZNHdIUVlEVlIwT0JCWUVGTHBoY2g2K29VbCt1SEh0aC9SVnVwM21YRnZ5TUZJR0ExVWQKSXdSTE1FbUFGTHBoY2g2K29VbCt1SEh0aC9SVnVwM21YRnZ5b1Nha0pEQWlNU0F3SGdZRFZRUUREQmN4T1RJdQpNVFk0TGpFdU16QkFNVFE1TXpReU1qVTVNNElKQUxkN29CVjZ1YlhlTUF3R0ExVWRFd1FGTUFNQkFmOHdDd1lEClZSMFBCQVFEQWdFR01BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQlpzcTdQMmZzVWJQUXNidWtxUldlcVo5emkKM3FVYmhXTlQrYUVIWXNrUFFZaFpmSmVySERHNHY4eG1Eek5qbnhoRHM4YjNuL2tMSEVIeXBZQnhlM29qMDJlcAoxUEZtZW0yUjE4MzIzTHRubDhDVDl0TVJGTEIwWE5OeWZialNpWGF3U0NYdVkxZDRFcW5FSkxUUmQyVUV2ekovCjRXWm5uK2FZYVo3bldFRjJRNE1aOTkrQ01UMlNXa0pLaU96YmlWaDI2TTNYdFhDd2JkaWFlY0J4aTdsb3FGb2MKUVU5eml4TTBTYzJIKzdLYy9qdUcxbm4vd2VaUzBoK0hxdnJyWHhLc2ZlS3FmM0YzUTlmZDhoNWo2UXkyR1U4OApqNGcwQjB1NGM1SUFHeDk5NmRaSkJMTEFLWDBxY3NRVGw4VGRoaUo0eXBWengxWVhhdEpsbkFxWDJVMjcKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
namespace: ZGVmYXVsdA==
token: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJblI1Y0NJNklrcFhWQ0o5LmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUprWldaaGRXeDBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbVJsWm1GMWJIUXRkRzlyWlc0dE4zRm1lRE1pTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1dVlXMWxJam9pWkdWbVlYVnNkQ0lzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVnlkbWxqWlMxaFkyTnZkVzUwTG5WcFpDSTZJamcxTkdVMlltUTFMVEpqTm1JdE1URmxOeTA0WkdOaExXTTROV0kzTmpBMk1ESTNOQ0lzSW5OMVlpSTZJbk41YzNSbGJUcHpaWEoyYVdObFlXTmpiM1Z1ZERwa1pXWmhkV3gwT21SbFptRjFiSFFpZlEuZ2hSaEFBbnkxNXJMcEsyQWVrc0hmODV1cmN2TEU2T1lXWU5ZXzF1YmlZZ1J3M051VUFJcG8xdDZIQi1WdFltMDJRN2cwbXJfM3hMTDhJd0wwMW82Z3VZNVFhVDdkX094b0xEdEZ4UWFvbXVrRFJaTloyZmVxOHVBWFpNVDhVRTVWbnc0ZkJhSllKSVI2bDBDQjJUbGR1QUw1R2l6eUs1eFZsNHlDVExFeVVrSXEzZVRFbzlFeTA5ckxQSExpVFJmcUpfalBUWlZ3c3J3Qk9vdDFhV1JIOVc5a2lPWnpmX19WZDFCUWpaU0hyTHU4bHA2eF9ZdHhkdnhlVVBuU09JRkpoUFllNW5MWXhNUUVSemRDREt0ajE5NjZlTjJMY084SFBzNEQzZExyOXh3bnJESDdyeVFvbXdNVXVMb1dGQm1IdHpVLVU3blRqQXVIRjcwTks4ZHFn
kind: Secret
metadata:
annotations:
kubernetes.io/service-account.name: default
kubernetes.io/service-account.uid: 854e6bd5-2c6b-11e7-8dca-c85b76060274
creationTimestamp: 2017-04-28T23:36:36Z
name: default-token-7qfx3
namespace: default
resourceVersion: "110"
selfLink: /api/v1/namespaces/default/secrets/default-token-7qfx3
uid: 85509f7e-2c6b-11e7-8dca-c85b76060274
type: kubernetes.io/service-account-token
17-07-06[14:01:44]:pachyderm:0$echo ZXlKaGJHY2lPaUpTVXpJMU5pSXNJblI1Y0NJNklrcFhWQ0o5LmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUprWldaaGRXeDBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbVJsWm1GMWJIUXRkRzlyWlc0dE4zRm1lRE1pTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1dVlXMWxJam9pWkdWbVlYVnNkQ0lzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVnlkbWxqWlMxaFkyTnZkVzUwTG5WcFpDSTZJamcxTkdVMlltUTFMVEpqTm1JdE1URmxOeTA0WkdOaExXTTROV0kzTmpBMk1ESTNOQ0lzSW5OMVlpSTZJbk41YzNSbGJUcHpaWEoyYVdObFlXTmpiM1Z1ZERwa1pXWmhkV3gwT21SbFptRjFiSFFpZlEuZ2hSaEFBbnkxNXJMcEsyQWVrc0hmODV1cmN2TEU2T1lXWU5ZXzF1YmlZZ1J3M051VUFJcG8xdDZIQi1WdFltMDJRN2cwbXJfM3hMTDhJd0wwMW82Z3VZNVFhVDdkX094b0xEdEZ4UWFvbXVrRFJaTloyZmVxOHVBWFpNVDhVRTVWbnc0ZkJhSllKSVI2bDBDQjJUbGR1QUw1R2l6eUs1eFZsNHlDVExFeVVrSXEzZVRFbzlFeTA5ckxQSExpVFJmcUpfalBUWlZ3c3J3Qk9vdDFhV1JIOVc5a2lPWnpmX19WZDFCUWpaU0hyTHU4bHA2eF9ZdHhkdnhlVVBuU09JRkpoUFllNW5MWXhNUUVSemRDREt0ajE5NjZlTjJMY084SFBzNEQzZExyOXh3bnJESDdyeVFvbXdNVXVMb1dGQm1IdHpVLVU3blRqQXVIRjcwTks4ZHFn | base64 -d
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tN3FmeDMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6Ijg1NGU2YmQ1LTJjNmItMTFlNy04ZGNhLWM4NWI3NjA2MDI3NCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.ghRhAAny15rLpK2AeksHf85urcvLE6OYWYNY_1ubiYgRw3NuUAIpo1t6HB-VtYm02Q7g0mr_3xLL8IwL01o6guY5QaT7d_OxoLDtFxQaomukDRZNZ2feq8uAXZMT8UE5Vnw4fBaJYJIR6l0CB2TlduAL5GizyK5xVl4yCTLEyUkIq3eTEo9Ey09rLPHLiTRfqJ_jPTZVwsrwBOot1aWRH9W9kiOZzf__Vd1BQjZSHrLu8lp6x_YtxdvxeUPnSOIFJhPYe5nLYxMQERzdCDKtj1966eN2LcO8HPs4D3dLr9xwnrDH7ryQomwMUuLoWFBmHtzU-U7nTjAuHF70NK8dqg

this thread suggests deleting the token helps:

kubernetes/kubernetes#22351

unclear how it would've changed // how my signing auth would've changed

yup! that worked

17-07-06[14:29:04]:pachyderm:0$kc get all
NAME                         READY     STATUS            RESTARTS   AGE
po/etcd-4197107720-zvnpf     1/1       Running           2          1d
po/pachd-3722220436-m6kf3    1/1       Running           4          1d
po/pipeline-edges-v1-57fnz   0/2       PodInitializing   0          2s

NAME                   DESIRED   CURRENT   READY     AGE
rc/pipeline-edges-v1   1         1         0         2s

NAME                    CLUSTER-IP   EXTERNAL-IP   PORT(S)                       AGE
svc/etcd                10.0.0.39    <nodes>       2379:32379/TCP                1d
svc/kubernetes          10.0.0.1     <none>        443/TCP                       22d
svc/pachd               10.0.0.22    <nodes>       650:30650/TCP,651:30651/TCP   1d
svc/pipeline-edges-v1   10.0.0.123   <none>        80/TCP                        2s

NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/etcd    1         1         1            1           1d
deploy/pachd   1         1         1            1           1d

NAME                  DESIRED   CURRENT   READY     AGE
rs/etcd-4197107720    1         1         1         1d
rs/pachd-3722220436   1         1         1         1d
17-07-06[14:29:06]:pachyderm:0$kc get all
NAME                         READY     STATUS    RESTARTS   AGE
po/etcd-4197107720-zvnpf     1/1       Running   2          1d
po/pachd-3722220436-m6kf3    1/1       Running   4          1d
po/pipeline-edges-v1-57fnz   1/2       Error     1          3s

NAME                   DESIRED   CURRENT   READY     AGE
rc/pipeline-edges-v1   1         1         0         3s

NAME                    CLUSTER-IP   EXTERNAL-IP   PORT(S)                       AGE
svc/etcd                10.0.0.39    <nodes>       2379:32379/TCP                1d
svc/kubernetes          10.0.0.1     <none>        443/TCP                       22d
svc/pachd               10.0.0.22    <nodes>       650:30650/TCP,651:30651/TCP   1d
svc/pipeline-edges-v1   10.0.0.123   <none>        80/TCP                        3s

NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/etcd    1         1         1            1           1d
deploy/pachd   1         1         1            1           1d

NAME                  DESIRED   CURRENT   READY     AGE
rs/etcd-4197107720    1         1         1         1d
rs/pachd-3722220436   1         1         1         1d
17-07-06[14:29:07]:pachyderm:0$kc get all
NAME                         READY     STATUS             RESTARTS   AGE
po/etcd-4197107720-zvnpf     1/1       Running            2          1d
po/pachd-3722220436-m6kf3    1/1       Running            4          1d
po/pipeline-edges-v1-57fnz   1/2       CrashLoopBackOff   1          6s

NAME                   DESIRED   CURRENT   READY     AGE
rc/pipeline-edges-v1   1         1         0         6s

NAME                    CLUSTER-IP   EXTERNAL-IP   PORT(S)                       AGE
svc/etcd                10.0.0.39    <nodes>       2379:32379/TCP                1d
svc/kubernetes          10.0.0.1     <none>        443/TCP                       22d
svc/pachd               10.0.0.22    <nodes>       650:30650/TCP,651:30651/TCP   1d
svc/pipeline-edges-v1   10.0.0.123   <none>        80/TCP                        6s

NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/etcd    1         1         1            1           1d
deploy/pachd   1         1         1            1           1d

NAME                  DESIRED   CURRENT   READY     AGE
rs/etcd-4197107720    1         1         1         1d
rs/pachd-3722220436   1         1         1         1d
17-07-06[14:29:10]:pachyderm:0$kc describe po/etcd-4197107720-zvnpf | grep service
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7qfx3 (ro)
17-07-06[14:29:19]:pachyderm:0$kc --namespace=kube-system logs po/k8s-master-127.0.0.1 apiserver > k8sapiserver^Crver-cm.log2
17-07-06[14:29:26]:pachyderm:130$kc describe po/pachd-3722220436-m6kf3 | grep service
      /var/run/secrets/kubernetes.io/serviceaccount from pachyderm-token-m7tpm (ro)
17-07-06[14:29:37]:pachyderm:0$kc describe po/pipeline-edges-v1-57fnz | grep service
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7qfx3 (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7qfx3 (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7qfx3 (ro)
17-07-06[14:29:43]:pachyderm:0$
17-07-06[14:29:48]:pachyderm:0$
17-07-06[14:29:48]:pachyderm:0$kc get secrets --all-namespaces
NAMESPACE     NAME                      TYPE                                  DATA      AGE
default       default-token-7qfx3       kubernetes.io/service-account-token   3         68d
default       pachyderm-token-m7tpm     kubernetes.io/service-account-token   3         1d
default       test-secret568a82d6c3d7   Opaque                                1         26d
default       test-secret7b61eec21ac2   Opaque                                1         52d
kube-public   default-token-28cjv       kubernetes.io/service-account-token   3         68d
kube-system   default-token-k8xbb       kubernetes.io/service-account-token   3         68d
kube-system   kube-dns-token-k0df7      kubernetes.io/service-account-token   3         68d
17-07-06[14:30:26]:pachyderm:0$kc delete secret default-token-7qfx3
secret "default-token-7qfx3" deleted
17-07-06[14:30:55]:pachyderm:0$kc get secrets --all-namespaces
NAMESPACE     NAME                      TYPE                                  DATA      AGE
default       default-token-gg6r3       kubernetes.io/service-account-token   3         2s
default       pachyderm-token-m7tpm     kubernetes.io/service-account-token   3         1d
default       test-secret568a82d6c3d7   Opaque                                1         26d
default       test-secret7b61eec21ac2   Opaque                                1         52d
kube-public   default-token-28cjv       kubernetes.io/service-account-token   3         68d
kube-system   default-token-k8xbb       kubernetes.io/service-account-token   3         68d
kube-system   kube-dns-token-k0df7      kubernetes.io/service-account-token   3         68d
17-07-06[14:30:57]:pachyderm:0$kc describe po/pipeline-edges-v1-57fnz | grep service
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7qfx3 (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7qfx3 (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7qfx3 (ro)
17-07-06[14:31:03]:pachyderm:0$kc delete  po/pipeline-edges-v1-57fnz 
pod "pipeline-edges-v1-57fnz" deleted
17-07-06[14:31:11]:pachyderm:0$kc get pod
NAME                      READY     STATUS    RESTARTS   AGE
etcd-4197107720-zvnpf     1/1       Running   2          1d
pachd-3722220436-m6kf3    1/1       Running   4          1d
pipeline-edges-v1-d9qng   2/2       Running   0          4s
17-07-06[14:31:15]:pachyderm:0$kc describe pipeline-edges-v1-d9qng | grep service
the server doesn't have a resource type "pipeline-edges-v1-d9qng"
17-07-06[14:31:24]:pachyderm:1$kc describe po/pipeline-edges-v1-d9qng | grep service
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-gg6r3 (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-gg6r3 (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-gg6r3 (ro)

@sjezewski sjezewski added this to the v1.6 milestone Sep 12, 2017

@dwhitena dwhitena added the docs label Sep 12, 2017

@sjezewski sjezewski removed this from the v1.6 milestone Oct 18, 2017

@JoeyZwicker

This comment has been minimized.

Show comment
Hide comment
@JoeyZwicker

JoeyZwicker Oct 20, 2017

Member

I think this is outdated. reopen if I'm wrong

Member

JoeyZwicker commented Oct 20, 2017

I think this is outdated. reopen if I'm wrong

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment