Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

elasticsearch failing due to /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory #10620

Closed
moix opened this Issue Jul 1, 2015 · 17 comments

Comments

Projects
None yet
6 participants
@moix
Copy link

commented Jul 1, 2015

It is happening in a 0.19 installation on centos servers.

docker version is:
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): ba1f6c3/1.6.2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): ba1f6c3/1.6.2
OS/Arch (server): linux/amd64

I disables selinux as commented #9208 even running a docker with the fix.

Still, elasticsearch_logging_discovery is failing at c, err := client.NewInCluster() with following error:

I0701 14:01:55.203543 5 elasticsearch_logging_discovery.go:42] Kubernetes Elasticsearch logging discovery
F0701 14:01:55.203817 5 elasticsearch_logging_discovery.go:46] Failed to make client: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory

@satnam6502

This comment has been minimized.

Copy link
Contributor

commented Jul 2, 2015

Interesting. I suspect something is broken in the 0.19 installation. I am pretty sure Elasticsearch logging works from HEAD -- I have made working builds for it many times this week. I wonder if other 0.19 cluster services are broken?

@moix

This comment has been minimized.

Copy link
Author

commented Jul 2, 2015

maybe something is wrong in my installation but I checked everything and couldn't find a reason. Checking the container I see /var/run/secrets/ volume is created but empty, any idea or clue what can I check?

@satnam6502

This comment has been minimized.

Copy link
Contributor

commented Jul 2, 2015

Are you in a hurry to get this looked at? If not, I can look at it when I get back to the office on Monday. Otherwise happy to look at it when I get a moment over the coming days (holiday in the US).

@moix

This comment has been minimized.

Copy link
Author

commented Jul 2, 2015

Oh dont worry, I can wait of course :)
It was just in case I can investigate it further in my environment and provide more information.

Thanks!

@liggitt

This comment has been minimized.

Copy link
Member

commented Jul 2, 2015

@moix can you include the pod spec (kubectl get pod <pod> -o json) and the results of running mount from within the container? It would also be helpful to know what specific version of docker you are using (1.6.2-x?) if installed with yum.

@moix

This comment has been minimized.

Copy link
Author

commented Jul 2, 2015

Hi, the exact version of docker is docker-1.6.2-14.el7.centos.x86_64.

And output from mount is:

/dev/mapper/docker-253:1-1074341495-a52c6a7e3514661122451cd2932ac063fe6da366beff115ae11c3cecb07f304e on / type ext4 (rw,relatime,seclabel,discard,stripe=16,data=ordered)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,seclabel,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=666)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel,size=65536k)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime,seclabel)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime,seclabel)
tmpfs on /run/secrets type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel)
/dev/mapper/vg00-lvroot on /data type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vg00-lvroot on /dev/termination-log type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vg00-lvroot on /etc/resolv.conf type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vg00-lvroot on /etc/hostname type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vg00-lvroot on /etc/hosts type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
proc on /proc/asound type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/fs type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
tmpfs on /proc/kcore type tmpfs (rw,nosuid,seclabel,mode=755)
tmpfs on /proc/timer_stats type tmpfs (rw,nosuid,seclabel,mode=755)
@liggitt

This comment has been minimized.

Copy link
Member

commented Jul 2, 2015

I don't see a service account token getting mounted at all. Do you have the pod spec to see if one was set up?

Also, this looks like this might shadow the service account token mount:

tmpfs on /run/secrets type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel)
@moix

This comment has been minimized.

Copy link
Author

commented Jul 2, 2015

The full pod resource is:

{
    "kind": "Pod",
    "apiVersion": "v1",
    "metadata": {
        "name": "elasticsearch-logging-v1-5ni7v",
        "generateName": "elasticsearch-logging-v1-",
        "namespace": "default",
        "selfLink": "/api/v1/namespaces/default/pods/elasticsearch-logging-v1-5ni7v",
        "uid": "ba971d48-1ff9-11e5-9ba4-5254000597d0",
        "resourceVersion": "69404",
        "creationTimestamp": "2015-07-01T14:01:54Z",
        "labels": {
            "k8s-app": "elasticsearch-logging",
            "kubernetes.io/cluster-service": "true",
            "version": "v1"
        },
        "annotations": {
            "kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicationController\",\"namespace\":\"default\",\"name\":\"elasticsearch-logging-v1\",\"uid\":\"ba9651ec-1ff9-11e5-9ba4-5254000597d0\",\"apiVersion\":\"v1\",\"resourceVersion\":\"69371\"}}"
        }
    },
    "spec": {
        "volumes": [
            {
                "name": "es-persistent-storage",
                "emptyDir": {}
            }
        ],
        "containers": [
            {
                "name": "elasticsearch-logging",
                "image": "gcr.io/google_containers/elasticsearch:1.4",
                "ports": [
                    {
                        "name": "es-port",
                        "containerPort": 9200,
                        "protocol": "TCP"
                    },
                    {
                        "name": "es-transport-port",
                        "containerPort": 9300,
                        "protocol": "TCP"
                    }
                ],
                "resources": {},
                "volumeMounts": [
                    {
                        "name": "es-persistent-storage",
                        "mountPath": "/data"
                    }
                ],
                "terminationMessagePath": "/dev/termination-log",
                "imagePullPolicy": "IfNotPresent"
            }
        ],
        "restartPolicy": "Always",
        "dnsPolicy": "ClusterFirst",
        "serviceAccount": "default",
        "nodeName": "edosdpci-server9"
    },
    "status": {
        "phase": "Running",
        "conditions": [
            {
                "type": "Ready",
                "status": "True"
            }
        ],
        "hostIP": "172.29.45.95",
        "podIP": "18.16.92.12",
        "startTime": "2015-07-01T14:01:55Z",
        "containerStatuses": [
            {
                "name": "elasticsearch-logging",
                "state": {
                    "running": {
                        "startedAt": "2015-07-01T14:01:55Z"
                    }
                },
                "lastState": {},
                "ready": true,
                "restartCount": 0,
                "image": "gcr.io/google_containers/elasticsearch:1.4",
                "imageID": "docker://b7747c8e3d62a62fadbfd803379f36fd53b6e9e13be5e4ce5d9dd8784853e587",
                "containerID": "docker://a52c6a7e3514661122451cd2932ac063fe6da366beff115ae11c3cecb07f304e"
            }
        ]
    }
}
@liggitt

This comment has been minimized.

Copy link
Member

commented Jul 2, 2015

@moix, ok... a couple issues.

  1. It looks like this pod was created before the service account API token was provisioned (which can take a few seconds immediately after a new namespace is created). #10523 prevents pods from being created until the token is ready, so the replication controller will keep trying to create the pod. Two things you can check in the meantime:
    1. Do you have a generated token in your namespace after a few seconds? kubectl get secrets should show it, and kubectl get serviceaccounts should show a default service account with one secret.
    2. If you delete the pod and let the replication controller recreate it, does the new one have a volumemount for a service account token?
  2. The mount on /var/run/secrets is concerning, since I'm wondering if that will shadow the API token mount.
@moix

This comment has been minimized.

Copy link
Author

commented Jul 2, 2015

thanks @liggitt ,

  • there's no token generated, kubectl get secrets shows empty list
  • default service account is present has no secrets generated, actually it was not generated automatically, I generated it during the setup of kubernetes.
  • deleting the pod forces the rc to recreate but same as above, no service account so volume is same status
@liggitt

This comment has been minimized.

Copy link
Member

commented Jul 2, 2015

How are you starting the controller manager? Are you passing in the service_account_private_key_file argument to enable generation of service account API tokens?

@moix

This comment has been minimized.

Copy link
Author

commented Jul 2, 2015

no, it runs without service_account_private_key_file, though it uses a self generated key. Is it needed then?

@liggitt

This comment has been minimized.

Copy link
Member

commented Jul 2, 2015

The controllers that provision service accounts and service account tokens run in controller manager, so that has to be started, and provided the private key to use to sign the service account tokens.

See hack/local-up-cluster.sh for an example of:

  • generating a public/private key
  • giving the API server to use the public key to verify tokens
  • giving the controller manager the private key to generate/sign tokens

For the salt-driven startups, I think the server's private key does double-duty as the signing/verifying key.

@moix

This comment has been minimized.

Copy link
Author

commented Jul 15, 2015

Thanks @liggitt, I took a look to local-up-cluster.sh and salt/generate-cert/make-ca-cert.sh and could setup it correctly.
But, as commented at #10265, the ca.cert was missing, only the token file was mounted. It was an issue in 0.19 and upgrading to 1.0 fixed it.

@moix moix closed this Jul 15, 2015

@robin-hunter

This comment has been minimized.

Copy link

commented Nov 13, 2016

OK, I have tried to find a solution to this problem on version 1.4, again running on a bare metal cluster but without success.

My apiserver is started using the command:

km apiserver --address=192.168.1.35 --etcd-servers=http://192.168.1.35:4001 --service-cluster-ip-range=10.10.10.0/24 --port=8888 --client-ca-file=/opt/kubernetes/certs/ca.crt --tls-cert-file=/opt/kubernetes/certs/server.crt --tls-private-key-file=/opt/kubernetes/certs/server.key --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota

My controller is started using the command:

km controller-manager --master=192.168.1.35:8888 --cloud-provider=mesos --cloud-config=/etc/kubernetes-mesos/mesos-cloud.conf --cluster-signing-cert-file=/opt/kubernetes/certs/server.pem --root-ca-file=/opt/kubernetes/certs/ca.crt --service-account-private-key-file=/opt/kubernetes/certs/server.key

My docker version is: Docker version 1.12.1, build 23cf638

The output of the pod mount is:

/dev/mapper/docker-253:0-102314-7b6ef954239b2790b271e01ecfed900d13ce333b366d49f7819961029ee282f0 on / type xfs (rw,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,relatime,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (ro,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/net_cls type cgroup (ro,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (ro,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (ro,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpuacct,cpu type cgroup (ro,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/freezer type cgroup (ro,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,memory)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
/dev/mapper/centos-root on /dev/termination-log type xfs (rw,relatime,attr2,inode64,noquota)
/dev/mapper/centos-root on /etc/resolv.conf type xfs (rw,relatime,attr2,inode64,noquota)
/dev/mapper/centos-root on /etc/hostname type xfs (rw,relatime,attr2,inode64,noquota)
/dev/mapper/centos-root on /etc/hosts type xfs (rw,relatime,attr2,inode64,noquota)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
/dev/mapper/centos-root on /run/secrets/kubernetes.io/serviceaccount type xfs (ro,relatime,attr2,inode64,noquota)
proc on /proc/asound type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/fs type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
tmpfs on /proc/kcore type tmpfs (rw,nosuid,mode=755)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,mode=755)
tmpfs on /proc/timer_stats type tmpfs (rw,nosuid,mode=755)
tmpfs on /proc/sched_debug type tmpfs (rw,nosuid,mode=755)

Get secrets returns

NAME TYPE DATA AGE
default-token-routl kubernetes.io/service-account-token 3 5d

My default serviceaccount is:

Name: default
Namespace: default
Labels:

Image pull secrets:

Mountable secrets: default-token-routl

Tokens: default-token-routl

Despite this I get no token file and I cannot get any service accounts working, this includes dashboard, DNS or any I try to create myself.

@liggitt

This comment has been minimized.

Copy link
Member

commented Nov 13, 2016

/dev/mapper/centos-root on /run/secrets/kubernetes.io/serviceaccount type xfs (ro,relatime,attr2,inode64,noquota)

That's likely to be a general issue with secret volume mounts. @pmorie, any suggestions for debugging that?

@anteyk

This comment has been minimized.

Copy link

commented Apr 6, 2017

Hi team.
I faced with simillar to robin-hunter's issue
kubectl describe serviceaccounts
Name: default
Namespace: default
Labels:
Image pull secrets:
Mountable secrets: default-token-9sm84
Tokens: default-token-9sm84

kubectl get secrets
NAME TYPE DATA AGE
default-token-9sm84 kubernetes.io/service-account-token 3 22h

default-token-9sm84 contain 3 files (ca.crt, namespace, token)
mount inside pod (it has only one container) returns
/dev/md1 on /run/secrets/kubernetes.io/serviceaccount type ext4 (ro,relatime,seclabel,data=ordered)

but folder /run/secrets/kubernetes.io/serviceaccount type ext4 is empty.

My OS Oracle Linux Server release 6.8.
Docker version
Client:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built:
OS/Arch: linux/amd64
Server:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built:
OS/Arch: linux/amd64
Kubernetes client and server v1.5.6, GoVersion:"go1.7.4"
@liggitt Should I create new issue as it's quite differs from original issue?
@robin-hunter did you found any solution?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.