Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kind does not work with docker installed via snap due to TMPDIR usage #431

Closed
pixelshaded opened this Issue Apr 3, 2019 · 6 comments

Comments

Projects
None yet
2 participants
@pixelshaded
Copy link

commented Apr 3, 2019

What happened:
failed to copy kubeadm config to node

What you expected to happen:
create cluster successfully

How to reproduce it (as minimally and precisely as possible):
kind create cluster --loglevel=debug

Anything else we need to know?:

Environment:

  • go version: 1.12.1
  • kind version: (use kind version): 0.3.0-alpha
  • Kubernetes version: (use kubectl version): 1.14.0
  • Docker version: (use docker info): 18.06.1-ce
  • OS (e.g. from /etc/os-release): Ubuntu 18.10

Create cluster doesn't seem to get very far.

DEBU[17:54:41] Running: /snap/bin/docker [docker ps -q -a --no-trunc --filter label=io.k8s.sigs.kind.cluster --format {{.Names}}\t{{.Label "io.k8s.sigs.kind.cluster"}}] 
Creating cluster "kind" ...
DEBU[17:54:41] Running: /snap/bin/docker [docker inspect --type=image kindest/node:v1.13.4] 
INFO[17:54:41] Image: kindest/node:v1.13.4 present locally  
 ✓ Ensuring node image (kindest/node:v1.13.4) 🖼
DEBU[17:54:41] Running: /snap/bin/docker [docker info --format '{{json .SecurityOptions}}'] 
DEBU[17:54:41] Running: /snap/bin/docker [docker run -d --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname kind-control-plane --name kind-control-plane --label io.k8s.sigs.kind.cluster=kind --label io.k8s.sigs.kind.role=control-plane --entrypoint=/usr/local/bin/entrypoint --expose 39653 -p 127.0.0.1:39653:6443 kindest/node:v1.13.4 /sbin/init] 
DEBU[17:54:42] Running: /snap/bin/docker [docker exec --privileged kind-control-plane rm -f /etc/machine-id] 
DEBU[17:54:42] Running: /snap/bin/docker [docker exec --privileged kind-control-plane systemd-machine-id-setup] 
DEBU[17:54:42] Running: /snap/bin/docker [docker info --format '{{json .SecurityOptions}}'] 
DEBU[17:54:42] Running: /snap/bin/docker [docker exec --privileged kind-control-plane mount -o remount,ro /sys] 
DEBU[17:54:42] Running: /snap/bin/docker [docker exec --privileged kind-control-plane mount --make-shared /] 
DEBU[17:54:43] Running: /snap/bin/docker [docker exec --privileged kind-control-plane mount --make-shared /run] 
DEBU[17:54:43] Running: /snap/bin/docker [docker exec --privileged kind-control-plane mount --make-shared /var/lib/docker] 
DEBU[17:54:43] Running: /snap/bin/docker [docker kill -s SIGUSR1 kind-control-plane] 
DEBU[17:54:43] Running: /snap/bin/docker [docker exec --privileged -t kind-control-plane systemctl is-active docker] 
DEBU[17:54:43] Running: /snap/bin/docker [docker exec --privileged -t kind-control-plane systemctl is-active docker] 
DEBU[17:54:43] Running: /snap/bin/docker [docker exec --privileged -t kind-control-plane systemctl is-active docker] 
DEBU[17:54:44] Running: /snap/bin/docker [docker exec --privileged -t kind-control-plane systemctl is-active docker] 
DEBU[17:54:44] Running: /snap/bin/docker [docker exec --privileged kind-control-plane /bin/bash -c find /kind/images -name *.tar -print0 | xargs -0 -n 1 -P $(nproc) docker load -i] 
DEBU[17:54:47] Running: /snap/bin/docker [docker exec --privileged -t kind-control-plane cat /kind/version] 
 ✓ Preparing nodes 📦 
DEBU[17:54:47] Running: /snap/bin/docker [docker ps -q -a --no-trunc --filter label=io.k8s.sigs.kind.cluster --format {{.Names}}\t{{.Label "io.k8s.sigs.kind.cluster"}} --filter label=io.k8s.sigs.kind.cluster=kind] 
DEBU[17:54:47] Running: /snap/bin/docker [docker inspect -f {{index .Config.Labels "io.k8s.sigs.kind.role"}} kind-control-plane] 
DEBU[17:54:47] Running: /snap/bin/docker [docker exec --privileged -t kind-control-plane cat /kind/version] 
DEBU[17:54:48] Running: /snap/bin/docker [docker exec --privileged -t kind-control-plane mkdir -p /kind] 
DEBU[17:54:48] Running: /snap/bin/docker [docker cp /tmp/500180063 kind-control-plane:/kind/kubeadm.conf] 
 ✗ Creating kubeadm config 📜 
DEBU[17:54:48] Running: /snap/bin/docker [docker ps -q -a --no-trunc --filter label=io.k8s.sigs.kind.cluster --format {{.Names}}\t{{.Label "io.k8s.sigs.kind.cluster"}} --filter label=io.k8s.sigs.kind.cluster=kind] 
DEBU[17:54:48] Running: /snap/bin/docker [docker rm -f -v kind-control-plane] 
Error: failed to create cluster: failed to copy kubeadm config to node: failed to copy file to the node: exit status 1

@pixelshaded pixelshaded added the kind/bug label Apr 3, 2019

@BenTheElder

This comment has been minimized.

Copy link
Member

commented Apr 3, 2019

Did you install docker via snap?
Edit: per the logs: yes

@BenTheElder

This comment has been minimized.

Copy link
Member

commented Apr 3, 2019

Quick workaround is TMPDIR=$HOME kind creat cluster, the docker snap is a bit overly restrictive about filesystem access imho, we can't create tempfiles / tempdirs :(

@pixelshaded

This comment has been minimized.

Copy link
Author

commented Apr 3, 2019

That must have been it then. Worked after changing the TMPDIR to something the snap could access. Guessing if docker had been installed with --classic this wouldn't have happened.

@BenTheElder BenTheElder changed the title Failed to create cluster: failed to copy kubeadm config to node kind does not work with docker installed via snap due to TMPDIR usage Apr 3, 2019

@BenTheElder

This comment has been minimized.

Copy link
Member

commented Apr 3, 2019

/assign
/priority important soon

I've filed a PR to document the work-around, but in this particular case we can avoid writing that file to disk I think.

kind build ... commands on the other hand more inherently leverage temp directories, but many users may never touch these ... 🤔

@BenTheElder

This comment has been minimized.

Copy link
Member

commented Apr 3, 2019

#433 should avoid using $TMPDIR for kind create cluster at least 🙃

@BenTheElder

This comment has been minimized.

Copy link
Member

commented Apr 4, 2019

I just tested in a clean ubuntu VM with docker installed from snap, before #433 I hit this issue, but with the latest installed via go get -u sigs.k8s.io/kind I have no issues with kind create cluster.

the kind build ... commands may still be an issue, I'll track that seperately if anyone needs this. Those would be trickier

@BenTheElder BenTheElder closed this Apr 4, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.