Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

❗ Docker >=v20.10.5 (runc >= v1.0.0-rc93) is required for k3d v5.x ❗ #807

Closed
manumartin opened this issue Oct 18, 2021 · 15 comments
Closed
Assignees
Labels
bug Something isn't working docs Documentation
Milestone

Comments

@manumartin
Copy link

manumartin commented Oct 18, 2021

TL;DR: Docker >=v20.10.5 is required for k3d v5.x

What did you do

I tried to create a k3d cluster with k3d 5.x but it fails. Its working with 4.x

  • How was the cluster created?

    • sudo k3d cluster create MYCLUSTER --trace --verbose
  • What did you do afterwards?
    It seems like some kind of k3d/docker integration issue. I have reproduced the error message by using this docker-compose.yml:

test-container:
    image: rancher/k3d-tools:5.0.1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    tmpfs:
      - /var/run

It seems you can't mount the docker sock volume inside the /var/run tmpfs mount and k3d is failing when trying to do the same with the k3d-tools container.

What did you expect to happen

I expected the cluster to start normally

Screenshots or terminal output

❯ sudo k3d cluster create MYCLUSTER --trace --verbose

...

INFO[0000] Starting new tools node...                   
TRAC[0000] Creating node from spec
&{Name:k3d-MYCLUSTER-tools Role:noRole Image:docker.io/rancher/k3d-tools:5.0.0 Volumes:[k3d-MYCLUSTER-images:/k3d/images /var/run/docker.sock:/var/run/docker.sock] Env:[] Cmd:[] Args:[noop] Ports:map[] Restart:false Created: RuntimeLabels:map[app:k3d k3d.cluster:MYCLUSTER k3d.version:v5.0.0] K3sNodeLabels:map[] Networks:[k3d-MYCLUSTER] ExtraHosts:[] ServerOpts:{IsInit:false KubeAPI:<nil>} AgentOpts:{} GPURequest: Memory: State:{Running:false Status: Started:} IP:{IP:zero IP Static:false} HookActions:[]} 
TRAC[0000] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-MYCLUSTER-tools Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[noop] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3d-tools:5.0.0 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:MYCLUSTER k3d.role:noRole k3d.version:v5.0.0] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-MYCLUSTER-images:/k3d/images /var/run/docker.sock:/var/run/docker.sock] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0xc00020630f} NetworkingConfig:{EndpointsConfig:map[k3d-MYCLUSTER:0xc00024e000]}} 
DEBU[0000] Created container k3d-MYCLUSTER-tools (ID: 6aed150173f55ae288a9334302ccb6684dda8e9bdc1bc77af3454e4ab8743c64) 
DEBU[0000] Node k3d-MYCLUSTER-tools Start Time: 2021-10-18 17:15:55.925990393 +0200 CEST m=+0.122850186 
TRAC[0000] Starting node 'k3d-MYCLUSTER-tools'               
INFO[0000] Starting Node 'k3d-MYCLUSTER-tools'               
ERRO[0000] Failed to run tools container for cluster 'MYCLUSTER' 

...

INFO[0001] Starting existing tools node k3d-devo-tools... 
INFO[0001] Starting Node 'k3d-devo-tools'               
ERRO[0001] failed to gather environment information used for cluster creation: error starting existing tools node k3d-devo-tools: docker failed to start container for node 'k3d-devo-tools': Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting "/var/run/docker.sock" to rootfs at "/var/run/docker.sock" caused: stat /var/run/docker.sock: no such file or directory: unknown 
ERRO[0001] Failed to create cluster >>> Rolling Back 

Which OS & Architecture

  • Linux x86

Which version of k3d

k3d version v5.0.1
k3s version v1.21.5-k3s1 (default)`

Which version of docker

Client: Docker Engine - Community
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        2291f61
 Built:             Mon Dec 28 16:17:32 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
@manumartin manumartin added the bug Something isn't working label Oct 18, 2021
@manumartin
Copy link
Author

It doesn't happen anymore after upgrading from Docker 2.10.2 and containerd 1.4.3 to Docker 2.10.9 and containerd 1.4.11

@franpog859
Copy link

Hey! I'm facing similar issue on Mac:

$ k3d cluster create mycluster

INFO[0000] Prep: Network
INFO[0000] Created network 'k3d-mycluster'
INFO[0000] Created volume 'k3d-mycluster-images'
INFO[0000] Starting new tools node...
INFO[0001] Creating node 'k3d-mycluster-server-0'
INFO[0002] Pulling image 'docker.io/rancher/k3d-tools:5.0.1'
INFO[0003] Pulling image 'docker.io/rancher/k3s:latest'
INFO[0006] Starting Node 'k3d-mycluster-tools'
ERRO[0007] Failed to run tools container for cluster 'mycluster'
INFO[0031] Creating LoadBalancer 'k3d-mycluster-serverlb'
INFO[0033] Pulling image 'docker.io/rancher/k3d-proxy:5.0.1'
INFO[0041] Using the k3d-tools node to gather environment information
INFO[0041] Starting existing tools node k3d-mycluster-tools...
INFO[0041] Starting Node 'k3d-mycluster-tools'
ERRO[0041] failed to gather environment information used for cluster creation: error starting existing tools node k3d-mycluster-tools: docker failed to start container for node 'k3d-mycluster-tools': Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/run/host-services/docker.proxy.sock\\\" to rootfs \\\"/var/lib/docker/overlay2/7fd7682bb903b7768c61cc827cb4638f13568661f1c5b3ab007b9e289ebca6eb/merged\\\" at \\\"/var/run/docker.sock\\\" caused \\\"stat /run/host-services/docker.proxy.sock: no such file or directory\\\"\"": unknown
ERRO[0041] Failed to create cluster >>> Rolling Back
INFO[0041] Deleting cluster 'mycluster'
INFO[0041] Deleted k3d-mycluster-serverlb
INFO[0041] Deleted k3d-mycluster-server-0
INFO[0041] Deleted k3d-mycluster-tools
INFO[0041] Deleting cluster network 'k3d-mycluster'
INFO[0041] Deleting image volume 'k3d-mycluster-images'
FATA[0041] Cluster creation FAILED, all changes have been rolled back!

I have these installed and just upgraded with brew:

k3d version v5.0.1
docker engine version 19.03.13
docker containerd version v1.3.7
docker runc version 1.0.0-rc10
docker docker-init version 0.18.0

Do you maybe have a suggestion on what to do?

@manumartin
Copy link
Author

At first I downgraded k3d to 4.x and that fixed the problem but k3d 5.x with the newest docker/containerd versions also fixed it for me.

@franpog859
Copy link

Thanks, @manumartin ! Upgrading it with brew didn't work for me. Either something is wrong with linking the binary or there is not the latest version of Docker available here. Anyway, upgrading it just via GUI (sic!) worked fine :D

@tiswanso
Copy link

Hit the same issue on Ubuntu 18.0.4 with the same docker, containerd, and runc versions @franpog859 mentioned above.

Upgraded docker via apt upgrade docker-ce to resolve it. The k3d prereqs should mention version compatibility or min version requirements.

The resulting versions that worked for me:

Client: Docker Engine - Community
 Version:           20.10.10
 API version:       1.41
 Go version:        go1.16.9
 Git commit:        b485636
 Built:             Mon Oct 25 07:42:57 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.10
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.9
  Git commit:       e2f740d
  Built:            Mon Oct 25 07:41:06 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.11
  GitCommit:        5b46e404f6b9f661a205e28d59c982d3634148f8
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

@mjpowersjr
Copy link

I had the same exact problem, 4.x worked, upgrade to 5.x broke. As @tiswanso and others suggest, upgrading Docker fixed the issue. In case it helps - this is the version of Docker I was using on Ubuntu when in a broken state:

Docker version 20.10.5, build 55c4c88

@iwilltry42 iwilltry42 self-assigned this Nov 11, 2021
@iwilltry42 iwilltry42 added this to the v5.1.1 milestone Nov 11, 2021
@iwilltry42
Copy link
Member

iwilltry42 commented Nov 11, 2021

Hi @manumartin , thanks for opening this issue and thanks to the others for providing additional input and a "fix"/workaround.
The last update related to Docker in the tools package was a dependency update from docker v20.10.5 to v20.10.8 which already landed in k3d v4.4.8 in August 🤔 (#679) (similar for k3d itself)
I'm not sure if we can fix forward here, but I'll investigate today 👍

@iwilltry42
Copy link
Member

iwilltry42 commented Nov 11, 2021

Tested some versions of Docker now with k3d v5.1.0 and I can confirm that it only starts working with Docker v20.10.4+ 🤔

@tiswanso
Copy link

@iwilltry42 -- in my case just documenting the version dependencies would have spared me a decent amount of effort. I'm just using k3d in an environment I completely control so updating dependencies is not much of a problem. Ideally, the k3d doc has a version compatibility matrix.

@iwilltry42
Copy link
Member

iwilltry42 commented Nov 12, 2021

@tiswanso , I'm definitely going to add a requirement for Docker >=v20.10.5.
Unfortunately, there are just 5 million things to consider in compatibility matrices: OS (Linux/Mac/Windows) x Docker Variant (plain docker, Docker Machine, Docker for Mac, Docker for Windows, Docker for Windows with WSL2 backend) x Docker Version (what we face now) x k3d version (including dependency versions) x K3s version x Kernel version.

k3d x K3s I already had in CI/CD, which broke CI due to taking way to long to go through the E2E Tests with all K3s versions.
I guess Docker versions we can cover with a simple test case that should be run before merging a dependency update though ✔️

Update: After some investigation, I feel like it's caused by runc (seems like the update v1.0.0-rc92...v1.0.0-rc93 fixed it)

Update 2: Updated requirements on k3d.io and in the Readme: https://k3d.io/v5.1.0/#requirements

@iwilltry42 iwilltry42 added the docs Documentation label Nov 12, 2021
@iwilltry42 iwilltry42 pinned this issue Nov 14, 2021
@iwilltry42 iwilltry42 changed the title [BUG] k3d-tools container fails to start when trying to mount /var/run/docker.sock within the /var/run tmpfs mount Docker >=v20.10.5 is required for k3d v5.x Nov 14, 2021
@iwilltry42 iwilltry42 changed the title Docker >=v20.10.5 is required for k3d v5.x ❗ Docker >=v20.10.5 is required for k3d v5.x ❗ Nov 14, 2021
@iwilltry42 iwilltry42 changed the title ❗ Docker >=v20.10.5 is required for k3d v5.x ❗ ❗ Docker >=v20.10.5 is required for k3d v5.x ❗ Nov 14, 2021
@schnatterer
Copy link

Thanks for this issue, it saved me from a lot of debugging!

I can report an occurrence of this issue on docker 20.10.5 running on Ubuntu 20.04.2 LTS, which seems to have been shipped with runc 1.0.0-rc92 😬

@iwilltry42 wouldn't it make more sense to clarify the requirement to "a docker version that uses runc >= v1.0.0-rc93?

$ docker version
Client: Docker Engine - Community
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:18:20 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:16:15 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ lsb_release -a
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.2 LTS
Release:	20.04
Codename:	focal

@iwilltry42 iwilltry42 changed the title ❗ Docker >=v20.10.5 is required for k3d v5.x ❗ ❗ Docker >=v20.10.5 (runc >= v1.0.0-rc93) is required for k3d v5.x ❗ Jan 12, 2022
@iwilltry42
Copy link
Member

iwilltry42 commented Jan 12, 2022

@schnatterer I didn't expect docker to ship the same exact version with a different version of runc to be honest.
I'll update the notes accordingly to reflect this.
Update: e82810a

@schnatterer
Copy link

@iwilltry42 yeah, I was wondering about this myself. Thanks for updating.

jmaupetit added a commit to openfun/arnold-apps that referenced this issue Apr 11, 2022
Following Arnold's recent recommendations, we've upgraded k3d to the
latest 4.x release as the 5.x+ releases are not compatible with
installed `runc` version.

For more information please refer to:
- openfun/arnold#733
- k3d-io/k3d#807

test docker executor

rollback to the current ubuntu image

Use the same circle version as Arnold

Setup remote docker

Upgrade k3d/kubectl releases

restore machine executor

Restore comment
jmaupetit added a commit to openfun/arnold-apps that referenced this issue Apr 12, 2022
Following Arnold's recent recommendations, we've upgraded k3d to the
latest 4.x release as the 5.x+ releases are not compatible with
installed `runc` version.

For more information please refer to:
- openfun/arnold#733
- k3d-io/k3d#807

test docker executor

rollback to the current ubuntu image

Use the same circle version as Arnold

Setup remote docker

Upgrade k3d/kubectl releases

restore machine executor

Restore comment
jmaupetit added a commit to openfun/arnold-apps that referenced this issue Apr 12, 2022
Following Arnold's recent recommendations, we've upgraded k3d to the
latest 4.x release as the 5.x+ releases are not compatible with
installed `runc` version.

For more information please refer to:
- openfun/arnold#733
- k3d-io/k3d#807

test docker executor

rollback to the current ubuntu image

Use the same circle version as Arnold

Setup remote docker

Upgrade k3d/kubectl releases

restore machine executor

Restore comment
jmaupetit added a commit to openfun/arnold-apps that referenced this issue Apr 12, 2022
Following Arnold's recent recommendations, we've upgraded k3d to the
latest 4.x release as the 5.x+ releases are not compatible with
installed `runc` version.

For more information please refer to:
- openfun/arnold#733
- k3d-io/k3d#807

test docker executor

rollback to the current ubuntu image

Use the same circle version as Arnold

Setup remote docker

Upgrade k3d/kubectl releases

restore machine executor

Restore comment
@regunakyle
Copy link

@schnatterer I didn't expect docker to ship the same exact version with a different version of runc to be honest. I'll update the notes accordingly to reflect this. Update: e82810a

Sorry for bumping.

Do I need to check the version of both Docker and runc, or just runc?
My Synology NAS has:

  • Docker 20.10.3
  • runc v1.0.0-rc93
  • containerd v1.4.3

Not sure if it can run K3d.

@iwilltry42
Copy link
Member

@schnatterer I didn't expect docker to ship the same exact version with a different version of runc to be honest. I'll update the notes accordingly to reflect this. Update: e82810a

Sorry for bumping.

Do I need to check the version of both Docker and runc, or just runc?
My Synology NAS has:

  • Docker 20.10.3
  • runc v1.0.0-rc93
  • containerd v1.4.3

Not sure if it can run K3d.

Hmm.. as far as I could tell, runc was the issue here, so that should be enough.
I'm not certain though, so better just try 🤔

lahabana added a commit to lahabana/kuma that referenced this issue Feb 8, 2023
A long time has passed and the reason for this was: k3d-io/k3d#807

We should be able to use the builtin runc instead

Signed-off-by: Charly Molter <charly.molter@konghq.com>
lahabana added a commit to kumahq/kuma that referenced this issue Feb 22, 2023
)

A long time has passed and the reason for this was: k3d-io/k3d#807

We should be able to use the builtin runc instead
---------

Signed-off-by: Charly Molter <charly.molter@konghq.com>
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@iwilltry42 iwilltry42 unpinned this issue Jul 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working docs Documentation
Projects
None yet
Development

No branches or pull requests

7 participants