Unimplemented: failed to start shim: start failed: unsupported shim version (3): not implemented

[ericzundel]

Unfortunately, I seem to have just broken my github self-hosted runners by running apt-get update && apt-get upgrade -y on my ubuntu 24.04 (Noble) hosts. The error I get when trying to run the docker inside docker is:

docker: Error response from daemon: failed to create task for container: Unimplemented: failed to start shim: start failed: unsupported shim version (3): not implemented

docker version on my host:

$ docker -v
Docker version 29.2.1, build a5c7197

My docker image is:

FROM nestybox/ubuntu-noble-docker:latest

(I just re-pulled it and rebuilt with --no-cache to be sure)

And Sysbox is:

RUN curl -LO https://github.com/nestybox/sysbox/releases/download/v0.6.7/sysbox-ce_0.6.7.linux_amd64.deb
RUN apt install ./sysbox-ce_0.6.7.linux_amd64.deb -y

I have rebooted, rebuilt, and restarted the containers a number of times.

This setup has been working for a number of months, I started upgrading today after I had a mismatch with the gcloud utility. Who knows if I fixed that, because now this problem crops up before I get to the point where I can test the upgraded version of gcloud-cli.

[ericzundel]

WORKAROUND: Pin the docker version inside your container

Inside my container, I did an apt upgrade at some point, and this upgraded docker too far.

Looks like this might be the upgrade that killed it docker-ce:amd64 (5:29.1.5-1~ubuntu.24.04~noble, 5:29.2.1-1~ubuntu.24.04~noble)

I downgraded my host back to 5:29.1.5-1ubuntu.24.04noble

I still saw the problem, so I went into my container and pinned docker inside the container at the version that was distributed with nestybox/ubuntu-noble-docker-latest

FROM nestybox/ubuntu-noble-docker:latest
# sha256:b6af70f204a27becd69ac8bd2f2ff9aad28ad0f3ba97d77ae3a4428212ce1554

RUN apt-mark hold docker-ce docker-ce-cli containerd.io

which reports version:

Docker version 27.1.1, build 6312585

Now the container seems to start again.

[linghengqian]

• I also noticed this bug.
• If the host machine's Ubuntu 24.04 WSL has Rootful Docker Engine 29.3.0 and Sysbox CE 0.7.0 installed, and then executes,

docker run --runtime=sysbox-runc --rm -d --name my_cont nestybox/ubuntu-noble-systemd-docker:latest
docker exec -it my_cont /bin/bash
apt update && apt upgrade --assume-yes
docker info
docker run hello-world

• This will result in the following error log:

root@b1dc35536901:/# docker info
Client: Docker Engine - Community
 Version:    29.3.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.31.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.1.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 1
 Server Version: 27.3.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc version: v1.3.4-0-gd6d73eb8
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.6.87.2-microsoft-standard-WSL2
 Operating System: Ubuntu 24.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 18.55GiB
 Name: b1dc35536901
 ID: e45a4b20-bed4-4ee7-8f52-3383a4d49284
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

root@b1dc35536901:/# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
17eec7bbc9d7: Pull complete
Digest: sha256:ef54e839ef541993b4e87f25e752f7cf4238fa55f017957c2eb44077083d7a6a
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: failed to create task for container: Unimplemented: failed to start shim: start failed: unsupported shim version (3): not implemented

Run 'docker run --help' for more information
root@b1dc35536901:/#

• To be honest, I'm quite confused as to why executing apt update && apt upgrade --assume-yes inside the container only updates the Docker CLI and not the Docker daemon. The content of https://github.com/nestybox/dockerfiles/blob/master/ubuntu-noble-systemd-docker/Dockerfile gives me the impression that the Docker daemon was installed via apt.
• If I don't execute apt update && apt upgrade --assume-yes, I obviously won't encounter any errors.