docker falling to run wasm containers with containerd shims

[emirhandurmus]

Description

Since from last two version of docker, containerd shims doesn't work with docker.

There are also two confirmed issue form spinframework and rancher desktop below.
#rancher-sandbox/rancher-desktop#9476
#spinframework/containerd-shim-spin#378

/etc/docker/daemon.json

{
    "storage-driver": "overlayfs",
    "features": {
        "containerd-snapshotter": true,
        "cdi": true
    },
    "cdi-spec-dirs": [
        "/etc/cdi/",
        "/var/run/cdi"
    ],
    "runtimes": {
        "io.containerd.wasmedge.v1": {
            "path": "/usr/local/bin/containerd-shim-wasmedge-v1"
        },
        "io.containerd.spin.v2": {
            "path": "/usr/local/bin/containerd-shim-spin-v2"
        }
    }
}

root@test:~# ls -l /usr/local/bin/containerd*
-rwxr-xr-x 1 1001 1001  85087584 Nov  1 22:03 /usr/local/bin/containerd-shim-spin-v2
-rwxr-xr-x 1 root root 124977784 Mar  9  2025 /usr/local/bin/containerd-shim-wasmedge-v1
-rwxr-xr-x 1 root root  32763464 Mar  9  2025 /usr/local/bin/containerd-shim-wasmer-v1
-rwxr-xr-x 1 root root  28913104 Mar  9  2025 /usr/local/bin/containerd-shim-wasmtime-v1

root@ip-10-0-37-151:/etc/docker# docker run --name spin --runtime io.containerd.spin.v2 --platform wasi/wasm --publish 8080:80 ghcr.io/spinframework/containerd-shim-spin/examples/spin-rust-hello:v0.22.0 --command /

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v2.task/moby/133254fbac340346273b7b67878e060c1c9a3ee6dbd081b245033b14ccf59999/log.json: no such file or directory): /usr/local/bin/containerd-shim-spin-v2 did not terminate successfully: exit status 134: unknown

Run 'docker run --help' for more information

Same issue also with runwasi wasm shims

root@ip-10-0-37-151:/home/ubuntu# docker run --rm --runtime=io.containerd.wasmedge.v1 --platform=wasi/wasm secondstate/rust-example-hello:latest
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v2.task/moby/0a10974605804b780da2123b752e94ce953f29580b9e5bd633147e6b8c626e0e/log.json: no such file or directory): /usr/local/bin/containerd-shim-wasmedge-v1 did not terminate successfully: exit status 134: unknown

Run 'docker run --help' for more information

Reproduce

/etc/docker/daemon.json

{
    "storage-driver": "overlayfs",
    "features": {
        "containerd-snapshotter": true,
        "cdi": true
    },
    "cdi-spec-dirs": [
        "/etc/cdi/",
        "/var/run/cdi"
    ],
    "runtimes": {
        "io.containerd.wasmedge.v1": {
            "path": "/usr/local/bin/containerd-shim-wasmedge-v1"
        },
        "io.containerd.spin.v2": {
            "path": "/usr/local/bin/containerd-shim-spin-v2"
        }
    }
}

wget https://github.com/spinframework/containerd-shim-spin/releases/download/v0.22.0/containerd-shim-spin-v2-linux-x86_64.tar.gz
tar -xvf containerd-shim-spin-v2-linux-x86_64.tar.gz
sudo mv containerd-shim-spin-v2 /usr/local/bin/
sudo chmod +x /usr/local/bin/containerd-shim-spin-v2

docker run --name spin --runtime io.containerd.spin.v2 --platform wasi/wasm --publish 8080:80 ghcr.io/spinframework/containerd-shim-spin/examples/spin-rust-hello:v0.22.0 --command /

git clone https://github.com/containerd/runwasi.git

cd runwasi
./scripts/setup-linux.sh
make build-wasmedge
INSTALL="sudo install" LN="sudo ln -sf" make install-wasmedge

docker run --rm --runtime=io.containerd.wasmedge.v1 --platform=wasi/wasm secondstate/rust-example-hello:latest

Expected behavior

$ docker run --rm --runtime=io.containerd.wasmedge.v1 --platform=wasi/wasm secondstate/rust-example-hello:latest
Hello WasmEdge!

docker version

root@ip-10-0-37-151:/etc/docker# docker version
Client: Docker Engine - Community
 Version:           29.1.2
 API version:       1.46 (downgraded from 1.52)
 Go version:        go1.25.5
 Git commit:        890dcca
 Built:             Tue Dec  2 21:55:19 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.0.1
  API version:      1.46 (minimum version 1.24)
  Go version:       go1.21.11
  Git commit:       ff1e2c0
  Built:            Mon Jun 24 14:58:03 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.0
  GitCommit:        1c4457e00facac03ce1d75f7b6777a7a851e5c41
 runc:
  Version:          1.3.4
  GitCommit:        v1.3.4-0-gd6d73eb8
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

root@ip-10-0-37-151:/etc/docker# docker info
Client: Docker Engine - Community
 Version:    29.1.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.30.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.0.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  model: Docker Model Runner (Docker Inc.)
    Version:  v1.0.3
    Path:     /usr/libexec/docker/cli-plugins/docker-model

Server:
 Containers: 13
  Running: 3
  Paused: 0
  Stopped: 10
 Images: 17
 Server Version: 27.0.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi/
  /var/run/cdi
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.spin.v2 io.containerd.wasmedge.v1 io.containerd.wasmer.v1 io.containerd.wasmtime.v1
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1c4457e00facac03ce1d75f7b6777a7a851e5c41
 runc version: v1.3.4-0-gd6d73eb8
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.14.0-1016-aws
 Operating System: Ubuntu 24.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 1.918GiB
 Name: ip-10-0-37-151
 ID: 0add84d9-353e-412b-acb6-abada691df55
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

[thaJeztah]

Thanks for reporting, and for the reproduction steps; I did some very quick checks and was able to reproduce the error; I also tried downgrading containerd (in case the problem was in the updated containerd binaries), but after downgrading to a v1.x version of containerd, I still had the same;

apt-get install -y --allow-downgrades containerd.io=1.7.27-1

docker version
...
 containerd:
  Version:          1.7.27
  GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef

One thing that did change between 27.x and 28.x is that the moby code started to use the containerd v2 module (client / sdk);

moby/vendor.mod

Line 28 in 7744655

github.com/containerd/containerd v1.7.24

moby/vendor.mod

Line 30 in 4e04130

github.com/containerd/containerd/v2 v2.1.4

If someone would start bisecting, that change may be a good one to look at; 0aa8fe0 (#48872)

[thaJeztah]

If someone would start bisecting, that change may be a good one to look at; 0aa8fe0 (#48872)

So, I actually tried, using the commit before that, but no dice so far;

git clone https://github.com/moby/moby.git
cd moby
git checkout 264801efb1f8676a6a4b1b6e023fbdbb56cf28e6
make dynbinary


command -v dockerd
/usr/bin/dockerd

systemctl stop docker

mv /usr/bin/dockerd /usr/bin/dockerd-orig
cp ./bundles/dynbinary/dockerd /usr/bin/dockerd
systemctl start docker

docker version
...
Server:
 Engine:
  Version:          dev
  API version:      1.48 (minimum version 1.24)
  Go version:       go1.23.4
  Git commit:       264801efb1f8676a6a4b1b6e023fbdbb56cf28e6
  Built:            Wed Dec 10 11:13:04 2025
  OS/Arch:          linux/amd64
  Experimental:     false

docker rm -fv spin
docker run --name spin --runtime io.containerd.spin.v2 --platform wasi/wasm --publish 8080:80 ghcr.io/spinframework/containerd-shim-spin/examples/spin-rust-hello:v0.22.0 --command /
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v2.task/moby/d6845733cc715c577889e898fe5164adb23150008f8b5f5f43820a3ce2df9570/log.json: no such file or directory): /usr/local/bin/containerd-shim-spin-v2 did not terminate successfully: exit status 134: unknown

[emirhandurmus]

@thaJeztah thanks for the response.

for docker version v27 you can you please try with below daemon.json config without additional runtime config

/etc/docker/daemon.json

{
    "storage-driver": "overlayfs",
    "features": {
        "containerd-snapshotter": true,
        "cdi": true
    },
    "cdi-spec-dirs": [
        "/etc/cdi/",
        "/var/run/cdi"
    ]
}

and then run

docker run --name spin --runtime io.containerd.spin.v2 --platform wasi/wasm --publish 8080:80 ghcr.io/spinframework/containerd-shim-spin/examples/spin-rust-hello:v0.22.0 --command /

normally we dont need to add containerd shims in daemon configuration. but when i starting to get this error. i also try adding it.

[emirhandurmus]

When engine downgraded to V27 it works.

root@ip-10-0-37-151:/home/ubuntu# docker version
Client: Docker Engine - Community
 Version:           29.1.2
 API version:       1.46 (downgraded from 1.52)
 Go version:        go1.25.5
 Git commit:        890dcca
 Built:             Tue Dec  2 21:55:19 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.0.1
  API version:      1.46 (minimum version 1.24)
  Go version:       go1.21.11
  Git commit:       ff1e2c0
  Built:            Mon Jun 24 14:58:03 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.0
  GitCommit:        1c4457e00facac03ce1d75f7b6777a7a851e5c41
 runc:
  Version:          1.3.4
  GitCommit:        v1.3.4-0-gd6d73eb8
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

root@ip-10-0-37-151:/home/ubuntu# cat /etc/docker/daemon.json 
{
    "storage-driver": "overlayfs",
    "features": {
        "containerd-snapshotter": true,
        "cdi": true
    },
    "cdi-spec-dirs": [
        "/etc/cdi/",
        "/var/run/cdi"
    ]
}
root@ip-10-0-37-151:/home/ubuntu# docker run   --runtime=io.containerd.spin.v2   --platform=wasi/wasm   ghcr.io/spinframework/containerd-shim-spin/examples/spin-rust-hello:v0.20.0   "/"
Preparing Wasm modules is taking a few seconds...


Serving http://0.0.0.0:80
Available Routes:
  hello: http://0.0.0.0:80/hello
  go-hello: http://0.0.0.0:80/go-hello

with engine v29

root@ip-10-0-37-151:/home/ubuntu# docker version
Client: Docker Engine - Community
 Version:           29.1.2
 API version:       1.52
 Go version:        go1.25.5
 Git commit:        890dcca
 Built:             Tue Dec  2 21:55:19 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          29.1.2
  API version:      1.52 (minimum version 1.44)
  Go version:       go1.25.5
  Git commit:       de45c2a
  Built:            Tue Dec  2 21:55:19 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.0
  GitCommit:        1c4457e00facac03ce1d75f7b6777a7a851e5c41
 runc:
  Version:          1.3.4
  GitCommit:        v1.3.4-0-gd6d73eb8
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

root@ip-10-0-37-151:/home/ubuntu# docker run   --runtime=io.containerd.spin.v2   --platform=wasi/wasm   ghcr.io/spinframework/containerd-shim-spin/examples/spin-rust-hello:v0.20.0   "/"

container hanged in "created" stage.

Dec 10 11:47:53 ip-10-0-37-151 containerd[2230256]: time="2025-12-10T11:47:53.481174326Z" level=error msg="failed to load runtime info" error="failed to run [/usr/local/bin/containerd-shim-spin-v2 -info]: signal: aborted (core dumped) (stderr: \"\\nthread 'main' panicked at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/containerd-shimkit-0.1.1/src/sandbox/cli.rs:208:18:\\nFailed to initialize logger: IoError { context: \\\"failed to init logger\\\", err: Os { code: 2, kind: NotFound, message: \\\"No such file or directory\\\" } }\\nnote: run with `RUST_BACKTRACE=1` environment variable to display a backtrace\\n\")" runtime=io.containerd.spin.v2
Dec 10 11:47:53 ip-10-0-37-151 containerd[2230256]: time="2025-12-10T11:47:53.490776925Z" level=info msg="connecting to shim 42f29f5986348d29d82bc913be1cdc4c102d3e948b1cc27138b692220b137074" address="unix:///run/containerd/s/f416b9d92e01fc6e1a1863362d62e5e6445de8f2f0b12bdd56402169929770a1" namespace=moby protocol=ttrpc version=2

[thaJeztah]

I think I found the regression point;

OK 6ac445c
KO 933fcc9

6ac445c...933fcc9

Re-remove the SetKey OCI prestart hook

• part of Allocate IPv6 addresses after detecting IPv6 support #47406

cat /etc/docker/daemon.json
{
    "storage-driver": "overlayfs",
    "features": {
        "containerd-snapshotter": true
    }
}


git checkout 933fcc981418bd6994abd76701a2833789e143f5
make dynbinary && systemctl stop docker && cp ./bundles/dynbinary/dockerd /usr/bin/dockerd && systemctl start docker && docker version
docker rm -fv spin && docker run --name spin --runtime=io.containerd.spin.v2 --platform wasi/wasm ghcr.io/spinframework/containerd-shim-spin/examples/spin-rust-hello:v0.22.0 --command /
# hangs

git checkout 6ac445c42badc98b416baa32cca5d6f8b712b359
make dynbinary && systemctl stop docker && cp ./bundles/dynbinary/dockerd /usr/bin/dockerd && systemctl start docker && docker version
docker rm -fv spin && docker run --name spin --runtime=io.containerd.spin.v2 --platform wasi/wasm ghcr.io/spinframework/containerd-shim-spin/examples/spin-rust-hello:v0.22.0 --command /
spin

Serving http://0.0.0.0:80
Available Routes:
  hello: http://0.0.0.0:80/hello
  go-hello: http://0.0.0.0:80/go-hello

cc @robmry

[thaJeztah]

That's not showing the OCI runtime error though, so wondering now if it's a separate issue; the above is without the custom runtime configured in daemon.json. Changing the daemon.json to this;

{
    "storage-driver": "overlayfs",
    "features": {
        "containerd-snapshotter": true
    },
    "runtimes": {
        "io.containerd.spin.v2": {
            "path": "/usr/local/bin/containerd-shim-spin-v2"
        }
    }
}

Will produce the OCI error;

docker rm -fv spin && docker run --name spin --runtime=io.containerd.spin.v2 --platform wasi/wasm ghcr.io/spinframework/containerd-shim-spin/examples/spin-rust-hello:v0.22.0 --command /
spin
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v2.task/moby/4858477f986eb6cf03ecda69d311d293fc310eda19af599eba37823392afee13/log.json: no such file or directory): /usr/local/bin/containerd-shim-spin-v2 did not terminate successfully: exit status 134: unknown

Run 'docker run --help' for more information

edit: I guess that config may not be valid, as (I think) that requires the runtime to be a drop-in for runc? And it doesn't appear to have flags that can be called;

/usr/local/bin/containerd-shim-spin-v2 --version

thread 'main' panicked at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/containerd-shimkit-0.1.1/src/sandbox/cli.rs:226:38:
called `Result::unwrap()` on an `Err` value: InvalidArgument("flag provided but not defined: -version")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Aborted (core dumped)

/usr/local/bin/containerd-shim-spin-v2 --help

thread 'main' panicked at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/containerd-shimkit-0.1.1/src/sandbox/cli.rs:226:38:
called `Result::unwrap()` on an `Err` value: InvalidArgument("flag provided but not defined: -help")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Aborted (core dumped)


/usr/local/bin/containerd-shim-spin-v2

thread 'main' panicked at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/containerd-shimkit-0.1.1/src/sandbox/cli.rs:208:18:
Failed to initialize logger: IoError { context: "failed to init logger", err: Os { code: 2, kind: NotFound, message: "No such file or directory" } }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Aborted (core dumped)