[ARM64] runc run fails with process_linux.go:722: waiting for init preliminary setup caused: unexpected 123 != 0

[AkihiroSuda]

# mkdir rootfs
# curl -fsSL https://dl-cdn.alpinelinux.org/alpine/v3.14/releases/aarch64/alpine-minirootfs-3.14.0-aarch64.tar.gz | tar Cxz ./rootfs
# runc spec
# runc run foo
ERRO[0000] container_linux.go:380: starting container process caused: process_linux.go:722: waiting for init preliminary setup caused: unexpected 123 != 0
# runc --debug run foo
DEBU[0000]github.com/opencontainers/runc/libcontainer/logs/logs.go:69 github.com/opencontainers/runc/libcontainer/logs.processEntry() child process in init()                      
ERRO[0000]github.com/opencontainers/runc/utils.go:57 main.fatal() container_linux.go:380: starting container process caused: process_linux.go:722: waiting for init preliminary setup caused: unexpected 123 != 0 
DEBU[0000]github.com/opencontainers/runc/utils.go:59 main.fatal() container_linux.go:380: starting container process caused: process_linux.go:722: waiting for init preliminary setup caused: unexpected 123 != 0 

runc version:

# runc --version
runc version 1.0.0
commit: v1.0.0-0-g84113eef
spec: 1.0.2-dev
go: go1.16.5

The runc binary is included in https://github.com/containerd/nerdctl/releases/download/v0.9.0/nerdctl-full-0.9.0-linux-arm64.tar.gz (https://github.com/containerd/nerdctl/blob/v0.9.0/Dockerfile#L43-L55)

Host OS: Ubuntu 21.04 on EC2 a1.large (Linux ip-XXX-XXX-XXX-XXX 5.11.0-1011-aws #11-Ubuntu SMP Thu Jun 17 01:38:32 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux)

---

Originally reported by @idrissneumann (lima-vm/lima#79)

[AkihiroSuda]

CGO_ENABLED=1 seems to solve the issue. We need more human-friendly error messages, though.

[tianon]

FWIW, I ran into a similar issue recently with the same exact root cause, and the 123 is {.

If you adjust

runc/libcontainer/process_linux.go

Line 710 in 1079288

inited := make([]byte, 1)

to read more than just one byte and print the entire array as a string, it's a generic JSON error bubbling up from the container initialization code (which seeing in hindsight is what made me realize that runc without cgo can't possibly start containers properly 🤦 😅).