runc run failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe #3937

alexcb · 2024-03-25T23:10:13Z

There's three known cases of earthly not working, which has something to do with a failure starting runc:

https://github.com/earthly/earthly.git#main *failed* | Repeating the failure error...
https://github.com/earthly/earthly.git#main *failed* | runc run failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe
https://github.com/earthly/earthly.git#main *failed* | ERROR
https://github.com/earthly/earthly.git#main *failed* |       The internal command
https://github.com/earthly/earthly.git#main *failed* |           GET GIT META github.com/earthly/earthly/examples/tutorial/go:main
https://github.com/earthly/earthly.git#main *failed* |       did not complete successfully. Exit code 1

Originally posted by @davidmoshal in #3167 (comment)

The text was updated successfully, but these errors were encountered:

alexcb · 2024-03-25T23:11:35Z

The other two cases are from @shepherdjerred as reported in #3167 (comment)

After struggling with this for several days, I wasn't able to come up with a solution. On some machines it seems impossible to use Earthly when some unknown situation arises.

Unfortunately this is a very poor experience for a core build tool; anyone would say that it is unacceptable to have a dependency on a tool that simply cannot run on some developer's machines without any remediation.

Things I tried:

Clearing the earthly-cache volume
Deleting the earthly-buildkitd instance
Updating Docker Desktop
Switching to Orbstack
Completely wiping Docker Desktop following their uninstall commands: https://docs.docker.com/desktop/uninstall/
Enabling/disabling Rosetta
Running earthly prune --reset --all
Disabling "experimental features" in Docker Desktop's settings
Restarting the machine

Even a minimal Earthfile wouldn't build:

VERSION 0.8
FROM alpine
RUN echo hello

This was on an arm64 M1 MacBook running Earthly 0.8.5. There is no issue running Docker containers on the machine outside of Earthly.

These are the errors I saw:

# With Docker Desktop
runc run failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe

# With Orbstack
runc run failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer

Related resources

alexcb · 2024-03-25T23:19:17Z

I put together a rough Dockerfile which extends the earthly-buildkitd container along with an alpine-based rootfs and runc config under https://github.com/earthly/test-runc-in-earthly-buildkitd

It contains two different versions: one for amd64 and another for arm64 -- I have tested this on both a linux amd64 host and M1, which work when running their own architecture.

However, when I run the arm64 version from my amd64 linux (ubuntu) host, I get the same error:

alex@surfperch:~/test$ git clone git@github.com:earthly/test-runc-in-earthly-buildkitd.git
Cloning into 'test-runc-in-earthly-buildkitd'...
remote: Enumerating objects: 169, done.
remote: Counting objects: 100% (169/169), done.
remote: Compressing objects: 100% (121/121), done.
remote: Total 169 (delta 14), reused 169 (delta 14), pack-reused 0
Receiving objects: 100% (169/169), 3.45 MiB | 3.86 MiB/s, done.
Resolving deltas: 100% (14/14), done.
alex@surfperch:~/test$ cd test-runc-in-earthly-buildkitd/
alex@surfperch:~/test/test-runc-in-earthly-buildkitd$ cd arm64/
alex@surfperch:~/test/test-runc-in-earthly-buildkitd/arm64$ git rev-parse HEAD
a3d0edd942c4fd052d6d0a464455084f3920f68b
alex@surfperch:~/test/test-runc-in-earthly-buildkitd/arm64$ docker build --platform linux/arm64 --tag foo . && docker run --rm --privileged --platform linux/arm64 foo
[+] Building 0.7s (10/10) FINISHED                                                                                                                                      docker:default
 => [internal] load build definition from Dockerfile                                                                                                                              0.0s
 => => transferring dockerfile: 196B                                                                                                                                              0.0s
 => [internal] load .dockerignore                                                                                                                                                 0.0s
 => => transferring context: 2B                                                                                                                                                   0.0s
 => [internal] load metadata for docker.io/earthly/buildkitd:v0.8.6                                                                                                               0.4s
 => [1/5] FROM docker.io/earthly/buildkitd:v0.8.6@sha256:d3bc052d1acb3bf00bffbfa747bd6dafd79242a573516a244db7fa28e7f57073                                                         0.0s
 => [internal] load build context                                                                                                                                                 0.1s
 => => transferring context: 8.41MB                                                                                                                                               0.0s
 => CACHED [2/5] COPY entrypoint.sh /root/entrypoint.sh                                                                                                                           0.0s
 => CACHED [3/5] WORKDIR /test-runc                                                                                                                                               0.0s
 => [4/5] COPY rootfs rootfs                                                                                                                                                      0.1s
 => [5/5] COPY config.json .                                                                                                                                                      0.0s
 => exporting to image                                                                                                                                                            0.1s
 => => exporting layers                                                                                                                                                           0.1s
 => => writing image sha256:da6d557c823886b14f1fe9e462c3c7fb27afd99d4063cbf74e700dcc37578027                                                                                      0.0s
 => => naming to docker.io/library/foo                                                                                                                                            0.0s
entrypoint.sh running under aarch64
detected cgroups v2; buildkit/entrypoint.sh running under pid=1 with controllers "cpuset cpu io memory hugetlb pids rdma misc" in group 0::/
moving 1 to my-test-group cgroup
running runc
NAME:
   runc - Open Container Initiative runtime

runc is a command line client for running applications packaged according to
the Open Container Initiative (OCI) format and is a compliant implementation of the
Open Container Initiative specification.

runc integrates well with existing process supervisors to provide a production
container runtime environment for applications. It can be used with your
existing process monitoring tools and the container will be spawned as a
direct child of the process supervisor.

Containers are configured using bundles. A bundle for a container is a directory
that includes a specification file named "config.json" and a root filesystem.
The root filesystem contains the contents of the container.

To start a new instance of a container:

    # runc run [ -b bundle ] <container-id>

Where "<container-id>" is your name for the instance of the container that you
are starting. The name you provide for the container instance must be unique on
your host. Providing the bundle directory using "-b" is optional. The default
value for "bundle" is the current directory.

USAGE:
   buildkit-runc [global options] command [command options] [arguments...]

VERSION:
   unknown
spec: 1.0.2-dev
go: go1.21.8
libseccomp: 2.5.5

COMMANDS:
   checkpoint  checkpoint a running container
   ... the --help command works, which shows the basic binary works
   
but then it fails:
   
time="2024-03-25T23:17:50Z" level=warning msg="invalid argument"
time="2024-03-25T23:17:50Z" level=fatal msg="nsexec[43]: could not ensure we are a cloned binary: No such file or directory"
time="2024-03-25T23:17:50Z" level=warning error="waiting for init preliminary setup: EOF"
time="2024-03-25T23:17:50Z" level=error msg="runc run failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe"

alexcb · 2024-03-25T23:51:01Z

I can replicate this on a M1 by tricking earthly into running the amd64 version of earthly-buildkitd instead of the arm64 version:

bash-3.2$ cat ~/.earthly/config.yml 
global:
    buildkit_additional_args: ["--platform", "linux/amd64"]

and also running the amd64-based version of earthly.

It fails with:

bash-3.2$ earthly-darwin-amd64 github.com/earthly/earthly+lint
 Init 🚀
————————————————————————————————————————————————————————————————————————————————

           buildkitd | Starting buildkit daemon as a docker container (earthly-buildkitd)...
           buildkitd | ...Done

Streaming logs to https://cloud.earthly.dev/builds/f58b24f5-39a9-4c4f-98bd-470cb18c8e1c

 Build 🔧
————————————————————————————————————————————————————————————————————————————————


================================== ❌ FAILURE ===================================

https://github.com/earthly/earthly.git# *failed* | Repeating the failure error...
https://github.com/earthly/earthly.git# *failed* | runc run failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe
https://github.com/earthly/earthly.git# *failed* | ERROR
https://github.com/earthly/earthly.git# *failed* |       The internal command
https://github.com/earthly/earthly.git# *failed* |           GET GIT META github.com/earthly/earthly
https://github.com/earthly/earthly.git# *failed* |       did not complete successfully. Exit code 1

alexcb · 2024-03-25T23:57:33Z

without the config file change, it still appears to work with an amd64 version of earthly:

file ./earthly-darwin-amd64-v0.8.6
./earthly-darwin-amd64-v0.8.6: Mach-O 64-bit executable x86_64

$ ./earthly-darwin-amd64-v0.8.6 +base
 Init 🚀
————————————————————————————————————————————————————————————————————————————————

           buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)

Streaming logs to https://cloud.earthly.dev/builds/cb300c65-6e29-4c96-bf8b-ae4269235f1a

 Build 🔧
————————————————————————————————————————————————————————————————————————————————

              logbus | Setting organization "" and project ""
              alpine | --> Load metadata alpine linux/arm64
               +base | --> FROM alpine
               +base | [----------] 100% FROM alpine
               +base | --> RUN echo hello
               +base | hello
              output | --> exporting outputs

and it still detects the host arch is arm64, and runs the correct version:

$ docker exec earthly-buildkitd uname -m
aarch64

alexcb · 2024-03-25T23:59:02Z

Maybe there's something specific to the macOS host which is failing to detect the arch type, and it's incorrectly falling back to using amd64?

As a work-around, maybe we can force the correct version in the ~/.earthly/config.yml:

global:
    buildkit_additional_args: ["--platform", "linux/arm64"]

alexcb · 2024-03-26T00:11:10Z

It's also possible to reproduce this issue by not changing the config.yml, but instead by pre-pulling the incorrect platform image.

for example (running on a M1):

rm ~/.earthly/config.yml
docker rmi docker.io/earthly/buildkitd:v0.8.6
docker pull --platform linux/amd64 docker.io/earthly/buildkitd:v0.8.6

file /Users/administrator/bin/earthly-v0.8.6
# /Users/administrator/bin/earthly-v0.8.6: Mach-O 64-bit executable arm64

/Users/administrator/bin/earthly-v0.8.6 +base
 Init 🚀
————————————————————————————————————————————————————————————————————————————————

           buildkitd | Starting buildkit daemon as a docker container (earthly-buildkitd)...
           buildkitd | ...Done

Streaming logs to https://cloud.earthly.dev/builds/1fba2d94-cfd9-401d-9daa-95f7837acc33

 Build 🔧
————————————————————————————————————————————————————————————————————————————————

              logbus | Setting organization "" and project ""
              alpine | --> Load metadata alpine linux/amd64
               +base | --> FROM alpine
               +base | [----------] 100% FROM alpine
               +base | --> RUN echo hello
               +base | runc run failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer
               +base | ERROR Earthfile line 3:0
               +base |       The command
               +base |           RUN echo hello
               +base |       did not complete successfully. Exit code 1

================================== ❌ FAILURE ===================================

               +base *failed* | Repeating the failure error...
               +base *failed* | --> RUN echo hello
               +base *failed* | runc run failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer
               +base *failed* | ERROR Earthfile line 3:0
               +base *failed* |       The command
               +base *failed* |           RUN echo hello
               +base *failed* |       did not complete successfully. Exit code 1

$ docker exec earthly-buildkitd uname -m
x86_64

shepherdjerred · 2024-03-26T03:41:38Z

Note: there have been three people at my company who have run into this issue. One was able to fix it by enabling Rosetta 2 under Docker Desktop settings, one has tried everything short of reinstalling macOS as mentioned in this comment, and we haven't done much debugging with the last individual.

alexcb · 2024-03-26T18:04:13Z

It turns out that some users had DOCKER_DEFAULT_PLATFORM set to linux/amd64, but were using an arm64-based mac (aka M1/2/3)

For those users, they should delete all existing earthly/buildkitd images, and unset the default platform environment variable:

docker rm -f earthly-buildkitd
docker images | grep earthly/buildkitd | awk '{print $3}' | xargs docker rmi -f
unset DOCKER_DEFAULT_PLATFORM

Examples of the new warning: When starting a new container: ``` buildkitd | Starting buildkit daemon as a docker container (earthly-dev-buildkitd)... buildkitd | Warning: earthly-dev-buildkitd was started using architecture amd64, but host architecture is arm64; is DOCKER_DEFAULT_PLATFORM accidentally set? ``` When checking an existing container: ``` buildkitd | Found buildkit daemon as docker container (earthly-dev-buildkitd) buildkitd | Warning: currently running earthly-dev-buildkitd under architecture amd64, but host architecture is arm64; is DOCKER_DEFAULT_PLATFORM accidentally set? ``` fixes #3937 Signed-off-by: Alex Couture-Beil <alex@earthly.dev>

alexcb mentioned this issue Mar 26, 2024

warn if arch mismatch between host and local buildkit #3944

Merged

alexcb closed this as completed in #3944 Mar 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runc run failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe #3937

runc run failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe #3937

alexcb commented Mar 25, 2024

alexcb commented Mar 25, 2024

alexcb commented Mar 25, 2024

alexcb commented Mar 25, 2024

alexcb commented Mar 25, 2024

alexcb commented Mar 25, 2024

alexcb commented Mar 26, 2024

shepherdjerred commented Mar 26, 2024

alexcb commented Mar 26, 2024

runc run failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe #3937

runc run failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe #3937

Comments

alexcb commented Mar 25, 2024

alexcb commented Mar 25, 2024

alexcb commented Mar 25, 2024

alexcb commented Mar 25, 2024

alexcb commented Mar 25, 2024

alexcb commented Mar 25, 2024

alexcb commented Mar 26, 2024

shepherdjerred commented Mar 26, 2024

alexcb commented Mar 26, 2024