Runc error when running Earthly under a Rosetta VM on Mac M1

[jmgilman]

What went wrong?
I attempted to run Earthly on a x86 Ubuntu VM running on my M1 macbook. Sample Earthfile:

VERSION 0.7
FROM debian:stable-slim

test:
  RUN apt-get update

When running this target:

 Init 🚀
————————————————————————————————————————————————————————————————————————————————

           buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)

 Build 🔧
————————————————————————————————————————————————————————————————————————————————

  debian:stable-slim | --> Load metadata debian:stable-slim linux/amd64
               +base | --> FROM debian:stable-slim
               +base | [----------] 100% resolve docker.io/library/debian:stable-slim@sha256:6fe30b9cb71d604a872557be086c74f95451fecd939d72afe3cffca3d9e60607
               +test | --> RUN apt-get update
               +test | runc run failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer
               +test | ERROR Earthfile line 5:2
               +test |       The command
               +test |           RUN apt-get update
               +test |       did not complete successfully. Exit code 1

============================= ❌ FAILURE [Build 🔧] ==============================

Repeating the output of the command that caused the failure
               +test *failed* | --> RUN apt-get update
               +test *failed* | runc run failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer
               +test *failed* | ERROR Earthfile line 5:2
               +test *failed* |       The command
               +test *failed* |           RUN apt-get update
               +test *failed* |       did not complete successfully. Exit code 1

What should have happened?
I expected the target to run to completion (especially given it's simplicity).

Other Helpful Information

You can reproduce this on an M1 mac:

1. Install OrbStack
2. Create a new x86 Ubuntu VM (set CPU type to Intel)
3. Install Docker + Earthly
4. Run the test target from the previously given Earthfile

It's quite possible this is a bug in OrbStack, Docker, and/or Rosetta. However, creating an example Dockerfile with the below contents:

FROM debian:stable-slim

RUN apt-get update

And then building it on the same machine has no issue. So, Earthly is doing something unique here that I cannot figure out. Perhaps if I understood it more I might be able to figure out exactly where the issue is.

[jmgilman]

Possibly related?

• Docker Desktop 4.16.0 Run x86_64 dind image fails to pull images docker/for-mac#6678
• Cannot run docker in Intel linux machine orbstack/orbstack#74

[alexcb]

I've never seen this error message before, it might take some time before we can get to this.

creating an example Dockerfile ...

Did you try using docker buildx . to ensure that docker used buidkit for the build rather than classical docker?

[jmgilman]

Yes, it appears to work fine:

josh@ubuntux86:~$ docker buildx build .
[+] Building 7.5s (6/6) FINISHED                                                                                                                                                                  docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                        0.0s
 => => transferring dockerfile: 83B                                                                                                                                                                         0.0s
 => [internal] load .dockerignore                                                                                                                                                                           0.0s
 => => transferring context: 2B                                                                                                                                                                             0.0s
 => [internal] load metadata for docker.io/library/debian:bookworm-slim                                                                                                                                     2.0s
 => [1/2] FROM docker.io/library/debian:bookworm-slim@sha256:741bae561f5c2261f4cdd535e4fd4c248dec0aafc1b9a1410b3d67ad24571340                                                                               2.4s
 => => resolve docker.io/library/debian:bookworm-slim@sha256:741bae561f5c2261f4cdd535e4fd4c248dec0aafc1b9a1410b3d67ad24571340                                                                               0.0s
 => => sha256:52d2b7f179e32b4cbd579ee3c4958027988f9a8274850ab0c7c24661e3adaac5 29.12MB / 29.12MB                                                                                                            1.1s
 => => sha256:741bae561f5c2261f4cdd535e4fd4c248dec0aafc1b9a1410b3d67ad24571340 1.85kB / 1.85kB                                                                                                              0.0s
 => => sha256:a60c0c42bc6bdc09d91cd57067fcc952b68ad62de651c4cf939c27c9f007d1c5 529B / 529B                                                                                                                  0.0s
 => => sha256:2e134e7f7afd768389a610b1b5ed9f14fef246c671118a1873e836a124613d0b 1.46kB / 1.46kB                                                                                                              0.0s
 => => extracting sha256:52d2b7f179e32b4cbd579ee3c4958027988f9a8274850ab0c7c24661e3adaac5                                                                                                                   1.2s
 => [2/2] RUN apt-get update                                                                                                                                                                                3.0s
 => exporting to image                                                                                                                                                                                      0.1s
 => => exporting layers                                                                                                                                                                                     0.1s
 => => writing image sha256:07dbc06c9ceee8b4ae8e14adb1865352730bd04de54e43b0a015b86c6a83a6f3

Compared to Earthly:

josh@ubuntux86:~$ earthly +test
 Init 🚀
————————————————————————————————————————————————————————————————————————————————

           buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)

 Build 🔧
————————————————————————————————————————————————————————————————————————————————

debian:bookworm-slim | --> Load metadata debian:bookworm-slim linux/amd64
               +base | --> FROM debian:bookworm-slim
               +base | [----------] 100% resolve docker.io/library/debian:bookworm-slim@sha256:741bae561f5c2261f4cdd535e4fd4c248dec0aafc1b9a1410b3d67ad24571340
               +base | [----------] 100% sha256:52d2b7f179e32b4cbd579ee3c4958027988f9a8274850ab0c7c24661e3adaac5
               +base | [----------] 100% extracting sha256:52d2b7f179e32b4cbd579ee3c4958027988f9a8274850ab0c7c24661e3adaac5
               +test | --> RUN apt-get update
               +test | runc run failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer
               +test | ERROR Earthfile line 5:2
               +test |       The command
               +test |           RUN apt-get update
               +test |       did not complete successfully. Exit code 1

============================= ❌ FAILURE [Build 🔧] ==============================

Repeating the output of the command that caused the failure
               +test *failed* | --> RUN apt-get update
               +test *failed* | runc run failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer
               +test *failed* | ERROR Earthfile line 5:2
               +test *failed* |       The command
               +test *failed* |           RUN apt-get update
               +test *failed* |       did not complete successfully. Exit code 1
🛰️ Reuse cache between CI runs with Earthly Satellites! 2-20X faster than without cache. Generous free tier https://cloud.earthly.dev

I suspect the issue is perhaps related to DinD. Doesn't Earthly use it at some point?

[alexcb]

I suspect the issue is perhaps related to DinD. Doesn't Earthly use it at some point?

runc is executed (via buildkit) within the earthly-buildkitd container, so there is a container-in-a-container aspect.

[jmgilman]

What's the easiest way to replicate this outside of Earthly? Run a dind image and then attempt to run the given Dockerfile in it? I'd love to remove Earthly as the potential problem here so I can file this issue more upstream.

[alexcb]

maybe try a docker buildx inside a docker container that's running buildkit? or maybe there's some runc integration tests that can be run within a docker container?

[hasan4791]

Hitting the same error. In mycase my vms are in lima and running some cri-o integration tests which are all failing with similar error.

[immesys]

I am hitting this when running Earthly on an M1 Mac that is using Orbstack instead of Docker Desktop:

earthly --artifact github.com/earthly/earthly/examples/tutorial/python:main+part2/part2 ./part2
 Init 🚀
————————————————————————————————————————————————————————————————————————————————

           buildkitd | Starting buildkit daemon as a docker container (earthly-buildkitd)...
           buildkitd | ...Done

 Build 🔧
————————————————————————————————————————————————————————————————————————————————


================================== ❌ FAILURE ===================================

git@github.com:earthly/earthly.git#main *failed* | Repeating the failure error...
git@github.com:earthly/earthly.git#main *failed* | runc run failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer
git@github.com:earthly/earthly.git#main *failed* | ERROR
git@github.com:earthly/earthly.git#main *failed* |       The internal command
git@github.com:earthly/earthly.git#main *failed* |           GET GIT META github.com/earthly/earthly/examples/tutorial/python:main
git@github.com:earthly/earthly.git#main *failed* |       did not complete successfully. Exit code 1
🛰️ Reuse cache between CI runs with Earthly Satellites! 2-20X faster than without cache. Generous free tier https://cloud.earthly.dev

[alexcb]

For M1/M2/M3 (arm-based mac users); are you able to run amd64 docker images?

For example, does

docker run --platform linux/amd64 alpine /bin/sh -c "uname -m"

return x86_64 ?

[alexcb]

perhaps more information can be pulled out of:

docker exec earthly-buildkitd cat /tmp/earthly/buildkit/runc-overlayfs/executor/runc-log.json