Since earthly 0.7.14, runc run failed: openrc: invalid argument

[andyli]

work-around based on andyli's latest comment:

colima v0.5.6 has a new --cgroups-v2 flag, specifying this will make the latest version of Earthly (v0.7.20) work.

---

original issue text below

Since earthly 0.7.14, I am getting the error as follows

runc run failed: unable to start container process: error during container init: error mounting "cgroup" to rootfs at "/sys/fs/cgroup": mount cgroup:/sys/fs/cgroup/openrc (via /proc/self/fd/6), flags: 0xf, data: openrc: invalid argument

It prevents me from building even a minimalist +test target.

VERSION 0.7
FROM alpine
test:
    RUN echo ok

Logs:

% earthly --verbose +test      
            frontend | Running command: docker info
            frontend | Running command: docker info --format={{.SecurityOptions}}
            frontend | Running command: docker info --format={{.DockerRootDir}}
            frontend | docker-shell frontend initialized.
 Init 🚀
————————————————————————————————————————————————————————————————————————————————

            frontend | Running command: docker container inspect earthly-buildkitd
           buildkitd | Starting buildkit daemon as a docker container (earthly-buildkitd)...
            frontend | Running command: docker container inspect earthly-buildkitd
            frontend | Running command: docker image inspect docker.io/earthly/buildkitd:v0.7.15
            frontend | Running command: docker run --privileged --env BUILDKIT_DEBUG=false --env BUILDKIT_TCP_TRANSPORT_ENABLED=false --env BUILDKIT_TLS_ENABLED=false --env BUILDKIT_MAX_PARALLELISM=20 --label dev.earthly.settingshash=e9fe7910efadcc66 --mount type=volume,source=earthly-cache,dst=/tmp/earthly --publish 127.0.0.1:8371:8371/tcp -d --pull missing --name earthly-buildkitd docker.io/earthly/buildkitd:v0.7.15
            frontend | Running command: docker container inspect earthly-buildkitd
            frontend | Running command: docker container inspect earthly-buildkitd
           buildkitd | ...Done
           buildkitd | Version github.com/earthly/buildkit v0.7.15 97d5352c297dda803aca907eebd9e3041493eebb
           buildkitd | Platforms: linux/amd64 (native) linux/amd64/v2 linux/amd64/v3 linux/arm64 linux/386
           buildkitd | Utilization: 0 other builds, 0/20 op load
           buildkitd | GC stats: 0 B cache, avg GC duration 0s, all-time GC duration 0s, last GC duration 0s, last cleared 0 B

 Build 🔧
————————————————————————————————————————————————————————————————————————————————

Logbus: setting organization "" and project "" at 2023-08-08T16:58:27.609911+01:00
              alpine | --> Load metadata alpine linux/amd64
              alpine | Completed in 2.008904s
               +base | --> FROM alpine
               +base | [          ]   0% resolve docker.io/library/alpine:latest@sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a
               +base | [----------] 100% resolve docker.io/library/alpine:latest@sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a
               +base | Completed in 277µs
               +test | --> RUN echo ok
               +test | runc run failed: unable to start container process: error during container init: error mounting "cgroup" to rootfs at "/sys/fs/cgroup": mount cgroup:/sys/fs/cgroup/openrc (via /proc/self/fd/6), flags: 0xf, data: openrc: invalid argument
               +test | ERROR Earthfile line 5:4
               +test |       The command
               +test |           RUN echo ok
               +test |       did not complete successfully. Exit code 1
               +test | Overriding args used: 
               +test | Completed in 128.196ms
earthfile2llb immediate error: unlazy force execution: process "/bin/sh -c PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /usr/bin/earth_debugger /bin/sh -c 'echo ok'" did not complete successfully: exit code: 1
earthfile2llb group error: <nil>

============================= ❌ FAILURE [Build 🔧] ==============================

Repeating the output of the command that caused the failure
               +test *failed* | --> RUN echo ok
               +test *failed* | runc run failed: unable to start container process: error during container init: error mounting "cgroup" to rootfs at "/sys/fs/cgroup": mount cgroup:/sys/fs/cgroup/openrc (via /proc/self/fd/6), flags: 0xf, data: openrc: invalid argument
               +test *failed* | ERROR Earthfile line 5:4
               +test *failed* |       The command
               +test *failed* |           RUN echo ok
               +test *failed* |       did not complete successfully. Exit code 1
               +test *failed* | Overriding args used: 
Summary of timing information
Note that the times do not include the expansion of commands like BUILD, FROM, COPY (artifact).
              alpine | () 2.008904s
               +test | () 128.196ms
               +base | () 19.118ms
===============================================================
Total           2.156218s
Total (real)    2.361315886s
🛰️ Reuse cache between CI runs with Earthly Satellites! 2-20X faster than without cache. Generous free tier https://cloud.earthly.dev

% docker logs earthly-buildkitd
starting earthly-buildkit with EARTHLY_GIT_HASH=ae8f65528ad37a278985de2e234deb42b91e308f BUILDKIT_BASE_IMAGE=github.com/earthly/buildkit:97d5352c297dda803aca907eebd9e3041493eebb+build
Autodetecting iptables
Detected iptables-legacy module
BUILDKIT_ROOT_DIR=/tmp/earthly/buildkit
CACHE_SIZE_MB=0
BUILDKIT_MAX_PARALLELISM=20
BUILDKIT_LOCAL_REGISTRY_LISTEN_PORT=8371
EARTHLY_ADDITIONAL_BUILDKIT_CONFIG=
CNI_MTU=1500
OOM_SCORE_ADJ=0

======== CNI config ==========
{
        "cniVersion": "0.3.0",
        "name": "buildkitbuild",
        "type": "bridge",
        "bridge": "cni0",
        "isGateway": true,
        "ipMasq": true,
        "mtu": 1500,
        "ipam": {
                "type": "host-local",
                "subnet": "172.30.0.0/16",
                "routes": [
                        { "dst": "0.0.0.0/0" }
                ]
        }
}
======== End CNI config ==========

======== Buildkitd config ==========
debug = false
root = "/tmp/earthly/buildkit"
insecure-entitlements = [ "security.insecure" ]





[worker.oci]
  enabled = true
  snapshotter = "auto"
  max-parallelism = 20
  gc = true
  networkMode = "cni"
  cniBinaryPath = "/usr/libexec/cni"
  cniConfigPath = "/etc/cni/cni-conf.json"
  
  


======== End buildkitd config ==========

======== OOM Adjust script ==========
#! /bin/sh

set -e

OOM_ADJ="0"
DEBUG="false"
INVOCATION=$(tr -dc A-Za-z0-9 </dev/urandom | head -c 6 ; echo '')

log() {
    ! "$DEBUG" || echo "$(date) [$INVOCATION] | $1" >> /var/log/oom_adj
}

adjust_oom() {
  echo "$1" > /proc/"$2"/oom_score_adj || true # It is ok if the OOM score fails - the PID may have exited, so it no longer matters anyways.
}

if [ "$OOM_ADJ" -eq "0" ]; then
    exit 0
fi

for PID in $(pidof buildkit-runc)
do
    PID_NAME=$(cat /proc/"$PID"/cmdline || echo "unknown")

    case "$PID_NAME" in
        # This is the POSIX way to do a string-starts-with, and accomodates the one prefix we do not want. Order here is important.
        "buildkit-runcinit"*) log "$PID was buildkit-runcinit, ignoring"; continue ;;
        "buildkit-runc"*)     log "$PID is runc parent($PID_NAME), proceeding" ;;
        *)                    log "$PID was $PID_NAME, ignoring"; continue ;;
    esac

    for CHILD_PID in $(pgrep -P "$PID")
    do
        CHILD_PID_NAME=$(cat /proc/"$CHILD_PID"/cmdline || echo "unknown")
        CHILD_OOM_ADJ=$(cat /proc/"$CHILD_PID"/oom_score_adj || echo "unknown")

        case "$CHILD_OOM_ADJ" in
            "unknown"*) log "$PID has child: $CHILD_PID($CHILD_PID_NAME), which was missing, ignoring"; continue ;;
            "0")        log "$PID has child: $CHILD_PID($CHILD_PID_NAME) with oom_score_adj: $CHILD_OOM_ADJ"; continue ;;
            *)          log "$PID has child: $CHILD_PID($CHILD_PID_NAME), oom_score_adj was set to 0: $CHILD_OOM_ADJ" ;;
        esac
        # The child may have started _after_ this script ran on the parent (or other invocation of) buildkit-runc.
        # This undoes the inherited adjustment to make sure we behave properly at OOM time.
        adjust_oom "0" "$CHILD_PID"
    done

    PID_NAME=$(cat /proc/"$PID"/cmdline || echo "unknown")
    case "$PID_NAME" in
        # Just in case the process exec-ed another program between the initial filter and now.
        "buildkit-runcinit"*) log "$PID was buildkit-runcinit, no longer a candidate for OOM adjustment, ignoring"; continue ;;
        "buildkit-runc"*)     log "$PID oom_score_adj was set to 0" ;;
        *)                    log "$PID was $PID_NAME, no longer a candidate for OOM adjustment, ignoring"; continue ;;
    esac
    adjust_oom "$OOM_ADJ" "$PID"
done
======== OOM Adjust script ==========

Detected container architecture is x86_64
time="2023-08-08T15:58:21Z" level=info msg="auto snapshotter: using overlayfs"
time="2023-08-08T15:58:21Z" level=warning msg="failed to release network namespace \"4x3sr4wenmoceeqtl2zwheaqm\" left over from previous run: plugin type=\"loopback\" failed (delete): unknown FS magic on \"/tmp/earthly/buildkit/net/cni/4x3sr4wenmoceeqtl2zwheaqm\": ef53"
time="2023-08-08T15:58:21Z" level=info msg="found worker \"oz5snbzu7ctv5pbq9nktk33fk\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:5847d9013db3 org.mobyproject.buildkit.worker.network:cni org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/arm64 linux/386]"
time="2023-08-08T15:58:21Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
time="2023-08-08T15:58:21Z" level=info msg="found 1 workers, default=\"oz5snbzu7ctv5pbq9nktk33fk\""
time="2023-08-08T15:58:21Z" level=warning msg="currently, only the default worker can be used."
time="2023-08-08T15:58:21Z" level=info msg="Starting local registry for outputs on port 8371"
time="2023-08-08T15:58:21Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." go.version=go1.20.7
time="2023-08-08T15:58:21Z" level=info msg="redis not configured" go.version=go1.20.7
time="2023-08-08T15:58:21Z" level=info msg="Starting upload purge in 3m0s" go.version=go1.20.7
time="2023-08-08T15:58:21Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
time="2023-08-08T15:58:28Z" level=error msg="/moby.buildkit.v1.frontend.LLBBridge/ReadDir returned error: rpc error: code = Unknown desc = process \"/bin/sh -c PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /usr/bin/earth_debugger /bin/sh -c 'echo ok'\" did not complete successfully: exit code: 1" revision=97d5352c297dda803aca907eebd9e3041493eebb version=v0.7.15
time="2023-08-08T15:58:28Z" level=error msg="/moby.buildkit.v1.frontend.LLBBridge/ReadDir returned error: rpc error: code = Unknown desc = process \"/bin/sh -c PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /usr/bin/earth_debugger /bin/sh -c 'echo ok'\" did not complete successfully: exit code: 1" revision=97d5352c297dda803aca907eebd9e3041493eebb version=v0.7.15
time="2023-08-08T15:58:28Z" level=error msg="releasing already released reference oz5snbzu7ctv5pbq9nktk33fk::vf1x6k4dnmykgo0la7nwfowuc" revision=97d5352c297dda803aca907eebd9e3041493eebb version=v0.7.15
time="2023-08-08T15:58:28Z" level=error msg="releasing already released reference oz5snbzu7ctv5pbq9nktk33fk::vf1x6k4dnmykgo0la7nwfowuc" revision=97d5352c297dda803aca907eebd9e3041493eebb version=v0.7.15
time="2023-08-08T15:58:28Z" level=error msg="releasing already released reference oz5snbzu7ctv5pbq9nktk33fk::vf1x6k4dnmykgo0la7nwfowuc" revision=97d5352c297dda803aca907eebd9e3041493eebb version=v0.7.15
time="2023-08-08T15:58:28Z" level=error msg="releasing already released reference oz5snbzu7ctv5pbq9nktk33fk::vf1x6k4dnmykgo0la7nwfowuc" revision=97d5352c297dda803aca907eebd9e3041493eebb version=v0.7.15
time="2023-08-08T15:58:28Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = unlazy force execution: process \"/bin/sh -c PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /usr/bin/earth_debugger /bin/sh -c 'echo ok'\" did not complete successfully: exit code: 1" revision=97d5352c297dda803aca907eebd9e3041493eebb version=v0.7.15

The error happens with earthly 0.7.15 and 0.7.14, but not 0.7.13.

I'm using colima version 0.5.5 on a Mac.

[nelsam]

@andyli on my openrc-based system (gentoo), adding rc_cgroup_mode="unified" to my /etc/rc.conf got me past this error. I'm not sure where you would need to set that on your system, but can you try to find that setting and change it?

Context:

Apparently openrc defaults to a "hybrid" mode between cgroup v1 and cgroup v2. In "hybrid" mode, everything defaults to cgroup v1. The openrc: invalid argument error seems to come from cgroup v1 specifically on openrc.

Setting rc_cgroup_mode="unified" in openrc turns on cgroup v2 globally.

[jschumacher-dexcom]

I'm also seeing this error with the latest Earthly v0.7.15 using Rancher Desktop v1.9.1 on OSX. The problem did not occur with Earthly v0.7.12 and Rancher Desktop v1.9.1. I see that Earthly v0.7.14 upgraded BuildKit and I imagine that's the change that caused this. The above fix to the rc_cgroup_mode worked.

For anyone else using rancher, run rdctl shell to get inside the VM and then you can sudo vi /etc/rc.conf to make the recommended changes.

[nelsam]

Possibly related to this buildkit issue

[sra]

I'm also seeing this on an M2 mac with these versions:

❯ colima version
colima version 0.5.5
git commit: 6251dc2c2c5d8197c356f0e402ad028945f0e830

runtime: docker
arch: aarch64
client: v24.0.5
server: v23.0.6

❯ earthly --version
earthly version v0.7.17 7b7d8f4abbc7a35034fcd29cfada52d3d25fcff2 darwin/arm64; macOS 13.5.2 homebrew

Specific error message:
+deps | runc run failed: unable to start container process: error during container init: error mounting "cgroup" to rootfs at "/sys/fs/cgroup": mount cgroup:/sys/fs/cgroup/openrc (via /proc/self/fd/6), flags: 0xf, data: openrc: invalid argument
+deps | ERROR Earthfile line 7:2
+deps | The command
+deps | RUN mvn dependency:resolve
+deps | did not complete successfully. Exit code 1

[alexcb]

@sra did you try setting rc_cgroup_mode="unified"? see abiosoft/colima#764 (comment) for details.

@andyli did this work?

[sra]

@alexcb sorry about the late reply, I did try it without effect, I just switched to Orbstack and it works fine :)

[andyli]

I've just tried again with colima v0.5.6's new --cgroups-v2 flag. Earthly works as expected.

Somehow I couldn't get earthly to work with colima v0.5.5, even with rc_cgroup_mode="unified", though I'm not sure if I got the setting right or not since I have no idea how to verify the cgroup mode.

[alexcb]

it appears that the --cgroups-v2 colima option solves the openrc: invalid argument error that was originally reported; I have opened a new issue to track the other case of runc failing due to an unrelated issue.