[ARC-DinD] AWF chroot mode should support ARC/DinD Docker daemon filesystems without manual staging

[rhardouin]

Summary

AWF chroot mode currently assumes that paths visible to the GitHub Actions runner process are also visible to the Docker daemon that creates the AWF containers. That assumption breaks on Actions Runner Controller (ARC) setups using Docker-in-Docker (DinD), where the runner container and the Docker daemon sidecar do not share the same root filesystem.

This causes AWF-generated bind mounts and chroot runtime paths to resolve against the DinD daemon filesystem instead of the runner filesystem. A workflow can be made to pass only by manually staging runner files into locations visible to the DinD daemon before AWF starts.

Symptoms

Observed on an ARC runner using an AWF-backed GitHub Agentic Workflow with Copilot, MCP Gateway, and safe outputs:

• AWF bind mounts were evaluated by the Docker daemon sidecar, not the runner container.
• Chroot mode required manual daemon-side staging for /bin, /usr/local/bin, /etc/passwd, /etc/group, /etc/hosts, writable $HOME/XDG state, firewall log directories, Copilot CLI, Node.js, capsh, shell, and core utilities.
• The workaround also had to create runner-side placeholder mount sources for AWF validation while separately materializing the real files through the DinD daemon, because validation and bind mounting happened from different filesystem views.
• Missing staged tools produced failures such as:

[entrypoint][ERROR] capsh not found on host system
error mounting "/tmp/gh-aw/.../bash" to rootfs at "/host/bin/bash": create mountpoint ... read-only file system
/usr/local/bin/copilot: line 9: mkdir: not found

• After the workflow could start successfully, the agent still failed to execute the safeoutputs CLI through Copilot's shell/PTTY path because the chroot did not contain the native module runtime dependencies:

safeoutputs noop --message "..."
Failed to load native module: pty.node
Error: libutil.so.1: cannot open shared object file: No such file or directory

• In that case the overall workflow could complete successfully, but the safeoutputs artifact stayed empty ({"items":[]}), so downstream detection and safe output handling were skipped.

• After staging the PTY dependency, the next failure showed that MCP Gateway itself was healthy but unreachable from inside the AWF chroot because host.docker.internal was not resolvable there. The AWF-generated chroot hosts bind source was mounted as a directory from the daemon point of view, so the entrypoint could not add the gateway hostname:

[entrypoint][WARN] Could not add host.docker.internal to chroot /etc/hosts
/usr/local/bin/entrypoint.sh: line ...: /host/etc/hosts: Is a directory
host.docker.internal unresolved

• Manually staging a daemon-visible hosts file and mounting it into the chroot made host.docker.internal resolvable. After that workaround, the agent emitted a real safeoutputs noop item, detection ran, and safe output processing completed successfully.

The workflow eventually succeeded after adding workflow-level pre-agent steps that copied or synthesized the required files into daemon-visible /tmp/gh-aw/... paths, created matching runner-visible placeholders for AWF mount validation, and mounted those staged directories into AWF.

No MCP Gateway issue is being filed from this investigation: MCP Gateway started, registered the expected routes, and served tools successfully once the AWF chroot filesystem was made usable.

Workaround Coverage Observed

The successful workflow-level workaround had to cover the following ARC/DinD gaps:

• Create daemon-visible /tmp/gh-aw runtime directories for MCP payloads, MCP logs, safeoutputs state, AWF firewall logs, and writable agent home/cache/config/state.
• Synthesize daemon-visible identity and name-resolution files: /tmp/gh-aw/arc-etc/passwd, /tmp/gh-aw/arc-etc/group, and /tmp/gh-aw/arc-etc/hosts.
• Stage shell/runtime tools into daemon-visible mounts: capsh, node, bash, BusyBox applets, executable shims, and a /bin/bash chroot mount that does not replace all of /bin with an invalid or read-only source.
• Stage Copilot as a chroot-visible wrapper with writable HOME/XDG defaults, copied dynamic library dependencies, libutil.so.1 for the PTY native module path, and MCP config generation from the gateway output.
• Transfer staged data into the DinD daemon filesystem via helper containers and tar streams instead of relying on runner-local paths.

Root Cause

AWF chroot mode builds container volumes from runner-side paths, then Docker Compose asks the Docker daemon to bind those paths. With a local Unix socket runner this usually works because the runner and daemon share the same filesystem. With ARC/DinD:

• The runner process sees one filesystem.
• The Docker daemon sidecar sees another filesystem.
• A bind source that exists in the runner may be absent, different, or a directory instead of a file from the daemon point of view.
• AWF's chroot then runs against the daemon-visible mount graph, so host runtime assumptions like /host/bin/bash, /host/usr/local/bin/copilot, /host/etc/passwd, and /host/home/runner can be wrong.
• Even after command binaries are staged, AWF may still miss dynamic/native runtime dependencies needed by agent tooling. Copilot shell execution can load a PTY native module, which in turn requires system libraries such as libutil.so.1. If those libraries are absent from the chroot, shell-based safeoutputs commands fail even though the top-level agent process may still exit successfully.
• AWF's chroot hosts handling has the same split-filesystem problem. A generated hosts file or /etc/hosts bind source can exist for the runner but be absent or have the wrong type for the Docker daemon, causing /host/etc/hosts to appear as a directory and preventing host.docker.internal from being added for MCP Gateway access.

Relevant ownership areas:

• AWF Docker environment handling: gh-aw-firewall/src/host-env.ts
• AWF agent volume construction: gh-aw-firewall/src/services/agent-volumes.ts and gh-aw-firewall/src/services/agent-service.ts
• AWF chroot entrypoint behavior: gh-aw-firewall/containers/agent/entrypoint.sh
• Existing related tests: gh-aw-firewall/src/services/agent-volumes.test.ts, gh-aw-firewall/tests/integration/volume-mounts.test.ts, and gh-aw-firewall/tests/integration/chroot-*.test.ts

Expected Behavior

AWF should either:

1. Natively support ARC/DinD chroot execution by staging or materializing the required chroot filesystem on the Docker daemon side, or
2. Fail early with a clear diagnostic that explains that the selected Docker daemon cannot see the runner filesystem and lists the unsupported/missing mount sources.

Users should not need to handcraft workflow-specific staging for AWF internals such as /bin, /usr/local/bin, /etc/passwd, /etc/group, Copilot/Node wrappers, or common POSIX tools.

Agent shell execution should also have the native runtime dependencies required by the selected engine and safeoutputs tooling. A successful AWF-backed run should not silently produce an empty safeoutputs artifact because a shell command failed to load a missing shared library inside the chroot.

When AWF exposes runner-host services to the chroot through host.docker.internal, the chroot should receive a daemon-valid /etc/hosts entry or equivalent host-gateway resolution. MCP Gateway and safeoutputs should not depend on workflow authors manually mounting a custom hosts file.

Proposed Implementation Plan

Please implement native ARC/DinD support in AWF:

1. Detect filesystem-split Docker environments.

   • Treat non-Unix DOCKER_HOST values and explicit TCP Docker daemon targets as candidates.
   • Add a lightweight probe that compares runner-visible paths with daemon-visible paths, for example by launching a helper container against the selected daemon and checking sentinel paths/content.
   • Preserve current behavior for normal local Unix socket Docker runners.

2. Introduce an AWF chroot staging layer.

   • Stage the minimum required chroot runtime inputs into a daemon-visible workspace.
   • Cover required identity and runtime files: passwd/group lookup, shell, capsh, Node/runtime wrapper support, writable home/cache/config/state paths, AWF log directories, and common applets required by shell wrappers.
   • Cover chroot networking files needed for runner-host service access, especially /etc/hosts entries for host.docker.internal or the configured host-gateway name.
   • Include dynamic library dependencies for staged binaries and native modules used by the agent shell path, including PTY dependencies such as libutil.so.1 when required by the selected engine/runtime.
   • Preserve file modes and executability for staged wrappers, applets, and binaries.
   • Materialize these paths through Docker API/helper containers and tar streams rather than assuming direct daemon access to runner paths.

3. Apply staging to generated agent mounts.

   • Keep existing selective mount security guarantees.
   • For ARC/DinD mode, rewrite mount sources for AWF-managed chroot paths to daemon-side staged paths.
   • Keep user-provided custom mounts explicit, but validate whether each source is daemon-visible; if not, emit a clear diagnostic or stage only where safe and documented.

4. Improve diagnostics.

   • When a source path exists for the runner but is missing or different for the daemon, report the exact mount and why it cannot work.
   • Detect file-vs-directory mismatches for sensitive chroot mounts such as /host/etc/hosts, /host/etc/passwd, /host/etc/group, /host/bin/*, and /host/usr/local/bin/*.
   • When a staged executable or native module cannot load a shared library from the chroot, report the missing library and the runtime path that required it.
   • When host.docker.internal or the configured host-gateway name is required but cannot resolve from the chroot, report that as an AWF chroot networking setup failure rather than surfacing it later as an MCP tool failure.
   • Use the project error-message style: what is wrong, what is expected, and an example remediation.

5. Document the behavior.

   • Update AWF environment/chroot documentation to explain local socket vs ARC/DinD behavior.
   • Document which runtime files AWF stages automatically and which user custom mounts must remain user-managed.

Test Plan

Add focused tests before implementation is considered complete:

• Unit tests for environment classification:

  • local Unix Docker socket keeps existing mount behavior.
  • TCP/ARC-like Docker daemon enables chroot staging/probing.
  • explicit AWF Docker host override preserves documented precedence.

• Unit tests for mount transformation:

  • AWF-managed /bin, /usr, /etc/passwd, /etc/group, /etc/hosts, home/cache/config/state, log directories, and runtime tool paths map to daemon-staged paths in ARC/DinD mode.
  • Existing local runner behavior is unchanged.
  • User custom mounts get clear diagnostics when daemon-invisible.

• Integration test with a TCP DinD daemon:

  • Runner /bin and daemon /bin differ.
  • AWF still runs a chrooted command using bash, capsh, node, and common utilities.
  • Regressions are covered for missing capsh, missing /bin/bash, missing applets such as mkdir, missing PTY/native dependencies such as libutil.so.1, read-only /host/bin/bash mountpoint creation, and /host/etc/hosts being created as a directory.
  • Staged wrappers remain executable after transfer through the daemon-side materialization path.
  • AWF runtime log directories are writable from the chroot.
  • host.docker.internal or the configured gateway hostname resolves from inside the chroot and can reach a runner-host HTTP service.
  • A shell-issued safeoutputs no-op succeeds and produces a non-empty output item instead of leaving agent_output.json as {"items":[]}; downstream detection and safe output processing are not skipped because of an AWF runtime setup failure.

• Run the full project finish target:

make agent-finish

Acceptance Criteria

• An AWF-backed command can run in chroot mode on ARC/DinD without workflow-authored staging of AWF internals.
• Agent shell execution, including PTY/native-module paths used by safeoutputs commands, works in the chroot.
• Host-gateway resolution used for MCP Gateway and safeoutputs works inside the chroot without workflow-authored /etc/hosts staging.
• Local Unix socket behavior and security posture remain unchanged.
• Failures caused by daemon-invisible mount sources produce actionable diagnostics.
• Documentation clearly identifies supported ARC/DinD behavior and remaining limitations.

[rhardouin]

Here is the workaround added to the workflows' frontmatter to make them run on ARC.
The workaround fixes this issue and #30840

sandbox:
  agent:
    id: awf
    mounts:
      - /tmp/gh-aw/arc-etc/passwd:/etc/passwd:ro
      - /tmp/gh-aw/arc-etc/group:/etc/group:ro
      - /tmp/gh-aw/arc-etc/hosts:/etc/hosts:ro
      - /tmp/gh-aw/arc-tools/bin:/usr/local/bin:ro
      - /tmp/gh-aw/arc-tools/chroot-bin:/bin:ro
  mcp:
    env:
      DOCKER_HOST: tcp://localhost:2375

pre-agent-steps:
  - name: Prepare ARC DinD temp directories
    env:
      DOCKER_HOST: tcp://localhost:2375
    run: |
      mkdir -p /tmp/gh-aw/.cache /tmp/gh-aw/.config /tmp/gh-aw/.local/state /tmp/gh-aw/home
      chmod -R 0777 /tmp/gh-aw/.cache /tmp/gh-aw/.config /tmp/gh-aw/.local /tmp/gh-aw/home
      docker run --rm --user 0:0 --entrypoint /bin/sh \
        -v /tmp:/host-tmp:rw \
        ghcr.io/github/gh-aw-mcpg:v0.3.6@sha256:2bb8eef86006a4c5963c55616a9c51c32f27bfdecb023b8aa6f91f6718d9171c \
        -c 'mkdir -p /host-tmp/gh-aw/.cache /host-tmp/gh-aw/.config /host-tmp/gh-aw/.local/state /host-tmp/gh-aw/home /host-tmp/gh-aw/mcp-logs /host-tmp/gh-aw/mcp-payloads /host-tmp/gh-aw/sandbox/firewall/logs /host-tmp/gh-aw/sandbox/firewall/logs/api-proxy-logs /host-tmp/gh-aw/sandbox/firewall/logs/cli-proxy-logs && chmod -R 0777 /host-tmp/gh-aw'
      mkdir -p /tmp/gh-aw/arc-etc
      printf '%s\n' 'runner:x:1001:1001:GitHub Actions Runner:/tmp/gh-aw/home:/bin/bash' 'nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin' > /tmp/gh-aw/arc-etc/passwd
      printf '%s\n' 'runner:x:1001:' 'nobody:x:65534:' > /tmp/gh-aw/arc-etc/group
      printf '%s\n' '127.0.0.1 localhost' '::1 localhost ip6-localhost ip6-loopback' '172.30.0.1 host.docker.internal' > /tmp/gh-aw/arc-etc/hosts
      chmod a+r /tmp/gh-aw/arc-etc/passwd /tmp/gh-aw/arc-etc/group /tmp/gh-aw/arc-etc/hosts
      tar -C /tmp/gh-aw -cf - arc-etc | docker run --rm -i --user 0:0 --entrypoint /bin/sh \
        -v /tmp:/host-tmp:rw \
        ghcr.io/github/gh-aw-mcpg:v0.3.6@sha256:2bb8eef86006a4c5963c55616a9c51c32f27bfdecb023b8aa6f91f6718d9171c \
        -c 'mkdir -p /host-tmp/gh-aw && tar -C /host-tmp/gh-aw -xf - && chmod -R a+rX /host-tmp/gh-aw/arc-etc'
      if [ -d /tmp/gh-aw/aw-prompts ]; then
        tar -C /tmp/gh-aw -cf - aw-prompts | docker run --rm -i --user 0:0 --entrypoint /bin/sh \
          -v /tmp:/host-tmp:rw \
          ghcr.io/github/gh-aw-mcpg:v0.3.6@sha256:2bb8eef86006a4c5963c55616a9c51c32f27bfdecb023b8aa6f91f6718d9171c \
          -c 'mkdir -p /host-tmp/gh-aw && tar -C /host-tmp/gh-aw -xf - && chmod -R a+rX /host-tmp/gh-aw/aw-prompts'
      fi
  - name: Prepare ARC DinD AWF chroot tools
    env:
      DOCKER_HOST: tcp://localhost:2375
    run: |
      docker run --rm --user 0:0 --entrypoint /bin/sh \
        -v /tmp:/host-tmp:rw \
        node:lts-alpine@sha256:d1b3b4da11eefd5941e7f0b9cf17783fc99d9c6fc34884a665f40a06dbdfc94f \
        -c '
          set -eu
          apk add --no-cache bash libcap busybox-static >/dev/null
          root=/host-tmp/gh-aw/arc-tools
          mkdir -p "$root/bin" "$root/chroot-bin" "$root/lib"
          busybox_static="$(command -v busybox.static || true)"
          if [ -z "$busybox_static" ]; then
            busybox_static=/bin/busybox.static
          fi
          cp -Lf "$busybox_static" "$root/chroot-bin/busybox"
          ln -sf busybox "$root/chroot-bin/sh"
          for applet in awk basename cat chmod chown cp cut date dirname env find grep head id ln ls mkdir mv pwd readlink rm sed sleep sort tail tar touch tr uname; do
            ln -sf busybox "$root/chroot-bin/$applet"
          done
          copy_deps() {
            ldd "$1" | while read -r line; do
              for field in $line; do
                case "$field" in
                  /*) [ -f "$field" ] && cp -Lf "$field" "$root/lib/" ;;
                esac
              done
            done
          }
          set -- /lib/ld-musl-*.so.1
          loader="$(basename "$1")"
          cp -Lf "$1" "$root/lib/$loader"
          cp -Lf "$(command -v capsh)" "$root/capsh.real"
          cp -Lf "$(command -v node)" "$root/node.real"
          cp -Lf "$(command -v bash)" "$root/bash.real"
          copy_deps "$(command -v capsh)"
          copy_deps "$(command -v node)"
          copy_deps "$(command -v bash)"
          for tool in capsh node bash; do
            {
              printf "%s\n" "#!/bin/sh"
              printf "%s\n" "exec /tmp/gh-aw/arc-tools/lib/$loader --library-path /tmp/gh-aw/arc-tools/lib /tmp/gh-aw/arc-tools/$tool.real \"\$@\""
            } > "$root/bin/$tool"
            chmod a+rx "$root/bin/$tool"
          done
          cp -Lf "$root/bin/bash" "$root/chroot-bin/bash"
          chmod -R a+rX "$root"
        '
      copilot_bin="$(command -v copilot)"
      copilot_stage="$(mktemp -d)"
      trap 'rm -rf "$copilot_stage"' EXIT
      mkdir -p "$copilot_stage/glibc"
      cp -Lf "$copilot_bin" "$copilot_stage/copilot.real"
      if ldd "$copilot_bin" >/dev/null 2>&1; then
        ldd "$copilot_bin" | awk '{ if ($1 ~ /^\//) print $1; if ($3 ~ /^\//) print $3 }' | while IFS= read -r dep; do
          cp -Lf "$dep" "$copilot_stage/glibc/" 2>/dev/null || true
        done
      fi
      for dep in \
        /lib/x86_64-linux-gnu/libutil.so.1 \
        /usr/lib/x86_64-linux-gnu/libutil.so.1
      do
        [ -f "$dep" ] && cp -Lf "$dep" "$copilot_stage/glibc/" || true
      done
      tar -C "$copilot_stage" -cf - . | docker run --rm -i --user 0:0 --entrypoint /bin/sh \
        -v /tmp:/host-tmp:rw \
        ghcr.io/github/gh-aw-mcpg:v0.3.6@sha256:2bb8eef86006a4c5963c55616a9c51c32f27bfdecb023b8aa6f91f6718d9171c \
        -c 'mkdir -p /host-tmp/gh-aw/arc-tools && tar -C /host-tmp/gh-aw/arc-tools -xf -'
      docker run --rm --user 0:0 --entrypoint /bin/sh \
        -v /tmp:/host-tmp:rw \
        ghcr.io/github/gh-aw-mcpg:v0.3.6@sha256:2bb8eef86006a4c5963c55616a9c51c32f27bfdecb023b8aa6f91f6718d9171c \
        -c '
          set -eu
          root=/host-tmp/gh-aw/arc-tools
          loader="$(find "$root/glibc" -maxdepth 1 -name "ld-linux-*.so.*" -type f | head -n 1 || true)"
          if [ -n "$loader" ]; then
            loader="/tmp/gh-aw/arc-tools/glibc/$(basename "$loader")"
            exec_line="exec $loader --library-path /tmp/gh-aw/arc-tools/glibc /tmp/gh-aw/arc-tools/copilot.real \"\$@\""
          else
            exec_line="exec /tmp/gh-aw/arc-tools/copilot.real \"\$@\""
          fi
          cat > "$root/bin/copilot" <<\EOF
      #!/bin/sh
      set -eu
      export HOME="\${HOME:-/tmp/gh-aw/home}"
      export USER="\${USER:-runner}"
      export LOGNAME="\${LOGNAME:-runner}"
      export XDG_CACHE_HOME="\${XDG_CACHE_HOME:-/tmp/gh-aw/.cache}"
      export XDG_CONFIG_HOME="\${XDG_CONFIG_HOME:-/tmp/gh-aw/home}"
      export XDG_STATE_HOME="\${XDG_STATE_HOME:-/tmp/gh-aw/.local/state}"
      export LD_LIBRARY_PATH="/tmp/gh-aw/arc-tools/glibc\${LD_LIBRARY_PATH:+:\$LD_LIBRARY_PATH}"
      mkdir -p "\$HOME/.copilot" "\$XDG_CONFIG_HOME/.copilot" "\$XDG_CACHE_HOME" "\$XDG_STATE_HOME" /tmp/gh-aw/sandbox/agent/logs
      if [ -n "\${RUNNER_TEMP:-}" ] && [ -f "\$RUNNER_TEMP/gh-aw/mcp-config/gateway-output.json" ]; then
        /usr/local/bin/node <<\NODE
      const fs = require("fs");
      const path = require("path");

      const input = path.join(process.env.RUNNER_TEMP || "/tmp", "gh-aw/mcp-config/gateway-output.json");
      if (fs.existsSync(input)) {
        const gateway = JSON.parse(fs.readFileSync(input, "utf8"));
        const servers = gateway.mcpServers || {};
        const domain = process.env.MCP_GATEWAY_DOMAIN || "host.docker.internal";
        const port = process.env.MCP_GATEWAY_PORT || "8080";
        const outputServers = {};

        for (const [name, entry] of Object.entries(servers)) {
          const transformed = { ...entry, tools: entry.tools || ["*"] };
          if (typeof transformed.url === "string") {
            transformed.url = transformed.url.replace(/^http:\/\/[^/]+(?=\/)/, `http://${domain}:${port}`);
          }
          outputServers[name] = transformed;
        }

        const dirs = new Set([
          path.join(process.env.HOME || "/tmp/gh-aw/home", ".copilot"),
          path.join(process.env.XDG_CONFIG_HOME || "/tmp/gh-aw/home", ".copilot"),
          "/tmp/gh-aw/.config/.copilot",
        ]);
        const payload = JSON.stringify({ mcpServers: outputServers }, null, 2);
        for (const dir of dirs) {
          fs.mkdirSync(dir, { recursive: true });
          fs.writeFileSync(path.join(dir, "mcp-config.json"), payload, { mode: 0o600 });
        }
      }
      NODE
      fi
      EOF
          printf "%s\n" "$exec_line" >> "$root/bin/copilot"
          chmod a+rx "$root/bin/copilot"
          chmod -R a+rX "$root"
        '
      # AWF validates mount sources in the runner, while Docker binds them from DinD.
      mkdir -p /tmp/gh-aw/arc-tools/bin
      mkdir -p /tmp/gh-aw/arc-tools/chroot-bin
      : > /tmp/gh-aw/arc-tools/chroot-bin/bash
      chmod a+rx /tmp/gh-aw/arc-tools/chroot-bin/bash
      chmod -R a+rX /tmp/gh-aw/arc-tools

engine:
  id: copilot
  model: GPT-5.4
  env:
    XDG_CACHE_HOME: /tmp/gh-aw/.cache
    XDG_CONFIG_HOME: /tmp/gh-aw/home
    XDG_STATE_HOME: /tmp/gh-aw/.local/state
    HOME: /tmp/gh-aw/home
    USER: runner
    LOGNAME: runner
    MCP_GATEWAY_DOMAIN: host.docker.internal
    MCP_GATEWAY_PORT: "8080"