DinD + Airlock: needs sidecar broker container to actually work end-to-end

[GordonBeeming]

Follow-up to #20 / PR #93. The brokered Docker socket (--dind) works end-to-end in standard mode (verified against the HubX Questions integration tests: 463/463 passing in ~88 seconds, 0 broker blocks). It does not work yet when combined with --enable-airlock.

What's wrong

Two stacked problems show up the moment you add --enable-airlock to a --dind session:

1. HTTP_PROXY routing. The airlock compose template sets HTTP_PROXY=http://proxy:58080 and HTTPS_PROXY=http://proxy:58080 on the workload container so all outbound HTTP traffic gets filtered. HTTP clients (including Docker.DotNet's Microsoft.Net.Http.Client.ManagedHandler used by Testcontainers .NET) dutifully respect that and route every request — including the one to host.docker.internal:NNN where the broker is listening — through the airlock proxy. The airlock proxy then rejects it with Host not allowed because host.docker.internal isn't in the allowlist.

   Already fixed in PR feat: Add brokered Docker socket (--dind, beta) for #20 #93: when --dind is enabled in airlock mode, AirlockRunner now also emits NO_PROXY=host.docker.internal,localhost,127.0.0.1 so the HTTP client bypasses the airlock proxy for broker traffic.

2. Internal-only network with no host route. Even with NO_PROXY in place, the airlock network in compose is internal: true. That deliberately strips any route from the workload container to the host or external network. Trying to reach host.docker.internal:NNN directly fails with dial tcp 0.250.250.254:NNN: connect: network is unreachable. The host-process broker is on the wrong side of the airlock — there's no path to it from inside the airlock'd container.

Why standard mode works

In standard mode the workload container is launched on the default bridge network with --add-host host.docker.internal:host-gateway. host-gateway resolves to the docker bridge gateway and is reachable from the container. The host process is bound to 127.0.0.1:NNN and (on macOS / Windows) Docker Desktop / OrbStack route host.docker.internal:NNN to the host loopback. End-to-end works — verified against the full hubx integration suite.

Proposed fix: sidecar broker container

To make --dind + airlock work, the broker has to live on the airlock network rather than on the host. Specifically:

1. Build a small Docker image (copilot_here:broker?) that runs the existing DockerSocketBroker C# code as its entrypoint. The image should be tiny — ASP.NET-free, native AOT, basically the same broker logic that lives in copilot_here.exe today but packaged to run standalone.

2. The broker container mounts the host's /var/run/docker.sock (or named pipe on Windows) and exposes its own TCP port internally.

3. When --dind is active and airlock is on, AirlockRunner adds the broker container as a service in the compose project. Both the broker and the workload sit on the airlock network. The workload container's DOCKER_HOST becomes tcp://copilot-broker:NNN (service name resolution) instead of tcp://host.docker.internal:NNN.

4. Standard mode keeps using the host-process broker (no need to spin up a sidecar container). Or we move standard mode to the sidecar container too for consistency — that's a follow-up decision.

Alternative considered: use the existing airlock proxy as a TCP forwarder

I looked at whether the existing secure-proxy (Rust binary in /proxy/) could also forward TCP traffic to the broker. It can't — it only handles HTTP CONNECT for HTTPS proxying, not arbitrary TCP. Adding raw TCP forwarding to the airlock proxy is doable but conflates two different responsibilities (HTTP allowlist vs raw TCP plumbing). The sidecar broker is cleaner.

Out of scope

• Body inspection of POST /containers/create (privileged, host bind mounts, host network/pid/ipc) — already filed as Phase 2: Docker broker body inspection (privileged, host bind mounts, host network/pid/ipc) #94, that's the Phase 2 work.
• Standard-mode --dind regressions — covered by PR feat: Add brokered Docker socket (--dind, beta) for #20 #93 and 4 broker bug fixes (see commit 1e3d929).

Testing notes

I have a full smoke runner at /tmp/dind-test/AirlockSmokeRunner (local only, not committed) that exercises the airlock + DinD path end-to-end against the HubX integration tests via reflection into AirlockRunner.GenerateComposeFile. Once the sidecar broker exists, that runner can be adapted into a permanent integration test.

Refs

• Phase 1 PR: feat: Add brokered Docker socket (--dind, beta) for #20 #93
• Phase 2 (body inspection): Phase 2: Docker broker body inspection (privileged, host bind mounts, host network/pid/ipc) #94
• Original issue: Feature: Enable Docker-in-Docker Support for Integration Tests 🐳 #20

[GordonBeeming]

Shipped in #93 — but not as the sidecar broker container this issue originally proposed.

Per Gordon's preference (airlock-only features should live IN the airlock proxy image, not as extra services in the compose project), the bridge moved INTO the existing proxy container instead of becoming a new sidecar:

What landed

• socat installed in the proxy image (docker/Dockerfile.proxy and docker/Dockerfile.proxy-runtime).
• proxy-entrypoint.sh starts socat in the background when the new BROKER_BRIDGE_TARGET env var is set, listening on tcp/2375 inside the proxy and forwarding to that target. The proxy is already dual-homed on the airlock network AND the external network, so the workload reaches it as tcp://proxy:2375 (Docker DNS on the airlock network) and the proxy reaches the host broker via its external network leg.
• AirlockRunner.GenerateComposeFile sets BROKER_BRIDGE_TARGET=host.docker.internal:<port> on the proxy service plus extra_hosts: host.docker.internal:host-gateway. Workload sets DOCKER_HOST=tcp://proxy:2375 plus NO_PROXY=proxy,localhost,127.0.0.1 so HTTP clients (docker CLI, Docker.DotNet's ManagedHandler) bypass the airlock HTTP proxy for daemon traffic.
• Spawned siblings get NetworkMode injected to the airlock compose network via Phase 2 body inspection (Phase 2: Docker broker body inspection (privileged, host bind mounts, host network/pid/ipc) #94). They join the same internal-only network as the workload, so the workload reaches them by Docker DNS without ever crossing the airlock boundary. This was the missing piece that made the spawned-Testcontainers-from-inside-airlock scenario actually work.
• No new image needed, no broker-bridge sidecar service in the compose file. Same :proxy image, optional socat path inside it that only fires when BROKER_BRIDGE_TARGET is set.

Why this approach beat the original sidecar idea

• One image to publish + maintain instead of two.
• The bridge code path doesn't run unless BROKER_BRIDGE_TARGET is set, so standard (non-airlock) --dind is unaffected — it keeps using the host process broker on TCP loopback (macOS / Windows) or UDS (Linux).
• The airlock proxy is already on both networks, so dual-homing came for free.
• No second container in the compose file means simpler lifecycle and one less thing to clean up.

End-to-end coverage

There's a new tests/CopilotHere.IntegrationTests/AirlockSmokeTests.cs that exercises this exact scenario: real proxy image, real airlock compose, exec into the workload, run a sibling, assert the broker rewrote NetworkMode. Gated behind RUN_LIVE_DOCKER_TESTS=1 plus COPILOT_HERE_PROXY_IMAGE / COPILOT_HERE_APP_IMAGE env vars so it only runs when the caller has provided test images. Wired into the CI workflow as integration-tests-airlock (main only, after the canonical proxy + app images are published).