Skip to content

feat(jail): fs-jail mode — confine worker reads to a system+toolchain allowlist (WORKER_FS_JAIL) - #79

Open
drewstone wants to merge 1 commit into
mainfrom
feat/worker-fs-jail
Open

feat(jail): fs-jail mode — confine worker reads to a system+toolchain allowlist (WORKER_FS_JAIL)#79
drewstone wants to merge 1 commit into
mainfrom
feat/worker-fs-jail

Conversation

@drewstone

Copy link
Copy Markdown
Owner

The hole

The host write-jail (#64) mounts the whole host root read-only--ro-bind / / in src/jail/linux-bwrap.ts. It confines a spawned coder CLI's writes to a scratch root but leaves reads wide open. So a coder running under the host executor can read the entire filesystem, including the benchmark repo that holds the task definitions and grader answer keys.

Observed in a live run — a coder escaped its clean temp workspace and read the hidden task + grader directly:

cd /home/drew/code/blueprint-agent/scripts/experiments/scenarios/verticals/infra
grep ... api-ghost-admin.ts      # ← the task spec + hidden grader source (91 KB), read = cheating

The workspace is a clean temp dir, but only writes were jailed; reads were not. (The heavy VB pools already run OPENCODE_EXECUTOR=docker and are fs-isolated; this closes the host-executor path, which was unconfined.)

The fix — an fs-jail mode (bwrap allowlist)

fs-jail is a superset of write-jail: same write confinement, plus read confinement. On Linux/bwrap it drops --ro-bind / / and binds only:

  • a minimal read-only system allowlist: /usr (+ /bin /sbin /lib* via --ro-bind-try, which follow the merged-usr symlinks), /etc (resolv/ssl/nss), /run/systemd/resolve (DNS), /opt; plus --proc /proc, --dev /dev;
  • a fresh empty --tmpfs /tmp — the host /tmp (sibling-run twins, other runs' scratch) is invisible;
  • the language + CLI toolchain, auto-derived at wrap time (node install prefix from the running interpreter; the CLI's on-PATH dir + its realpath install root). A fail-closed guard refuses /, /home, $HOME, or any ancestor of the workspace (so it can never re-open the home tree or a /tmp sibling). Operator-extensible via BRIDGE_JAIL_RO_PATHS;
  • the workspace itself, READ-WRITE, so a coding agent still builds its solution in place.

Backends that materialize MCP/runtime config under the host /tmp (opencode, kimi) re-expose those specific dirs read-only through a new registerJailReadable helper, so the fresh tmpfs doesn't hide them. Network is untouched (--share-net, no --unshare-net) so the localhost twin and real APIs stay reachable. The tangle-router inference call is the bridge's own network, not the worker's, and is unaffected.

Env flag + rollout (fail-closed, default off)

Default is off — live runs are untouched until an operator opts in. Turn it on with any of:

  • WORKER_FS_JAIL=1 — shorthand for the fs-jail floor;
  • BRIDGE_JAIL_MODE=fs-jail — same, via the existing jail-mode env;
  • per-request execution.jail.mode: "fs-jail".

The env floor is a MAX: a request can raise confinement (write-jailfs-jail) but never lower it. If a jail is requested and bwrap can't run here, the server still fails closed at startup (unchanged; BRIDGE_JAIL_FALLBACK=warn to override).

Proof — scripts/verify-fs-jail.ts (real bwrap spawn, this host)

[PASS] BLOCK read host secret file (grader key stand-in) — exit=1
[PASS] BLOCK list the secret dir                          — exit=2
[PASS] BLOCK list host repo /home/drew/code               — exit=2
[PASS] BLOCK cat the real ghost-admin task/grader file    — exit=1
[PASS] ALLOW write a file in the workspace                — out="built"
[PASS] ALLOW run node                                     — out="4"
[PASS] ALLOW run python3                                  — out="42"
[PASS] ALLOW curl the localhost twin                      — out="twin-ok"

Composition tests (tests/jail.test.ts) assert the argv shape (no whole-root read bind, tmpfs /tmp, workspace read-write, floor MAX semantics). Full suite green: 41 jail tests, 322 total (pnpm test); tsc --noEmit clean. bwrap 0.9.0 is installed on the host.

Operator: restart the bridge with the jail ON (after merge)

fs-jail needs unprivileged user namespaces. Enable once (host is modern Ubuntu):

sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0   # or: sudo chmod u+s /usr/bin/bwrap

Then restart the bridge with the flag set, e.g.:

WORKER_FS_JAIL=1 pnpm --dir /path/to/cli-bridge start
# (add WORKER_FS_JAIL=1 to the bridge's env / systemd unit / dotenvx env block, then restart that unit)

On boot the log prints [cli-bridge] jail default: fs-jail root=<cwd>/.agent-home. To confirm the jail holds on the target host: pnpm verify:fs-jail.

Note: do NOT restart the currently-running bridges (:3355, :3390) as part of merging — other live runs depend on them. This PR is build-only; the operator restarts on their own schedule.

… allowlist (WORKER_FS_JAIL)

The host write-jail mounts the whole host root read-only (`--ro-bind / /`),
so a spawned coder CLI's bash can read the entire filesystem — including the
benchmark repo's task definitions and grader answer keys. A coder was observed
grepping /home/drew/code/blueprint-agent/.../infra/api-ghost-admin.ts to read
the hidden task + grader, i.e. cheating. Only WRITES were jailed; reads were not.

Add an `fs-jail` mode (superset of write-jail) that, on Linux/bwrap, drops the
whole-root read bind and instead binds ONLY:
  - a minimal read-only system allowlist (/usr + /bin,/sbin,/lib* via ro-bind-try,
    /etc for resolv/ssl/nss, /run/systemd/resolve for DNS, /opt), /proc, /dev;
  - a FRESH empty tmpfs at /tmp (so host /tmp twins + other runs are invisible);
  - the language + CLI toolchain, auto-derived at wrap time (node install prefix,
    the CLI's on-PATH dir + realpath install root) with a fail-closed guard that
    refuses /, /home, $HOME, or any ANCESTOR of the workspace; operator-extensible
    via BRIDGE_JAIL_RO_PATHS;
  - the workspace, bound READ-WRITE, so a coding agent still builds its solution.

Backends whose MCP/runtime config is materialized under the host /tmp
(opencode, kimi) re-expose those dirs read-only via registerJailReadable so the
tmpfs does not hide them. Network is untouched (--share-net) so the localhost
twin and real APIs stay reachable.

Gated + fail-closed: default off; opt in with WORKER_FS_JAIL=1 (shorthand) or
BRIDGE_JAIL_MODE=fs-jail / per-request execution.jail.mode=fs-jail. The env
floor is a MAX — a request can raise confinement, never lower it. When a jail is
requested but bwrap can't run here, the server still fails closed (unchanged).

Proof (scripts/verify-fs-jail.ts, real bwrap spawn on this host):
  BLOCK  cat host secret / ls /home/drew/code / cat the real ghost-admin file
  ALLOW  write in workspace, node -e, python3 -c, curl the localhost twin
Composition tests assert the argv shape + floor semantics; full suite green
(41 jail tests, 322 total).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant