Research: can an OS-level filesystem sandbox make reviewers actually read-only? - #80
Merged
Conversation
Findings for whether orca could run an agent CLI so the OS, not the model, stops it writing to the workspace. Measured on Linux; macOS is documentation only. Answer: it works on Linux, and it is not worth building as an orca-owned wrapper. Both bubblewrap and Landlock block a real write-capable claude turn, but both break codex, which is the one backend whose read-only mode is already enforced. claude and codex both ship their own sandboxes, which is the cheaper route on both platforms.
An audit and two probe runs found errors in 10-filesystem-sandbox.md. codex: a direct probe shows its read-only sandbox holds — apply_patch refused by codex's own policy, shell redirect refused by the kernel with EROFS. strace shows it execs the system /usr/bin/bwrap, not the copy it ships, and falls back to real Landlock syscalls when told to. So the vendored bwrap being broken standalone is latent, not load-bearing, and codex's Enforcement.Hard is accurate. That makes the case against an orca-owned wrapper stronger, not weaker: the wrapper would destroy the one guarantee orca has. Landlock cost: the 2.7 ms figure was the C helper, not landrun. Measured over 30 spawns of /bin/true: bare 1.3, helper 1.8, landrun 10.7, bwrap 13.8. landrun is not cheaper than bubblewrap, so that reason for preferring it goes. The Landlock-breaks-codex error message depends on whether /proc is on the write allowlist. Both are now listed so a reader can reproduce it. claude: all three caveats re-probed and confirmed. The Write and Edit tools wrote files straight through the sandbox; denyWrite on the cwd gives no Bash at all; and the startup failure is Ubuntu's bwrap-userns-restrict profile, not a host quirk. The Answer section now carries these instead of calling claude's sandbox a cheap route. One line added: the property is still reachable for claude via a tool allowlist, which is the drop-read-only record's ground. Also noted: the C helper omits LANDLOCK_ACCESS_FS_REFER, and unhandled REFER makes Landlock deny cross-directory rename with EXDEV.
Two fixes to 10-filesystem-sandbox.md. The claude escape hatch is --tools, not --allowedTools. `claude --tools` sets the list of available tools from the built-in set; a probe with --tools Read,Grep,Glob,WebFetch,WebSearch advertised only those five plus MCP tools, and seven escape routes failed with an empty permission_denials list because the tools were not there. Subagents inherit it and it survives --resume. --allowedTools is a permission rule and #74 measured it as additive, so it does not restrict the agent to what it lists. The owner has also decided the feature stays and claude gets repaired, so the document no longer says it is being removed. The enforcement-table correction now says what is measured per backend instead of marking three cells unchecked. opencode holds and a live integration test asserts it (OpencodeIntegrationTest.scala:94-104); pi is flag-level only, never run; gemini has no credentials on this host.
This was referenced Aug 5, 2026
adamw
added a commit
that referenced
this pull request
Aug 5, 2026
Rewrites `docs/research/run-cost/11-drop-read-only.md` as a decision record plus a repair plan, and renames it to `11-repair-read-only.md`. Docs only — no code or behaviour changes. ## What changed The earlier version of this file planned the **removal** of `withReadOnly` / `ToolSet.ReadOnly`. The owner decided **not** to remove it. claude gets repaired instead. ## Why the premise flipped The failure was never the feature. It is the mechanism two backends use. Backends that **remove the capability** enforce it. Backends that **set an approval mode and ask the model to respect it** do not. | backend | mechanism | measured | |---|---|---| | codex | `--sandbox read-only` | **blocked** — policy refusal plus kernel `EROFS`; system `bwrap`, Landlock fallback also blocks; `strace`-confirmed | | opencode | write/edit/bash/patch disabled on the message body | **blocked** — `OpencodeIntegrationTest`, live server | | pi | `--tools read,grep,find,ls` | unverified — flag-level only | | claude | `--permission-mode plan` | **not blocked** — `init` tool list byte-identical to default mode; #73 measured 199 Bash calls, zero denials | | gemini | `--approval-mode plan` | unverified — no credentials on this host | Removing the feature would have discarded three working mechanisms — one an OS sandbox — to fix two that never could work. ## The repair Three candidates were probed with real turns. Only one works: - **claude's own sandbox: no.** It covers Bash only — the `Write` tool created a file and `Edit` rewrote another with `sandbox.filesystem.denyWrite` in force. `denyWrite` on the cwd also breaks Bash entirely, and cwd = the repo is orca's shape. - **`--disallowedTools`: no.** With `Write,Edit,NotebookEdit,Bash` disallowed, opus still wrote a file through the `Monitor` tool. A denylist only removes the names you thought of. - **`--tools` (allowlist): yes.** Only the named tools are advertised, seven escape routes failed with nothing to deny, `ToolSearch` cannot resurrect the dropped tools, subagents inherit it, and it survives `--resume`. Reproduced against the installed claude (2.1.222): plan mode and default mode advertise the same 32 built-in tools; `--tools Read,Grep,Glob,Skill,WebFetch,WebSearch` advertises exactly those six plus MCP tools. ## The plan Five tasks, each with a *Done when*: 1. **Replace the mechanism** — `--permission-mode plan` → a `--tools` allowlist. Includes a wrinkle the brief did not anticipate: `NetworkOnly` gets its `gh` access from command-scoped `Bash(gh …)` entries, which cannot appear in a `--tools` list. Either the planner loses `gh`, or `Bash` goes in the allowlist and `NetworkOnly` on claude drops to `PromptOnly`. 2. **Assess what loses Bash** — per call site. Stack discovery is unaffected (ADR 0019 already says it has no shell). Reviewers are the real cost: they made 199 Bash calls in #73 and 47 more in the T6.2 re-measurement. The claim that the inlined diff removed the need is stated as a bet, with what would falsify it. 3. **Make the enforcement claims true** — the table becomes true for claude; gemini stays unverified and must not be asserted. 4. **Pin the allowlist with a test** — unknown tool names are dropped silently (`Read,Grep,NoSuchTool` → `Grep,Read`, exit 0, no warning), so a CLI rename would strip a tool with no signal. 5. **Record that `--tools` is not a complete boundary** — MCP tools pass through it unfiltered. ## Corrections carried over The removal plan's call-site enumeration and false-documentation sweep are kept, with four errors fixed: - Its "Step 1" was not a pure documentation and test change — `EnforcementTableTest` compares against the live backend, so `ClaudeArgs` and `GeminiArgs` must change with it. - `Plan.reviewed` on codex was **not** a no-op — dropping the tier there emits `--dangerously-bypass-approvals-and-sandbox`, a widening from workspace-write to no sandbox. - `CodexArgsTest`'s `execResume` test was missing from the test sweep. - The file count said 15 where its own list named 17. Actual: 37 references across 17 files. ## Related - #80 — whether an orca-owned OS filesystem sandbox could replace the per-backend mechanisms. Answer: no. - The false scaladocs this plan names are being fixed separately on master. --------- Co-authored-by: Adam Warski <ai@warski.org>
adamw
added a commit
that referenced
this pull request
Aug 5, 2026
…sh (#84) Docs only. Follows #78, which decided to swap `--permission-mode plan` for a `--tools` allowlist on claude. That drops `Bash` from read-only agents. This measures what breaks and what to give back. ## What was measured 87 reviewer transcripts, 1016 `Bash` calls, deduplicated by `tool_use` block id (the correction #73 recorded). #73's baseline run reproduces exactly — 10 sessions, 199 calls. Plus 16 live `claude --print` probe runs on 2.1.222 (`claude-opus-5`), $1.17 total. ## Findings **The cheap route does not work.** `--tools …,Bash` plus `--allowedTools "Bash(git log:*)"` grants the matching command but does not confine anything else: `ls -la`, `wc -l`, `uname -r` and `git status --short` all ran, none matching a rule, none denied. Writes were denied — but by claude's own command classifier, not by the rule. A control run with no flags at all denied the same `touch`. Same unowned CLI-side gate #73 found behind plan mode. Prefix matching itself did not leak in the forms tried: a chained `&& touch` was denied as a whole, and a `$(…; …)` substitution came back "Contains shell syntax (;) that cannot be statically analyzed". The matcher is fine; it is just not the boundary. **MCP works, and is now tested.** With `--tools Read,Grep,Glob` and no `Bash`, an MCP tool named only in `--allowedTools` was listed, called, and returned its result. #78 claimed this; nobody had run it against a live server. **Most Bash use is already covered.** 64% of calls are search, file reads and listings. 3% write files or run programs — and they really ran: `mkdir`, `rm -rf`, `git init`, `git push`, `scala-cli run`, `sbt testOnly`, checked against each `tool_result`. **The git surface should be small.** 82% of git use re-derives the change set the prompt already carries. Only 12% reaches outside it, in 24 of 87 sessions. **`NetworkOnly` is broken the same way.** Its `Enforcement.Hard` claim is false: three planner sessions called `Write` and created files. That is in the recorded corpus, no probe needed. Measured planner `gh` usage across 6 sessions is **zero**, and orca already fetches issue bodies host-side through `GitHubTool`. ## Proposed Reviewers: `--tools Read,Grep,Glob,Skill` plus two MCP tools — `git_show(rev, paths, stat)` and `git_file_at(rev, path)`. Structured parameters, no command string, no flag passthrough. No `git_diff` and no `git_status`, despite being 82% of measured use — that use is redundant with the prompt. Planner: keep `WebFetch`/`WebSearch` in `--tools`, and expose `github_issue` / `github_pr` as MCP wrappers over the existing `GitHubTool`. Drop `gh api` and `gh search` — zero measured use, and `gh api -X POST` can mutate GitHub. ## Not established Why the classifier let writes through in the corpus but denies them now (version, flags and permission mode are confounded); the cost of losing Bash's batching, which 75% of calls rely on; per-round normalisation (only 2 transcripts carry the current re-review wording); gemini. ## Related - #78 — the `--tools` decision this builds on - #80 — OS-level filesystem sandbox: no - #73 — the 199-call baseline and the block-id dedup correction - #74 — plain `--allowedTools` measured as additive - #79, #81 — the base-commit note and the diff cap this surface has to serve - ADR 0012 — the MCP bridge these tools would live on Co-authored-by: Adam Warski <ai@warski.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Research only — one new file under
docs/research/run-cost/, no code changes.The question
withReadOnly/ToolSet.ReadOnlydoes not hold on claude: read-only reviewersmade 199 Bash calls with zero denials, and one wrote a file and ran
scala-cli.This asks whether an OS-level filesystem sandbox is the honest replacement —
read the workspace, never write to it, enforced by the kernel rather than by the
prompt.
Answer
Linux: a sandbox works, and it is not worth building as an orca-owned
wrapper. Bubblewrap and Landlock both do the job. Both were verified end to
end against a real
claudeturn that was allowed to write and actively triedits
Writetool and thenBash:EROFSunder bubblewrap,EACCESunderLandlock, turn completed, file not created. Descendants are covered, including
setsidones. Network keeps working. Per spawn of/bin/true: bare 1.3 ms, CLandlock helper 1.8 ms,
landrun10.7 ms, bubblewrap 13.8 ms.The blocker is codex, and it blocks harder than it first looked. codex runs
its own bubblewrap sandbox for every shell command it issues. Probed directly:
codex exec --sandbox read-onlyhadapply_patchrefused by codex's own policyand the shell redirect it fell back to refused by the kernel with
EROFS.straceshows it execs the system/usr/bin/bwrap, not the copy it ships, andit falls back to real Landlock syscalls when told to. So codex's guarantee is
real. Wrapping it in either sandbox breaks every shell command — bubblewrap
cannot nest, and the kernel forbids a Landlock-restricted thread from changing
the mount topology. An orca wrapper would destroy the one guarantee orca has, so
it would have to exempt codex. Add a per-backend write allowlist (measured:
claude needs
~/.claude,~/.claude.json,~/.cache,~/.npm,/tmpand/dev; pi writes inside.orca/cache/) and the wrapper is no longer generic —it is per-backend work with a native dependency attached.
macOS: unanswered, and treated as such. No Mac was available, so that
section is read, not run, and says so.
sandbox-execstill works and childreninherit the profile, but it is deprecated with no replacement and no removal
date, and a wrong profile is a silent no-op — a shipping agent product was
escaped exactly that way. App Sandbox entitlements need code signing, which
rules them out for binaries orca does not build.
claude's own sandbox is not the answer either. Three caveats were re-probed
and all confirmed: it covers Bash only (with the sandbox in force and
permissions bypassed, the
Writetool created a file and theEdittoolrewrote another),
denyWriteon its own working directory gives no Bash at allrather than read-only Bash, and off the cwd it failed to start at claude's
seccomp step — which is Ubuntu's
bwrap-userns-restrictprofile, shippedenabled since 24.04, not a host quirk.
claude does not need a sandbox:
--toolsis capability removal.claude --tools <tools...>sets "the list of available tools from the built-in set".Measured with
--tools Read,Grep,Glob,WebFetch,WebSearch: only those five plusMCP tools advertised, seven attempted escape routes failing with
permission_denials: []— nothing to deny because the tools were not there —subagents inheriting it, and it surviving
--resume. Not--allowedTools,which is a permission rule and was measured in #74 as additive. So the feature
is being repaired rather than removed, and the repair is a flag, not a sandbox.
One thing to fix either way
EnforcementTableTestassertsReadOnlyisHardon all five backends. codexholds (probed here) and opencode holds (a live-server integration test asserts
the turn cannot write). claude is measurably false. pi is flag-level only and
gemini has no credentials on this host, so both are unproven. The table should
say what is true.
What was actually run
Everything on Linux was measured on one host (Ubuntu, kernel 7.0, bubblewrap
0.11.1, Landlock ABI 8, claude 2.1.222, codex-cli 0.145.0): real agent turns
under each sandbox,
straceof what each CLI writes, the codex and claudesandbox probes, the
--toolsprobe, the Docker matrix, nested namespace escapeattempts, and spawn timings. macOS remains unverified. The document tags every
claim as read, measured, inferred or unknown, and lists the experiment for each
open question.