Skip to content

test(opencode): pin every permission rule individually β€” 39 of 77 deletions were invisible to the gate - #352

Merged
ZacxDev merged 9 commits into
mainfrom
opencode-per-rule-permission-coverage
Aug 6, 2026
Merged

test(opencode): pin every permission rule individually β€” 39 of 77 deletions were invisible to the gate#352
ZacxDev merged 9 commits into
mainfrom
opencode-per-rule-permission-coverage

Conversation

@ZacxDev

@ZacxDev ZacxDev commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

The class, not two more instances

#348 restored 52 ask values a blind sed had flattened to allow. An adversarial audit then found it had closed two instances of a coverage class, not the class itself: a permission rule could be deleted outright with the suite fully green, while a realistic command in that family dropped to plain allow at both the glob layer and the layered (glob + guard_core) layer.

This PR closes the class. No permission rule is changed β€” the config was already correct; the suite could not tell.


(a) Deletion sweep β€” independently constructed, before/after

I did not trust the audit's list. I re-derived the rule set, re-ran the sweep, and report my own numbers.

Scope, re-derived: the audit's "77 rule lines" = every single-line "pattern": "verdict" row in the file. I confirmed the decomposition rather than assuming it: 64 in the bash block (1 allow + 50 ask + 13 deny) + 13 outside it (10 top-level tool permissions + 3 single-line plan-agent rows) = 77.

Method: delete one row, repair the trailing comma, run test_opencode_config.py, attribute the kill by failing test id. Counts parsed from pytest's summary line β€” never an exit code.

mutant class before after
delete a rule 39 / 77 survive 0 / 77
comment a rule out (token stays present) 39 / 77 0 / 77
invert verdict β†’ allow 2 / 68 0 / 68
invert verdict deny β†’ ask 1 / 16 0 / 16
reorder (hoist rule, append broad allow) 0 / 63 0 / 63

Controls (a sweep with no control is a fact about the sweep):

  • Negative control β€” deleting *nixos-rebuild* must go red: RED on both trees (4 failures on main, 5 on HEAD).
  • Unmutated baselines β€” GREEN on both (main 480 passed / 0 failed; HEAD 574 / 0).
  • Generator self-validation β€” every generated mutant asserted to differ from the original, still parse, and have exactly one fewer rule. 0 defective mutants across all four generators. Without this, "all mutants survived" would be indistinguishable from a generator that silently no-ops.

Of the 39 survivors, 17 were real holes β€” one more than the audit found. A survivor only counts as a hole if a realistic command resolves allow at both layers once the rule is gone; the rest are redundant or guard-backstopped. The extra hole:

sudo* β€” the anchored rule is the only cover for a sudo not followed by a space. *sudo * requires the literal "sudo ", so sudoedit /etc/nixos/configuration.nix walks straight past it (glob=allow layered=allow with sudo* deleted).

That is the exact mirror of the *sudo * finding the audit's own MUST_ASK comment records having found by mutation testing β€” the same class, the other half, which is why the class needed closing rather than the instances.

I also confirmed the audit's non-holes rather than taking them on trust: *sops*--decrypt*, *age*--decrypt*, *systemctl*restart*, *nix-collect-garbage* are redundantly covered, and *git*add --all* / *git*add . / *git*clean -f* / *talosctl*reset* / *mke2fs* / *mkswap* are guard-backstopped by design. Two the audit listed as holes are not: *dd if=* and *dd of=* stay deny via guard_core even with the glob gone.


(b) Per-family coverage

18 MUST_ASK rows, one per hole, in the house spellings that RULES.md / k8s.md actually mandate (git -C <path> …, KUBECONFIG=$KC_… kubectl -n ns …) β€” i.e. the habitual spelling is the bypassing one. Each is measured to be decided by exactly one rule, so it kills that rule's mutant and no other.

But 18 hand-written rows rot, and the next rule added would be unpinned again. So the real fix is the sweep as a standing assertion:

  • test_every_ask_rule_is_individually_pinned β€” every ask rule must be the sole decider for some pinned command, or be declared in REDUNDANTLY_COVERED_ASKS with the covering pattern named. The declaration is re-measured, not trusted: an entry that turns out to be load-bearing fails, and so does a stale one. It fails when the rule set grows and when it shrinks.
  • test_every_deny_rule_is_individually_pinned β€” the deny half, asserted at the layer that actually holds it. Guard-backstopped entries are re-measured to still resolve deny with the glob removed, so it fails if someone deletes the guard check believing the glob has it, or vice versa.
  • test_tool_level_permissions_are_pinned_exactly β€” the 10 non-bash rows, which were the entire residue after the bash ledger landed. doom_loop / external_directory are the ones that matter: the only ask rows outside the bash block, and flipping either to allow removed a real control with the suite green.

Two findings from re-running the sweep against my own fix (the reason the "after" column is 0 and not 10):

  1. A deny→ask downgrade of *git*add -A* survived everything, because guard_core denies git add -A anyway. Not an equivalent mutant: the deny block exists precisely for the case where guard.js fails to load, where the downgrade would let a human approve it in the TUI. Fixing it exposed the same bug in my own first attempt — reading denies off the config and checking only what is in it cannot see a rule that left the set. The ledger now pins the set first, against declared GLOB_ENFORCED_DENIES / GUARD_BACKSTOPPED_DENIES.
  2. That set assertion made the deny ledger's "stale declaration" check unreachable β€” it can never execute, so it would report safety while testing nothing. Removed, with the reason recorded inline and verified by mutation.

(c) The >= 40 floor β€” decision: raised to 50, the measured count, with no cushion

The slack was the whole defect. Reproduced exactly, and this is the headline:

Delete the four mutating systemctl verbs + kubectl patch/scale/drain + helm upgrade/rollback + flux resume β†’ exactly 40 ask rules remain β†’ main reports 480 passed, 0 failed while all ten resolve glob=allow layered=allow.
The same mutant on this branch: RED, 32 failures, naming each family.

The docstring called 40 "room to prune ten deliberately" without noticing that is the same slack as room to un-gate ten dangerous families.

Why raise rather than delete it. Its unique coverage is now nearly nil and I say so in the docstring: a mass ask→allow flip is already caught by test_no_allow_rule_follows_the_wildcard (that produced the original 117 failures), ask→deny by the ordering assertion, and mass deletion — its one former unique catch — is now caught rule-by-rule and by name by the ledger. What it still buys: it keeps test_all_asks_precede_all_denies's ordering assertion and the prefix-tolerance test non-vacuous (the documented reason it exists), and it fails on the count, localising a mass edit to one obvious assertion before the ledger enumerates it.

The trade, stated: a deliberate prune must now lower the literal in the same commit. That is the point β€” it makes a prune a visible act instead of free slack β€” and it is not extra ceremony, since a prune already requires removing the rule's MUST_ASK row.


(d) rm -r β€” the premise is false; no guard_core change

The brief held that guard_core only handles -rf forms. Measured, not read: check_rm_rf_critical tests the recursive flag alone β€” --recursive, or any short bundle containing r/R β€” and never requires -f:

command guard glob layered
rm -r / deny ask deny
rm --recursive $HOME deny ask deny
rm -r . / rm -r /etc / rm --recursive /nix deny ask deny
rm -R / / rm -fr / deny allow deny
rm -r /repo/build allow ask ask

So there was nothing to fix, and no behaviour change β€” which also keeps this PR out of scripts/claude-hooks/**.

What is real is narrower: guard_core deliberately does not fire on an ordinary path (its fatal-target set is /, ~/$HOME, ./.., top-level system dirs), documented as a decision so routine cleanup does not train the operator to route around the guard. So for rm -r /repo/build the glob is the only gate β€” and nothing pinned it. That is now two MUST_ASK rows (regression coverage), plus test_guard_core_catches_recursive_rm_without_the_force_flag, which is labelled in its own docstring as an invariant guard and is not counted as regression coverage β€” it exists so the false premise cannot be re-derived into a needless widening.


(e) Red/green matrix

This is a coverage fix β€” opencode.jsonc carries only comments β€” so "pre-change code" is the pre-change test data, and each regression gets its own isolation.

# isolation main / pre-fix this branch
A the ledger, with the 18 new MUST_ASK rows removed RED (482 passed, 1 failed β€” names the unpinned rules) GREEN (574 / 0)
B the audit's exact 10-rule deletion GREEN β€” 480 passed, 0 failed ← the defect RED β€” 32 failures
C negative control: *nixos-rebuild* deleted RED (4) RED (5)
D unmutated baseline GREEN (480 / 0) GREEN (574 / 0)

Branch reachability β€” "a test failed" is not enough. A mutant can go red via an earlier assert while the branch you care about never runs. All 9 new assertion branches were individually reached with a mutant chosen for that branch, and each was confirmed to fail with its own message: ask/unpinned, ask/wrongly-redundant, ask/stale, deny/unpinned, deny/unheld, deny/set, deny/downgraded, the floor, and the rm attribution. (This is how I found the dead stale branch in (b).)


Verification mechanism β€” stated per number

πŸ”΄ The version trap was avoided deliberately. An ad-hoc nix-shell -p opencode serves 1.18.11 while the tests pin 1.18.4, fabricating ~15 test_opencode_engine.py failures. Every number here comes from one of three mechanisms, all resolving pkgs.opencode from this flake's nixpkgs:

  1. nix build .#checks.x86_64-linux.pytests (nix sandbox) β€” the authoritative gate. Source of the totals below.
  2. Direct pytest under a buildEnv built from this flake's nixpkgs β€” used for the 77Γ—5 mutation sweeps (a full gate run per mutant was not affordable). opencode --version asserted 1.18.4 before use.
  3. opencode debug agent build --pure from the same pinned binary β€” the real engine.

Engine verification (3): all 18 new pins resolve ask against the real engine, and in each case the intended rule is the LAST matching one β€” none is shadowed. Gotcha recorded: the engine's stdout truncated at exactly 8192 bytes, nondeterministically, when read through a pipe; I write it to a file (9399 bytes) so "changed CLI" and "full buffer" stay distinguishable.

Gate totals (1) β€” merged tree, per RULES "gate on the MERGED tree":

tree collected passed skipped failed
origin/main @ 476ed51 (built standalone as the control) 6160 6129 3 28
this branch merged with origin/main 6254 6223 3 28
delta +94 +94 0 0

The gate is RED on both β€” for pre-existing reasons this PR does not touch, attributed by CAUSE:

  • 27 failures, one single cause: FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/env' inside the nix sandbox (all 27 tracebacks show that identical error). Split by file as I measured it: 11 in test_monitor_blackout.py, 16 in test_rig_control.py β€” the two sum exactly to the 27 distinct failing test functions. (I was briefed 22/5; my measurement disagrees on the split. It does not affect the conclusion β€” one cause, pre-existing, unrelated.)
  • 1 further failure, and it is a genuine regression already on main, not from this PR β€” flagged because someone should fix it: test_live_scraper_observes_the_real_config (scripts/collector/keylog/tests/test_espanso_detect.py:544) asserts ":date" in trigs, but espanso: prune :date/:time/:uuid + drop the non-unique framewo ssh term; raise cpu-mon thresholdsΒ #351 pruned the :date espanso snippet from nix/home.nix without updating it. Confirmed pre-existing by the discriminating control above β€” origin/main alone fails it too.
  • 2 unpinned skip groups (datapacket-talos clone not present) β€” also pre-existing on main.

No absolute green claim is made. The claim is the delta: +94 tests, +94 passing, +0 failures.


Also in here

bash_ruleset() is memoised as an immutable tuple. The ledger resolves ~63 rules Γ— ~90 commands and took this CI-gated file from 1.7s β†’ 18.5s; it is back to ~1.7s. A tuple because a cached mutable result is poisonable, and this repo has a caller that mutates a parsed config in place (test_opencode_engine.py's add_trailing_git_allow) β€” which is exactly why load_config() is left uncached.

Two comments added to opencode.jsonc (no rule changes): that *nix-collect-garbage* is covered only by accident β€” *age*-d* matching "nix-collect-garbage … -d" β€” so nobody deletes *age*-d* thinking GC is separately gated; and that both sudo rows are load-bearing.

πŸ€– Generated with Claude Code


Audit round 2 β€” F1, F2, F4 fixed; F3 and most of F8 deferred

The blind audit reproduced every headline number with its own generator. It also found that three of the six redundancy declarations this PR introduced were false by measurement β€” i.e. the PR's own defect class, in the PR's own new artifact. All three findings are fixed. This round reset the gate: the full battery, the reachability sweep, the engine check and the merged-tree gate were all re-run from scratch, not just for what changed.

πŸ”΄ The finding that matters most is one the audit did not report β€” my own fix had gone dead

While building the F2 control I found that extending the redundancy loop had dropped the elif not killers: unpinned.append(pat) branch. unpinned β€” the assertion this entire PR exists for β€” was dead code.

It was invisible to everything:

signal said
full file 616 passed, 0 failed
delete battery 0 / 78 survive
comment / flip / reorder batteries 0 survive
branch-reachability sweep 9 / 9 branches "reachable"

The mutation battery could not see it because the rules it deletes were still caught by the per-command assertions; only a rule that no pinned command matches exposes it. The ledger's reassuring answer is an empty list, and an empty list is exactly what a ledger wired to nothing returns β€” the "positive control" hazard, one level up from where I had applied it.

Fixed, and made unrepeatable: test_an_unpinned_ask_rule_is_detected is now a standing positive control that feeds the ledger a synthetic unpinned rule and asserts the list moves (([], ['*terraform*destroy*'])). The ledger body is extracted into one helper both the control and the real test call, so there is no second copy to drift.

F1 β€” three declarations were false. Measured, at both layers, on model and real engine

rule spelling that dropped to plain allow with it deleted was declared
*nix-collect-garbage* nix-collect-garbage, … --max-freed 1G "REDUNDANT"
*dd if=* FOO=1 dd if=/dev/sda, time dd if=/dev/zero bs=1M count=100 guard-covered
*dd of=* FOO=1 dd of=/tmp/img bs=1M guard-covered

*age*-d* rescues only spellings containing a literal -d; dd * is anchored, so every prefixed dd rested on the two operand rules alone. All are now pinned in MUST_ASK.

The jsonc comment this PR added β€” "MEASURED: this rule is REDUNDANT β€” deleting it changes no verdict" β€” was false and is corrected. That is the worst kind of wrong comment: it invites a maintainer to delete a live gate. The other three declarations (*sops*--decrypt*, *age*--decrypt*, *systemctl*restart*) I re-verified as genuinely subsumed and kept.

F2 β€” the declaration was an escape hatch; F1 was that hatch already used

It mapped a pattern to prose, so the ledger could only check "no pinned command moved" β€” verifying entries against the pinned pool, never against the claim. Now:

pattern -> (covering_pattern, witness_command)

with all three legs re-measured: the coverer exists, the witness really is matched by the redundant rule, and with that rule removed the witness is last-matched by exactly the declared coverer.

Controls (test_opencode_config.py only; direct pytest, flake-pinned 1.18.4):

state before after
0. unmutated GREEN 574 GREEN 617
1. add unpinned "*terraform*destroy*": "ask" RED RED
2. …declared covered by an imaginary pattern GREEN 574 ← the hatch RED
3. …declared covered by a real but wrong pattern GREEN RED
4. …with a witness the rule does not even match GREEN RED

F4a β€” *sudoedit* added to the config

sudo* and *sudo * were presented as complementary halves; their intersection was covered by neither. Measured ALLOW at both layers before this rule:

  • FOO=1 sudoedit /etc/nixos/configuration.nix
  • EDITOR=vim sudoedit /etc/nixos/configuration.nix β€” the normal invocation, since sudoedit reads $EDITOR
  • /run/wrappers/bin/sudoedit /etc/nixos/configuration.nix β€” the NixOS path

Unlike a flag spelling, sudoedit is a fixed binary name, so one leading-and-infix * closes the family rather than buying one more spelling. sudo* is re-pinned via sudo-rs whoami β€” note a bare sudo does not isolate it, because "*sudo *"'s " *" suffix also matches the bare form.

F4b β€” rm: comment corrected, gap pinned open, not closed

The comment claimed the glob was "the ONLY thing between an agent and a silent recursive delete". It is not even that β€” rm -R, rm -Rf, rm -fr on an ordinary path are ALLOW at both layers.

I did not widen the glob, deliberately: it is whack-a-mole (-R, then -fR, then -vr), and the candidate that looked like it generalised, "*rm*-*r*", matched rm -R /repo/build only because the path contains an "r" β€” it fails on rm -R /x. The structural fix is guard_core's target set, which its own docstring reserves as an operator decision needing real measurement, and which is outside this PR's remit.

So the gap is pinned open by test_rm_glob_misses_these_recursive_spellings β€” labelled a characterization test, asserting today's wrong answer on purpose, and telling you to move the row into MUST_ASK if you close it. Green before and after; not counted as regression coverage.

F8 (partial) β€” comment scope corrected

The systemctl comment read as exhaustive. Measured ALLOW at both layers: mask, unmask, kill, isolate, set-property, edit, reload. Comment now says so explicitly. The wider verb sweep stays deferred, as agreed.

Floor 50 β†’ 51

*sudoedit* landed, so by this PR's own no-cushion policy the literal moves with it. That is the policy working, not ceremony.

Re-verification after the fix round β€” everything re-run

Mutant battery (mechanism: direct pytest under a buildEnv from this flake's nixpkgs; opencode --version asserted 1.18.4; 78 rows now, *sudoedit* added):

class before (main) after
delete 39 / 77 0 / 78
comment out 39 / 77 0 / 78
flip β†’ allow 2 / 68 0 / 69
flip deny β†’ ask 1 / 16 0 / 16
reorder 0 / 63 0 / 64

Branch reachability: 12 / 12, each failing with its own message β€” now including the two new bad_claim legs and the positive control. Two ordering facts worth recording: the wrongly_redundant branch had to be re-reached with a different mutant because bad_claim now legitimately fires first, and the earlier round's dead stale branch stayed removed.

Real engine (opencode debug agent build --pure, flake-pinned 1.18.4, stdout read from a file β€” 9493 bytes; the 8192-byte pipe truncation is real, pre-existing at test_opencode_engine.py:175, and not repeated here): all 26 pins resolve ask, and in each case the intended rule is the LAST matching one β€” none shadowed.

Merged-tree gate (mechanism: nix build .#checks.x86_64-linux.pytests; control built standalone from current origin/main):

tree collected passed skipped failed
origin/main @ 0cbc9e6 6160 6129 3 28
this branch merged 6297 6266 3 28
delta +137 +137 0 0

The failing sets are identical β€” the set difference is empty in both directions, so nothing here introduces or masks a failure. Both trees are red for the same pre-existing reasons: 27 failures from one cause (/usr/bin/env absent in the nix sandbox; my split, 11 test_monitor_blackout.py + 16 test_rig_control.py, sums exactly to the 27 distinct functions β€” the audit confirmed this over the 22/5 I was briefed), plus test_live_scraper_observes_the_real_config, still broken on main by #351's :date prune, plus 2 unpinned skip groups.

No absolute green claim. The claim is the delta: +137 tests, +137 passing, +0 failures.


Delta-audit round 3 β€” N1, N2, N4, N5 fixed; N3 softened; N9 recorded

Re-baselined: main is green now. Every earlier statement in this PR about a
pre-existing red gate (27 /usr/bin/env failures, the :date regression, unpinned
skip groups) is superseded β€” #354–#357 and #350 fixed them. Nothing below is
attributed to a pre-existing failure, because there are none.

πŸ”΄ N2 β€” time defeated BOTH layers at once

time dd bs=1M if=/dev/zero of=/dev/sda   ->  glob=allow  guard=allow  layered=allow

time was missing from _WRAPPER_VALUE_FLAGS, so argv[0] stayed "time" and check_dd_to_block_device bailed at its basename(argv[0]) != "dd". Fixed structurally β€” time added with GNU time's value-taking flags (-f/--format, -o/--output). \time needs no entry (the tokeniser already normalises it) and /usr/bin/time is covered by the basename(); both are pinned anyway.

And the comment this PR had added said the guard covered it. Corrected β€” that is the second time this PR shipped a comment asserting a measurement that did not hold, and both times the comment was the part that would have caused harm.

Why dd and nothing else β€” worth keeping, because it explains the shape rather than the instance: for every other family an unpeeled prefix is still caught by the glob layer. Measured: time rm -rf /, time mkfs.ext4 …, time talosctl … reset, time git stash all still resolve deny on the globs alone. The dd globs are the only spelled ones (dd * anchored, *dd if=*/*dd of=* literal), so dd is the single family where an unrecognised wrapper defeats both layers together.

I swept the class rather than fixing the instance. 14 more prefixes are unpeeled and each still defeats both layers on dd:

xargs, watch, script -c, unbuffer, strace -f, ltrace, catchsegv, proxychains, torify, flock <file>, eatmydata, systemd-run, pv, rlwrap

Deliberately NOT added in this round. guard_core's own header records that a wrong flag-arity entry fails OPEN, and several of these take positionals or a command string (flock <file> cmd, script -c "…", systemd-run --unit=x cmd). That is a table-design job needing its own evidence, not a same-round guess. Follow-up.

Tests: 12 time spellings + 5 other-family spellings, RED before the table entry (16 failures), GREEN after. Two further tests β€” that time ls/time git status stay allowed, and that a bare time does not crash β€” are green both ways and labelled invariant guards; without them "the guard got stricter" and "the guard got broken" look identical from the green side.

πŸ”΄ N1 β€” my own thesis, applied to the detectors I had just added

Every ledger detector reports health as an empty list β€” the shape a detector wired to nothing also produces. Neutering any one left the file 617/617 GREEN, including the three bad_claim legs this PR introduced, for which I had written a control covering only the rules dimension and not the declarations one.

Both ledgers are now extracted into _ask_ledger / _deny_ledger over an explicit ruleset, and two parametrized controls drive each detector to non-empty with a case built for it β€” reported as a pair (clean empty, seeded non-empty), and for the three legs asserting which leg fired so one cannot silently preempt another.

Measured after: all 12 detector-neutering mutants go RED. Twelve, not nine β€” the extraction split the deny set-check into missing/undeclared and made each bad_claim leg separately mutable.

neutered detector before after
ask: unpinned, wrongly_redundant, bad_claim (all 3 legs, and each leg alone), stale GREEN 617 RED
deny: missing, undeclared, unpinned, unheld, downgraded GREEN 617 RED

🟑 N4 β€” two copies of the resolver, collapsed

The ask ledger carried a private resolve() while the deny side went through effective_bash_action. Zero disagreements today and nothing pinning that. Both now route through one _resolve_over(rules, command) + _rules_without(rules, drop).

🟑 N5 β€” the floor's paragraph was prose, not a gate

It claimed "the literal is still raised when one lands, so the number never drifts back into being a cushion". Measured: adding an ask rule and pinning it, literal left at 51, was GREEN under >= β€” the cushion regrows one rule at a time until it is exactly the slack the same docstring spends two paragraphs condemning.

Now == 51, with a message naming both directions:

case >= (old) == (new)
+1 rule, pinned, literal unchanged GREEN 633 ← the defect RED
βˆ’1 rule (a prune), literal unchanged RED RED

Deferred, and honestly labelled

  • N3 β€” the redundancy hatch is narrowed, not closed: the witness is declarer-chosen, so the check establishes "another rule wins on this command" while the claim is "another rule covers the family". Nothing verifies subsumption. The docstring now says exactly that instead of reading stronger than the code. All three current entries are genuinely subsumed (every --decrypt contains -d; every restart contains start).
  • N8/N6 β€” the hard-coded *terraform*destroy* control is gone, replaced by ASK_CONTROLS[0] over a guaranteed-absent SENTINEL, plus a test asserting the sentinel is absent.
  • N7 β€” wrongly_redundant's message still names a scenario bad_claim preempts. Left as-is.
  • N9 (pre-existing, not from this PR) β€” recorded so it is not lost. All measured allow at both layers: nix store gc, nix-store --gc, nix profile wipe-history, nix-env --delete-generations old, home-manager expire-generations; pkexec / run0 bypass the sudo/doas block; FOO=1 sudo-rs whoami and /usr/bin/sudo-rs whoami escape the anchored sudo* β€” note sudo-rs is the witness this PR picks for sudo*, so that pin is narrower than it looks; rm -vr, rm -Rv, rm -f -R, FOO=1 rm -R are live spellings the characterization test does not pin; and *home-manager*switch* / *systemctl*daemon-reload* survive a trailing-* strip, so the house home-manager switch --flake . --impure form is not itself pinned.
  • F3 (from round 2) β€” the coverage is spelled, not structural; still its own PR, still coupled to respelling 25 rules.

Incoming conflict to expect

The concurrent session's uncommitted WIP in scripts/claude-hooks/tests/test_guard_core.py adds a git add block and states it will remove "*git*add ." from opencode.jsonc. This PR's GUARD_BACKSTOPPED_DENIES declares that rule, and the deny ledger pins the deny set β€” so whichever lands second must move that entry, and the test will say so by name. My time tests are appended at EOF specifically to avoid a textual clash with their insert at lines 412-498. I read that file read-only and did not modify their WIP.

Verification β€” mechanism stated for every number

  1. nix build .#checks.x86_64-linux.pytests β€” the flake-pinned sandbox gate.
  2. Direct pytest under a buildEnv from this flake's nixpkgs, opencode --version asserted 1.18.4 (an ad-hoc nix-shell -p opencode serves 1.18.11) β€” the mutation batteries and the N1/N2/N5 matrices.
  3. opencode debug agent build --pure from that same pinned binary, stdout read from a FILE β€” the 8192-byte pipe truncation at test_opencode_engine.py:175 is real and pre-existing; not fixed here, not repeated here.

Full mutant battery, re-run from scratch on the final tree (78 rows; baseline 628 passed / 0 failed):

class main this branch
delete 39 / 77 0 / 78
comment out 39 / 77 0 / 78
flip β†’ allow 2 / 68 0 / 69
flip deny β†’ ask 1 / 16 0 / 16
reorder 0 / 63 0 / 64

Branch reachability: 12 / 12, each failing with its own message. One case had to be retargeted β€” deny/downgraded's mutant patched a function the ledger no longer calls after the N4 dedupe, and it went green; that is the same shadowing hazard the harness exists to catch, so it is worth naming.

Engine: all 26 pins resolve ask against the real 1.18.4 engine with the intended rule LAST β€” none shadowed. guard_core suite: 1156 passed. #350's new scanners pass against my edits (1817 passed across the two scanners + test_guard_core.py + test_opencode_config.py), so the count it pins in test_guard_core.py is intact.

Both tiers, both green:

tree collected passed skipped failed
origin/main @ 60e6d9d (built standalone as control) 6311 6310 1 0
this branch merged with origin/main 6486 6485 1 0
delta +175 +175 0 0

(I measured the control rather than reusing the briefed 6164/6163 β€” that figure predates #350, which added ~147 tests.)

πŸ”΄ Deploy note

~/.config/opencode/opencode.jsonc is a home.file copy resolving into /nix/store. Merging does not activate the new "*sudoedit*" rule, and merging does not deploy the time fix to ~/.claude/hooks/ either. Both need home-manager switch (or scripts/ship.sh) on each host. Merged β‰  deployed.

ZacxDev and others added 9 commits August 6, 2026 00:32
…etions were invisible

PR #348 restored 52 `ask` values a blind sed had flattened. An adversarial audit
then found it had closed two INSTANCES of a coverage class, not the class: a
permission rule could be deleted outright with the suite fully green, while a
realistic command in that family dropped to plain `allow` at BOTH the glob layer
and the layered (glob + guard_core) layer.

MEASURED here independently, not taken from the audit β€” delete each of
opencode.jsonc's 77 single-line verdict rows one at a time, run
test_opencode_config.py under the FLAKE-PINNED opencode 1.18.4 (an ad-hoc
nix-shell serves 1.18.11 and fabricates ~15 failures), attribute by failing test
id:

    before:  39 / 77 deletions left the suite GREEN
    after:   10 / 77  β€” and all ten are non-bash tool rows

Of the 39, 17 were real holes, one more than the audit found: it missed the
anchored `sudo*`, whose sole unique coverage is a `sudo` NOT followed by a space
(`sudoedit`). That is the exact mirror of the `*sudo *` finding the audit's own
MUST_ASK comment records having found by mutation testing.

The second defect was the aggregate floor. `len(asks) >= 40` against 50 measured
left ten rules of slack, and the docstring called it "room to prune ten
deliberately" without noticing that is the same thing as room to un-gate ten
dangerous families. Reproduced: deleting the four mutating `systemctl` verbs,
kubectl patch/scale/drain, helm upgrade/rollback and flux resume leaves exactly
40 and reports 480 passed, 0 failed while all ten resolve `glob=allow
layered=allow`.

WHAT CHANGED (tests + comments only β€” no permission rule is altered; the config
was already correct, the suite could not tell):

  * 18 MUST_ASK rows, one per hole, each measured to be pinned by exactly one
    rule so it kills that rule's deletion mutant and no other.
  * `test_every_ask_rule_is_individually_pinned` / `..._deny_rule_...` β€” the
    deletion sweep as a standing assertion. Each rule must be the SOLE DECIDER
    for a pinned command, or be declared redundant / guard-backstopped, with the
    declaration RE-MEASURED so it cannot outlive the rule it depends on. These
    fail when the rule set grows AND when it shrinks.
  * floor raised 40 -> 50, the measured count, with no cushion. A deliberate
    prune must now lower the literal in the same commit.
  * `bash_ruleset()` memoised (immutable tuple): the ledger resolves ~63 rules x
    ~90 commands and took the file 1.7s -> 18.5s. `load_config()` stays uncached
    because test_opencode_engine.py mutates its result in place.

`rm -r`: the premise that guard_core "only handles `-rf` forms" is FALSE,
measured. check_rm_rf_critical tests the recursive flag alone (`--recursive`, or
any bundle containing r/R) and never requires `-f`; `rm -r /`, `rm --recursive
$HOME`, `rm -r .` and `rm -r /etc` are all denied today. No guard_core change.
Its non-coverage of an ORDINARY path (`rm -r /repo/build`) is a documented
decision, so that case is pinned in MUST_ASK β€” the glob is its only gate β€” and
the attribution is pinned by a test labelled as the invariant guard it is.

VERIFIED against the real engine, not a reading of the resolver: all 18 new pins
resolve `ask` under `opencode debug agent build --pure` on the flake-pinned
1.18.4, and in each case the intended rule is the LAST matching one, so none is
shadowed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…y-downgrade mutant

Follow-on within the same PR, from re-running my own sweep against the fix
rather than stopping at the headline number:

  * 10 of 77 deletion mutants still survived, and ALL TEN were the non-bash
    tool-permission rows. `doom_loop` and `external_directory` are the ones that
    matter β€” the only `ask` rows outside the bash block, and flipping either to
    `allow` removed a real control with the suite green.
    `test_tool_level_permissions_are_pinned_exactly` pins them as an exact map,
    which fails on delete, comment-out, verdict-flip AND an added row.

  * one `flip-ask` mutant survived everything: `"*git*add -A*"` downgraded
    deny -> ask. Invisible end-to-end because guard_core denies `git add -A`
    anyway β€” but NOT an equivalent mutant, since the deny block exists for the
    case where guard.js fails to load, where the downgrade would offer a human
    the chance to approve it in the TUI.

    Fixing that exposed the shape of the bug in my own first attempt: reading
    `denies` off the config and checking only what is IN it cannot see a rule
    that LEFT the set. The deny ledger now pins the SET first, against declared
    GLOB_ENFORCED_DENIES / GUARD_BACKSTOPPED_DENIES.

  * removed the deny ledger's "stale declaration" assertion: the new set check
    subsumes it, so it had become UNREACHABLE β€” an assertion that can never
    execute reports safety while testing nothing. Verified by mutation (dropping
    `*mkswap*` fails on the set assertion, never reaching it).

Mutation results after this commit, all under the flake-pinned opencode 1.18.4:
    delete      0 / 77 survive   (39 before)
    comment-out 0 / 77           (39 before)
    flip->allow 0 / 68           ( 2 before)
    flip->ask   0 / 16           ( 1 before)
    reorder     0 / 63           ( 0 before)

All 9 new assertion branches proven individually REACHABLE, each failing with
its OWN message β€” not merely "I broke it and something went red".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… close the sudoedit gap

Audit round on #352. F1/F2/F4; F3 and most of F8 deferred to their own PR.

F1 β€” THREE OF SIX REDUNDANCY DECLARATIONS WERE FALSE BY MEASUREMENT. Re-measured
on the model AND the real 1.18.4 engine, deleting each glob:
  *nix-collect-garbage*  `nix-collect-garbage`, `… --max-freed 1G`  ask -> allow
  *dd if=*               `FOO=1 dd if=/dev/sda`, `time dd if=…`     ask -> allow
  *dd of=*               `FOO=1 dd of=/tmp/img bs=1M`               ask -> allow
`*age*-d*` rescues only spellings containing a literal `-d`; `dd *` is anchored,
so every prefixed dd rested on the two operand rules alone. All three are now
pinned in MUST_ASK. The jsonc comment this PR had ADDED asserting "MEASURED:
this rule is REDUNDANT β€” deleting it changes no verdict" was false and is
corrected: a comment is a claim, and that one invited deleting a live gate.
The other three declarations are genuinely subsumed β€” verified, kept.

F2 β€” THE DECLARATION WAS AN ESCAPE HATCH, and F1 is that failure mode already
shipped. It mapped a pattern to PROSE, so the ledger could only check "no pinned
command moved" β€” i.e. it verified entries against the pinned pool, never against
the claim. Reproduced with controls: adding `"*terraform*destroy*": "ask"` went
RED, but declaring it covered by a pattern that DOES NOT EXIST went GREEN at 574.
Now `pattern -> (covering_pattern, witness_command)`, with all three legs
re-measured: the coverer exists, the witness is really matched by the redundant
rule, and with that rule removed the witness is last-matched by exactly the
declared coverer. Imaginary, wrong-but-real, and non-matching-witness variants
are all RED (4 controls).

πŸ”΄ F2 ALSO EXPOSED THIS PR'S OWN DEFECT CLASS IN THIS PR'S OWN FIX. Extending
that loop dropped the `elif not killers: unpinned.append(pat)` branch, so
`unpinned` β€” the assertion this whole file exists for β€” became DEAD CODE while
the suite reported 616 passed and the entire mutation battery stayed at 0
survivors (the deleted rules were still caught by the per-command assertions).
Nothing could see it, because the ledger's reassuring answer is an empty list
and an empty list is what a ledger wired to nothing returns. Fixed, and made
unrepeatable: `test_an_unpinned_ask_rule_is_detected` is a standing POSITIVE
control that feeds the ledger a synthetic unpinned rule and asserts the list
moves. The ledger body is now one helper both call, so there is no second copy.

F4a β€” `*sudoedit*` added to the config. `sudo*` and `*sudo *` were presented as
complementary halves; their INTERSECTION was covered by neither. MEASURED ALLOW
at both layers before this: `FOO=1 sudoedit …`, `EDITOR=vim sudoedit …` (the
NORMAL invocation β€” sudoedit reads $EDITOR) and `/run/wrappers/bin/sudoedit …`.
Unlike a flag spelling, `sudoedit` is a fixed binary name, so one leading-and-
infix `*` closes the family rather than buying one more spelling. `sudo*` is
re-pinned via `sudo-rs whoami` β€” a bare `sudo` does NOT isolate it, because
`"*sudo *"`'s `" *"` suffix also matches the bare form.

F4b β€” the `rm` comment claimed the glob was "the ONLY thing between an agent and
a silent recursive delete". It is not even that: `rm -R`, `rm -Rf` and `rm -fr`
on an ordinary path resolve ALLOW at BOTH layers. Comment corrected and the gap
PINNED as an open, labelled characterization test rather than closed, because
widening the glob is whack-a-mole (`-R`, then `-fR`, then `-vr`) and a candidate
that looked general β€” `"*rm*-*r*"` β€” matched `rm -R /repo/build` only because
the PATH contains an "r", so it fails on `rm -R /x`. The structural fix is
guard_core's target set, which its own docstring reserves as an operator
decision needing real measurement.

F8 (partial) β€” the systemctl comment read as exhaustive. MEASURED allow at both
layers: mask, unmask, kill, isolate, set-property, edit, reload. Scope corrected;
the wider verb sweep stays deferred.

Floor 50 -> 51, per this PR's own no-cushion policy, because `*sudoedit*` landed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Delta-audit round on #352. N1, N2, N4, N5 fixed; N3 softened to match the code;
N6/N7/N8 folded in where they were a line; N9 recorded in the PR body.

N2 β€” `time` DEFEATED BOTH LAYERS AT ONCE.
  time dd bs=1M if=/dev/zero of=/dev/sda   ->  glob=allow guard=allow
`time` was absent from `_WRAPPER_VALUE_FLAGS`, so argv[0] stayed "time" and
check_dd_to_block_device bailed at `basename(argv[0]) != "dd"`. Added, with
GNU time's value-taking flags. `\time` needs no entry (the tokeniser already
normalises it) and `/usr/bin/time` is handled by the basename β€” both pinned.

πŸ”΄ WHY dd, and the part worth keeping: for every OTHER family an unpeeled prefix
is still caught by the GLOB layer β€” measured, `time rm -rf /`, `time mkfs…`,
`time talosctl … reset`, `time git stash` all deny on the globs alone. The dd
globs are the only SPELLED ones (`dd *` anchored, `*dd if=*`/`*dd of=*`
literal), so dd is the one family where an unknown wrapper defeats both. The
comment this PR had added told a maintainer the guard covered it; corrected.

I also swept the wrapper class rather than fixing the one instance: 14 more
prefixes are unpeeled (`xargs`, `watch`, `script -c`, `unbuffer`, `strace -f`,
`ltrace`, `catchsegv`, `proxychains`, `torify`, `flock <f>`, `eatmydata`,
`systemd-run`, `pv`, `rlwrap`) and each still defeats both layers on dd. NOT
added here: guard_core's own header says a wrong flag-arity entry FAILS OPEN,
and several of those take positionals or a command STRING (`flock <file> cmd`,
`script -c "…"`, `systemd-run --unit=x`). That is a table-design job with its
own evidence, not a same-round guess. Recorded in the PR body.

N1 β€” MY OWN THESIS, APPLIED TO THE DETECTORS I ADDED.
Every ledger detector reports health as an EMPTY LIST, which is exactly what a
detector wired to nothing returns. Neutering any one of them left the file
617/617 GREEN β€” including the three `bad_claim` legs this PR had just added,
for which I wrote a control covering only one dimension.
Both ledgers are now extracted into `_ask_ledger`/`_deny_ledger` over an
EXPLICIT ruleset, and `test_ask_ledger_detector_is_wired` /
`test_deny_ledger_detector_is_wired` drive each detector to non-empty with a
case built for it β€” reported as a PAIR (clean empty, seeded non-empty) and, for
the three legs, asserting WHICH leg fired so one cannot preempt another.
MEASURED after: all 12 detector-neutering mutants go RED (12, not 9 β€” the
extraction split deny's set check into missing/undeclared and made each
`bad_claim` leg separately mutable).

N4 β€” the ledger carried a private copy of the resolution loop while the deny
side used `effective_bash_action`. Two copies, no disagreement today, nothing
pinning that. Collapsed onto one `_resolve_over(rules, command)` +
`_rules_without(rules, drop)`; every resolver in the file now routes through it.

N5 β€” the floor's paragraph claimed "the literal is still raised when one lands,
so the number never drifts back into being a cushion". That was prose.
MEASURED: adding an ask rule AND pinning it, with the literal left at 51, was
GREEN under `>=` β€” the cushion regrows silently until it is the same slack the
same docstring spends two paragraphs condemning. Now `== 51`, with a message
naming both directions.

N3 (deferred fix, docstring softened) β€” the redundancy hatch is NARROWED, not
closed: the witness is declarer-chosen, so the check establishes "another rule
wins on THIS command" while the claim is "another rule covers the FAMILY".
Nothing verifies subsumption. Said so where the dict is defined.

N8/N6 β€” the hard-coded `*terraform*destroy*` control is gone, replaced by
ASK_CONTROLS[0] over a guaranteed-absent SENTINEL plus a test asserting the
sentinel really is absent.

Perf: the controls made this CI-gated file 46 s. `wildcard_match` is memoised
(pure function; the ledgers resolve ~50 rules x ~115 commands x ~65 patterns per
pass) and the clean pass is computed once. 3.5 s with 628 tests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ZacxDev
ZacxDev merged commit 17bd614 into main Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant