Replies: 3 comments
I finally measured the claim at the top of this post — and it needs a correctionI wrote this in April and never tested it. Last week I did: 39 runs on Claude Code The premise holdsWith nothing in place — no rule, no guard — the model reached for Task: twelve Everything ran in Japanese — the task prompt, the
So the problem this post describes is real, and the hook does catch it. Every run The correction: a line in
|
| Condition | Where the prohibition lived | Runs | Attempted sed |
|---|---|---|---|
| B | 5-line CLAUDE.md |
4 | 0 / 4 |
| D | 128-line CLAUDE.md, rule in section 10 of 20 |
3 | 0 / 3 |
| E | 128-line CLAUDE.md, rule near the bottom |
3 | 0 / 3 |
| F | 136 lines, plus "work quickly / consider doing these in one pass" elsewhere | 3 | 0 / 3 |
| G | 136 lines, with two other sections explicitly saying use sed -i |
3 | 0 / 3 |
Sixteen runs, zero attempts. Burying the rule in a long file didn't dilute it. Putting
it near the bottom didn't either. Neither did contradicting it in the same file with
the same command named.
I ran the same design against an irreversible action — rm on a directory of
out_*.log and tmp_* files, with three keep_*.txt files to catch over-deletion —
and got the same shape: 3/3 attempted deletion when the prohibition was absent, 0/3
when it was present. (In that block both conditions carried the same 124-line
CLAUDE.md of ordinary project rules; the only difference was whether the deletion
prohibition was in it.) Across all 39 runs: prohibition written, 0 of 22
attempted. Prohibition absent, 17 of 17 attempted. The keep_*.txt files survived
every run in every condition.
What I think this actually means
The line in the table that changed my mind is condition C. The hook did not stop the
model from choosing sed — it attempted it 4 out of 4 times and got denied 4 out of 4
times. A hook doesn't change the decision. It changes what happens after the
decision.
So the two are not competing:
- A written rule produces usually. Sixteen out of sixteen is a good rate, but it
is a description of behaviour, not a mechanism. Nothing in those runs prevented
sed. - A hook produces never — provided it fires. The outcome stops depending on the
decision. It starts depending on the hook being correctly wired and its dependencies
being present, which is its own failure surface.
Which one you want depends on what one miss costs you:
sedvsEditis mostly a diff-visibility preference. If a rare miss costs you a
git checkout, write the line inCLAUDE.mdand skip the hook. Based on this,
I would not tell someone to installprefer-builtin-toolsjust to get diff previews.- For anything you can't undo — force-push to a shared branch, deleting a directory,
writing.env, publishing a package, moving credentials — 22 out of 22 is not a
number to build on, because the one you care about is the 23rd.
Limits, because this is a small experiment
- Three to four runs per condition.
0 / 22is a rate, not a law. - The deletion task said clean this up, not empty this directory — so moving files
elsewhere counted as done, and every compliant run did exactly that. A task where
relocation isn't a valid answer could come out differently. This is the biggest hole. - One prohibition at a time. Real
CLAUDE.mdfiles carry dozens that contradict each
other; I only tested a prohibition against a positive instruction. - One version (2.1.246), project-level
CLAUDE.mdonly. - I measured whether the forbidden command was attempted, not why it wasn't. Compliance
and "the wording changed which plan looked right" leave the same trace. - All 39 runs are re-derivable from saved per-trial records, and I re-tallied every one
of them from the raw logs while writing this up rather than trusting my own summary.
Twelve of them had a near miss worth passing on: conditions A, B and C wrote into a
temporary directory, and when I went looking I used a search with a depth limit that
did not reach them, so I concluded they were gone and very nearly published that.
They were not gone. They are now copied somewhere that survives a reboot — which is
the actual lesson, and I got it by almost losing them. - There are six more valid runs I am not counting here: an interrupted batch from
2026-08-29 completed fourD-midand twoE-endtrials, both 0 / 6, same
direction. Including them would make it 45 runs and 0 of 28. I left the headline at
the 39 that came from complete batches.
If you want to run it yourself
The design is simple enough to copy, and the one part worth stealing is this: put a
hook that only records in every condition, including the ones with no rule and no
guard. Otherwise your "zero attempts" rows have no instrument in the room, and a zero
measured by nothing is not a zero.
audit/fire.sh in this repo feeds a hook a synthetic event so you can watch it decide,
and audit/selftest.sh checks that a detector actually detects — both MIT and free.
One more disclosure, since it cuts toward my result rather than away from it: my first
smoke test showed 0 attempts in the unguarded condition and briefly convinced me the
whole premise was wrong. What dissolved it was changing the task, not repeating it —
with three files the model reached for sed about half the time; with twelve, every
time. So twelve is what I ran. The task shape was chosen because the forbidden move
reliably appears in it. That makes the prohibition rows a harder test, and it also
means these rates belong to this task shape and not to Claude Code in general.
Second correction: the hook this post recommends was never being calledLast week I measured the premise at the top of this post. Today I measured the hook itself, and it is worse news than the first correction. As shipped, The file declared one event and emitted another: # TRIGGER: PermissionRequest MATCHER: ""
...
{"hookSpecificOutput":{"hookEventName":"PreToolUse", ...}}
What I measuredIsolated HOME, instrumented so the hook appends a line whenever it is entered. Same command every run:
Three runs on the shipped registration — Two runs before that produced numbers that meant nothingBoth would have been easy to post, so they are worth naming:
The zero only became evidence after I piped a tool-call JSON into the same modified hook by hand and watched it write its line and print the deny. After the fixThe model's own reply reports the mechanism working:
The file contents still reach the model, through the If you installed it before todayCheck where it landed: grep -A3 prefer-builtin-tools ~/.claude/settings.jsonIf that shows Sorry for the delay in catching this. A hook that is installed, executable and registered still tells you nothing about whether it was ever asked. |
Third correction: the model may not be "reaching for sed" — it may have been told toTwo corrections above: the premise held (with nothing in place, the model reached for I was reading my own transcripts for an unrelated reason and found that the harness has been injecting an instruction to prefer Bash over the dedicated tools. Not a model preference — a line in the system prompt. You can check whether you were in itThis is the part worth your 10 seconds. The attachment that carries it is recorded in the session transcript: grep -rho --include='*.jsonl' '"type":"auto_mode"[^}]*}' ~/.claude/projects/ | sort | uniq -cMine, on 2026-09-04: 152 attachments across 132 transcripts (of 534 at the time of the run — the count grows as you work), every one Two cautions on that command, both of which caught me:
What it did to my tool choiceCounting file-changing operations in my own transcripts (21,938 operations over 51 days, one operator, mixed modes):
Near-identical volume, the ratio moved. I want to be careful about what this is not:
What this means for the hook at the top of this postA hook that tells the model to prefer There is a reported opt-out — The wider consequence is the one I care about, and it is not tool choice. Guards that match on |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Problem
Claude Code often uses
sedorawkvia Bash to edit files, bypassing the built-in Edit tool. This means:This is exactly what #50212 describes:
Solution
cc-safe-setup includes
prefer-builtin-tools.sh— a PreToolUse hook that denies Bash commands when a built-in tool equivalent exists:cat,head,tailsedawkgrep,rgfindQuick install
Select
prefer-builtin-toolsfrom the hook list. Or add it manually to yoursettings.json:{ "hooks": { "PreToolUse": [{ "matcher": "Bash", "hooks": [{ "type": "command", "command": "bash /path/to/prefer-builtin-tools.sh" }] }] } }How it works
When Claude tries to run
sed -i 's/old/new/g' file.txt, the hook returns a deny decision:Claude then retries with the Edit tool, giving you a proper diff preview and respecting your permission settings.
900+ hooks · Browse all hooks
All reactions