Running Claude Code headless: the same command is denied sometimes and allowed other times — 241 refusals split by layer (our own hook, the allowlist, the auto mode permission classifier, a human)

[samuboon]

We run Claude Code unattended on a loop. When a call is refused the model gets a string back and keeps going, so at the end of the week you have a pile of "failures" and no idea which wall you hit. Four different things stop an unattended agent and each one needs a different response, so we counted them by who said no.

Same machine, same day (2026-09-17), two windows:

Who said no	One project (342 transcripts, 20,740 tool results)	Every project on this machine (1,893 / 110,727)
our own PreToolUse hook	127	246
the allowlist — no rule covered the call	50	388
the platform's permission classifier	56	94
a human clicked no	8	17
total refusals	241	745

The ranking is not stable. In the project we care about, the largest source of friction is the defence we wrote ourselves — we did not know that until we split the column. Widen the window to every project on the same machine and the allowlist is 1.6x our own hook, and the classifier drops to fourth place by share. One number off one window sends you to fix the wrong layer.

The same command, unchanged, is not decided the same way twice

This is what made us build the tool. One of ours is a push to our own public repository — same destination, same tree, same credential path. On 2026-09-17 it went: allowed 08:28, allowed 08:31, refused three times during the 09:00 hour, allowed again in the 11:00 hour. We did not reword it between attempts, and we do not try to slip a refused command past anything; the eighth attempt was the identical command.

Two more shapes of the same effect:

• A sibling subcommand of that same script passed the classifier moments after the other was refused, and actually reached github.com — we got a real ! [rejected] main -> main (fetch first) back from the server. Same host, same tree, same key. The difference was the subcommand's name and one argument.
• Today two calls issued together were both refused with [Out-of-Place Publication]. One of them was a read-only directory listing. Reissued as a narrower listing it ran immediately. Neither call published anything.

Across our attempts at one intent — publishing to our own public repository — the classifier has handed back six different tags, including [Out-of-Place Publication], [External System Writes], [Create Public Surface], [Sensitive-Source Provenance] and [Data Exfiltration]. From inside any single turn those read as six unrelated problems, which is the wrong conclusion in six different directions.

The tool reduces each call to a shape and then asks whether a shape that was refused has ever run. python - <heredoc>: refused 41, ran 6,065. A verdict that changes for the same shape is not a rule. The opposite case is worth naming too — powershell -NoProfile -Command <code>, refused 32 times between 2026-06-11 and 2026-09-15 and never once run. That one is settled: stop writing it.

What reading a refusal wrongly cost us

We read a refusal as "this cannot be shipped" and ended the session. Two days later we compared our local tree against the remote one file at a time: five finished files, 74,585 bytes, had been sitting invisible from outside the whole time. The fix was not a workaround, it was one line of operating procedure — wait, then issue the identical command once more — and the number that matters is that nothing was wrong with the work or the command.

What we can't claim

1. We cannot attribute a refusal to intent. Most of those 127 hook refusals were the hook doing its job. The tool reports what was refused, not whether refusing was right.
2. The four layers are only as separable as their refusal strings. They are matched by prefix; a future wording change makes a layer go quiet rather than wrong, which is the failure mode we most want you to know about.
3. The time-of-day pattern is eight samples of one intent on one machine. It is something to measure, not a fact about the platform. We would rather be disproved than repeat it.
4. Counting by grep overcounts. In an earlier scan of this same log today, the sentence "denied by the Claude Code auto mode classifier" appeared 242 times while 52 were actual refusals — 4.7x. The rest is the agent quoting the refusal in its own notes. Only a tool_result carrying is_error is counted.
5. 38 times a different call was refused within two minutes of a refusal (same session, different shape). That does not prove one caused the other. It is the number to look at before you go hunting for a bug in your own hook.
6. Zero, against three allow rules, for the case where your own config says yes and the call was refused anyway. Not because it never happened to us — it happened five times over two days — but because the permission we were relying on lived in a design document we wrote for ourselves rather than in settings.json. A permission that exists only in prose does not run, and the instrument can only see config.

The question

If you run an agent unattended: what are your four counts, and does your ranking match either of our two columns? We are looking for one thing in particular — a second observation of the same command being decided differently at different times of day, or a clean disproof of ours.

python permission_gap.py scan
python permission_gap.py scan --json          # for a weekly tally

The tool is in this repository under permission-gap/. It reads the transcripts Claude Code already writes under ~/.claude/projects: standard library only, reads only, writes nothing, and anything token-shaped is masked before printing. Read the output before you paste it anywhere — the shapes are safe by construction, your commands are your own.

One machine's log is not a number.