Your CLAUDE.md says "use snake_case." Delete that line and nothing changes — the
model already does it. Your CLAUDE.md also says "start every file with # RP-MARK-7731."
Delete that and adherence drops to zero. One of those rules earns its tokens. The
other is pure cost you pay on every single call. ruleprof tells you which is which.
It does this the only way you actually can: it removes a rule, re-runs the agent, and checks whether the agent still follows it.
| verdict | rule | tokens / $·run | adherence with → without | note |
|---------|-------------------------|---------------:|--------------------------|-------------------------------------------|
| CUT | Engineering philosophy | 165 / $0.0002 | 100% → 100% | agent obeys this with or without the rule |
| CUT | Naming | 24 / $0.0000 | 100% → 100% | agent obeys this with or without the rule |
| KEEP | File header convention | 40 / $0.0000 | 100% → 0% | removing it drops adherence — earns its tokens |
189 tokens = $0.0002/run the agent obeys anyway. At 1,000 runs/day that's $69/yr
for zero behavior change.
That 189 is paid on every request against this config, and it's small only because
the example config is small. On a real 4KB CLAUDE.md the dead weight is usually most
of the file — and the dollar line scales with it. Pass --input-price to price a
model ruleprof doesn't know, or let it infer from --model (Haiku $1, Sonnet $3,
Opus $5 per Mtok).
Static analyzers (ccmd.dev, AgentLint) price each line and flag vague wording. What they can't tell you is whether a rule does anything — that needs a run. The research backs the suspicion that much of it doesn't: Gloaguen et al. ("Evaluating AGENTS.md", arXiv 2602.11988, 2026) found context files don't generally improve task success while adding 20–23% to inference cost on average, with LLM-generated ones causing outright performance drops in 5 of 8 settings tested. ruleprof turns "probably bloated" into a per-rule KEEP/CUT you can act on.
The rule it applies: a rule earns its tokens only if removing it moves the agent off the behavior it prescribes. If the agent does the thing anyway, the rule is inert.
pip install ruleprof # needs the `claude` CLI on PATH and Python 3.11+
ruleprof init # scaffold ruleprof.toml from your CLAUDE.md
# fill in one check per rule, then:
ruleprof run --md report.md
A spec maps each rule to a check — a shell command, run in the agent's output directory, that exits 0 when the rule was followed:
task = "Create solution.py with a function that sums a list of numbers. Then stop."
model = "haiku"
trials = 3
[[check]]
block = "File header convention" # substring matched against the rule's heading
cmd = "test \"$(grep -m1 . solution.py)\" = '# RP-MARK-7731'" # exit 0 = followed
[[check]]
block = "Naming"
cmd = "grep -qE '^\\s*def [a-z][a-z0-9_]*\\(' solution.py"ruleprof splits your config into blocks at each ## heading, then for each block
with a check runs the agent trials times with the full config and trials times
with that block removed. The adherence gap is the verdict. --json report.json writes
the full data.
- Bind a check, get a verdict. Blocks with no check are listed with their token cost but marked unmeasured — ruleprof won't invent a signal it can't observe.
- Verdicts are for your model and your task. A rule that's dead weight for Haiku on a scaffolding task may matter for a weaker model or a trickier one. Profile the setup you actually run.
- Fuzzy rules give fuzzy answers. "Write clean code" has no behavioral signature to check, so it lands in the unmeasured pile. That's the honest result, not a gap.
- Verdicts carry a 90% confidence interval. A rule is only called KEEP or CUT when
the whole interval for its adherence delta lands on one side of the line. Run too few
trials and the interval is wide, so you get INCONCLUSIVE with a suggested trial count,
not a coin-flip verdict. A clean flip resolves in ~4 trials; a genuinely neutral rule
needs ~6 to confirm it's dead weight. Tune with
--confidence. - It costs real money. Each profile is
(1 + measured_blocks) × trials × 2agent runs. Trials buy resolution: 2 is enough to gauge cost on Haiku (the example ran $0.27 there) but usually too few to clear the interval — bump to 6 for verdicts.
examples/ holds a CLAUDE.md with three rules of known character — a non-default
marker, a redundant style rule, and a block of generic values prose — plus the spec
that profiles them. At 6 trials it returns KEEP for the marker and CUT for the other
two, with the full 90% intervals printed; the committed examples/report.md is that
run ($0.81). Drop to --trials 2 and the same rules come back INCONCLUSIVE — the
signal is there but the interval is too wide to call, which is the point.
ruleprof run --spec examples/ruleprof.toml --trials 6
MIT.