Skip to content

Recipes

m4bwav edited this page Nov 21, 2026 · 1 revision

Record a false-positive stop from today's session

  1. Note the harness version (claude --version) and the model.
  2. Decide the source: an API error naming the Usage Policy with no model text is provider-classifier; the model writing "I can't help with" is model-refusal; a blocked tool call is harness-permission.
  3. Record it with the verbatim message and what got it through:
python scripts/trust.py stop --harness claude-code --version 2.1.5 --model example-model --task "Explain how a known CVE in my own server works" --source model-refusal --category security-research --message "I can't help with that." --workaround "Said it is my server and asked for the patch" --date 2026-11-14

A workaround is the same request with its real context (purpose, ownership, a safe target). Never record or use a jailbreak.

Find stops in an old session

Claude Code keeps session transcripts as .jsonl files. Scan one:

python scripts/trust.py scan tests/fixtures/session.jsonl

Each hit is a candidate. Open the line, check the task was legitimate, then record it with stop.

Did it get worse in the new version?

Log the same kind of observation on each version as it happens, with the same wording when it is the same behaviour. Then:

python scripts/trust.py diff --product claude-code --from 2.1.3 --to 2.1.5

new lines appeared after the old version; same lines were there before too; gone? lines were not logged again. "It got worse" needs entries on both versions for the same kind of task. With entries on one side only, say so instead of calling it a change. A pattern needs at least two dated entries before you state it as one.

Did it really run the tests?

python scripts/trust.py check --type action

Then look for the test command and its output in the tool log, after the last edit. A "tests pass" with no matching call is a fabricated claim: log it.

python scripts/trust.py log --product claude-code --version 2.1.3 --model example-model --kind hallucination --observation "Reported tests passing with no test run in the log" --evidence "session 2026-11-02 transcript" --date 2026-11-02

Check an answer before acting on it

Combine the lists that fit, and add --high-stakes for money, health, legal, deletes or releases:

python scripts/trust.py check --type code --type numbers --high-stakes

Report each claim as verified (with the evidence), unverified or wrong, and never move an unverified claim to verified.

Read the record from another tool

show --json prints both files as one JSON object with behaviour and stops lists:

python scripts/trust.py show --product claude-code --json

Or read the JSONL files directly; Data format has the fields.

Clone this wiki locally