Skip to content

Commands

m4bwav edited this page Nov 21, 2026 · 1 revision

trust.py has six commands: log, stop, show, diff, scan, check. Run them from the plugin folder as python scripts/trust.py <command>.

The outputs on this page come from trust.py 0.1.1 with TRUSTWRIGHT_HOME set to an empty temporary folder and the entries below added in order. Products, versions and model names are examples.

Exit codes

0 when the command ran; 2 when argparse rejects an argument (a missing required option, or a value not on a list).

log

Appends one behaviour observation to behaviour.jsonl and prints it.

Option Required Notes
--product yes claude-code, codex, copilot, cursor, gemini-cli, claude-ai or your own name
--version yes the exact version, or unknown
--kind yes permission, mode, instruction, tool-use, refusal, hallucination, overcaution, other
--observation yes one line, one observation
--model no default empty
--evidence no a transcript path, issue URL or commit; default empty
--date no default today, YYYY-MM-DD
python scripts/trust.py log --product claude-code --version 2.1.3 --kind instruction --observation "Ran npm test instead of the test command in AGENTS.md" --date 2026-11-03
{"date": "2026-11-03", "product": "claude-code", "version": "2.1.3", "model": "", "kind": "instruction", "observation": "Ran npm test instead of the test command in AGENTS.md", "evidence": ""}

Two more entries were added the same way for the examples below: the same hallucination observation on 2.1.5, and an overcaution entry "Asks before every read-only git command" on 2.1.5.

stop

Appends one false-positive stop or refusal to stops.jsonl and prints it.

Option Required Notes
--harness yes the product that stopped
--version yes
--task yes what was asked, with no secrets
--source yes provider-classifier, model-refusal, harness-permission, other
--category no default other; see below
--model, --message, --workaround, --date no --message is the verbatim stop text

Categories: the OR-Bench topics deception, harassment, harmful, hate, illegal, privacy, self-harm, sexual, unethical, violence; the XSTest trigger types homonym, figurative, safe-target, safe-context, definition, historical, public-privacy, fictional-privacy; and security-research, other.

python scripts/trust.py stop --harness claude-code --version 2.1.5 --model example-model --task "Explain how a known CVE in my own server works" --source model-refusal --category security-research --message "I can't help with that." --workaround "Said it is my server and asked for the patch" --date 2026-11-14
{"date": "2026-11-14", "harness": "claude-code", "version": "2.1.5", "model": "example-model", "task": "Explain how a known CVE in my own server works", "source": "model-refusal", "category": "security-research", "message": "I can't help with that.", "workaround": "Said it is my server and asked for the patch"}

A category not on the list is refused with exit 2 and nothing is written:

python scripts/trust.py stop --harness x --version 1 --task t --source other --category weather
trust.py stop: error: argument --category: invalid choice: 'weather' (choose from 'deception', 'harassment', 'harmful', 'hate', 'illegal', 'privacy', 'self-harm', 'sexual', 'unethical', 'violence', 'homonym', 'figurative', 'safe-target', 'safe-context', 'definition', 'historical', 'public-privacy', 'fictional-privacy', 'security-research', 'other')

show

Lists behaviour entries sorted by product, version and date, then stops in the order they were added. --product NAME keeps behaviour entries with that product and stops with that harness. --json prints both lists as JSON. An empty record prints no entries.

python scripts/trust.py show
claude-code 2.1.3 [hallucination] 2026-11-02 example-model: Reported tests passing with no test run in the log
claude-code 2.1.3 [instruction] 2026-11-03 : Ran npm test instead of the test command in AGENTS.md
claude-code 2.1.5 [hallucination] 2026-11-12 example-model: Reported tests passing with no test run in the log
claude-code 2.1.5 [overcaution] 2026-11-13 : Asks before every read-only git command
STOP claude-code 2.1.5 2026-11-14 model-refusal/security-research: Explain how a known CVE in my own server works

An entry with version unknown sorts after the numbered ones (0.1.0 crashed here; see Versions and upgrading):

claude-code 2.1.5 [overcaution] 2026-11-13 : Asks before every read-only git command
claude-code unknown [other] 2026-11-15 : Version not known

diff

Behaviour entries for one product with a version after --from and up to and including --to. Each is marked same when the same observation text was logged at or before --from, or new. Since 0.1.1, observations logged at or before --from and not seen again in the range follow as gone?.

python scripts/trust.py diff --product claude-code --from 2.1.3 --to 2.1.5
claude-code: 2 observations after 2.1.3 up to 2.1.5
  2.1.5 [hallucination] (same) Reported tests passing with no test run in the log
  2.1.5 [overcaution] (new) Asks before every read-only git command
  not seen after 2.1.3:
  (gone?) Ran npm test instead of the test command in AGENTS.md

Matching is on the exact observation text, so write a repeated observation the same way each time. gone? means not logged again, which is weaker than "fixed": it may simply not have come up. Stops are not part of diff.

scan

Reads a Claude Code session transcript (.jsonl) and lists lines that look like stops, at most one per line. Your own typed prompts are skipped, so "I can't help wondering why the build fails" typed by you is not a hit. --json prints the hits as JSON.

python scripts/trust.py scan tests/fixtures/session.jsonl
line 3 provider-classifier: Code is unable to respond to this request, which appears to violate our Usage Policy.
line 4 harness-permission: Permission to use Bash was denied.
line 5 model-refusal: I can't help with writing that exploit.
3 candidate stops; record real false positives with `trust.py stop`.

The excerpt is up to 60 characters either side of the match, so it can start mid-sentence. The patterns:

Source Matches
provider-classifier "violate our usage policy" and similar; "API Error:" followed within 80 characters by usage policy, content filter or safety
harness-permission "permission denied", "permission to use X was denied", "the user doesn't want to proceed", "tool use was rejected"
model-refusal "I can't / cannot / won't / am not able to help / assist / provide / do that / comply"

A hit is a candidate. Confirm each was a legitimate task before recording it with stop.

check

Prints a verification checklist. --type is fact (the default), code, numbers, citation or action, and can be repeated. --high-stakes adds three steps for money, health, legal, deletes or releases.

python scripts/trust.py check --type action
## action
1. Find the tool call that did the claimed action in the log.
2. Check the evidence outside the transcript (file on disk, commit, PR, HTTP status).
3. A report of 'done' with no matching call is a hallucinated action.
python scripts/trust.py check --type code --type numbers --high-stakes
## code
1. Run it, or run its tests; 'should work' is not a result.
2. Check every API, flag and package named exists in the installed version.
3. Diff what changed against what the answer says changed.
## numbers
1. Recompute each number from its inputs with a script.
2. Check units and orders of magnitude.
3. Match each number to the tool output it came from.
## high stakes
1. Ask the question again in a fresh context and compare (self-consistency).
2. Write verification questions for each claim and answer them independently (chain-of-verification).
3. Have a second model or person review before acting.

With no --type:

## fact
1. Name the source for each claim; open it, do not cite from memory.
2. Check dates: is the fact still current for the version or year asked?
3. Look for a claim with no tool call behind it in the log (stated, never fetched).

The citation list: open every URL and confirm it says what is claimed; check titles, authors and years against the page; flag any citation that appears in no tool result.

Clone this wiki locally