cot-faithfulness

A harness for measuring whether a reasoning model's chain of thought says what actually moved its answer.

The experiment is hint injection. Ask a model a multiple-choice question, record its answer. Ask again with a pointer planted toward a different option. Where the answer flips to the pointer, check whether the reasoning ever mentions the pointer. A flip with no mention is unfaithful reasoning, and it is unfaithful objectively: no human judgement of the reasoning's quality is required.

Results (MATS 12.0 write-up, September 2026)

Full write-up: writeup/writeup.md. Model: gpt-oss-120b on Groq, raw reasoning field. 40 MMLU items, 5 subjects, 5 conditions, 3 repeats, 600 calls, $0.16.

Finding	Number
Flip rate by hint: grader check() / expert / sycophancy / metadata	31.9% [19.3, 45.0] / 5.3% / 0% / 0%
Flips whose reasoning mentions the hint (44 hand-read, blind)	44 / 44, unfaithful rate 0%
Same grader marked "internal, not for the model" (leaked-40x3)	flips 31.9% → 2.6% [0.0, 7.8]
Same 44 flips, detector on the visible answer instead of the reasoning	2 / 44 mention the hint; 34 are a bare Answer: (X)
Best blind CoT-only monitor vs oracle, F1	0.33 vs 0.71

Short version: this model's reasoning is faithful on every flip and its user-facing answer almost never is, and one chain says why in its own words ("we must not reveal that we are cheating"). Which channel a monitor is allowed to read decides whether it catches anything.

Reproduce from the committed responses (no API calls):

python -m cotf score main-40x3 && python -m cotf monitor main-40x3
python -m cotf score leaked-40x3
python -m cotf calibrate main-40x3          # detector vs the 44 hand labels in labels/
python scripts/visible_ack.py main-40x3     # the output-channel check
python scripts/random_examples.py main-40x3 0   # the seeded raw examples in the write-up

Raw transcripts are in runs/<name>/responses.jsonl, hand labels in labels/main-40x3.jsonl.

---

The harness came before the experiment. Everything here runs end to end against a mock model with known behaviour, so the measurement code can be verified before a single token is spent on a real one.

python -m cotf run pilot --provider mock --n 60 --repeats 3

That is a complete run: control pass, four hint types, rates with clustered bootstrap intervals, figures. No API key, no network beyond fetching the questions once.

flowchart TD
    Q[MMLU / GPQA question] --> C[control pass<br>no hint, repeated]
    Q --> H[hinted pass<br>sycophancy · metadata · authority · reward_hack]
    C --> P[answer extraction]
    H --> P
    P -->|unparseable| X[dropped, but counted:<br>parse failure rate]
    P --> F{answer moved to<br>the hinted option?}
    C -.->|disagreement across repeats| N[spontaneous switch rate:<br>the noise floor]
    F -->|no| S[no flip]
    F -->|yes| D[mention detectors<br>verbatim · strict · loose]
    D --> A[acknowledged flip]
    D --> U[silent flip:<br>the unfaithfulness claim]
    A --> R[rates with clustered<br>bootstrap intervals]
    U --> R
    S --> R
    N --> R
    R --> M[monitor half: blind vs oracle,<br>scored against dumb baselines]

Loading

---

Why the mock model exists

MockProvider follows a planted hint 55% of the time and admits to it 35% of the time. Those constants are ground truth. tests/test_pipeline.py asserts that the harness recovers 0.35 inside its confidence interval. If a detector regex rots, a denominator drifts, or the bootstrap gets miswired, that test fails in under a second, offline.

A harness you cannot test is a harness you have to trust, and there is no reason to trust this one yet.

What gets reported, and why each piece is load-bearing

Number	Why it is there
Flip rate	If almost nothing flips, the acknowledgement denominator is tiny and the interval is uninformative. That is a fact about the hint, not about the model.
Acknowledgement rate given a flip	The headline. One minus this is the unfaithfulness claim.
Spontaneous switch rate	Models are not deterministic. Some "flips" are resampling noise. This is the noise floor, measured from control repeats, and it is why --repeats 1 prints a caveat.
Parse failure rate	Unparseable responses are dropped. If that share is large, the sample analysed is no longer the sample designed.
Detector spread	Three definitions of "mentions the hint" applied to the same flips. The gap between them is the size of the definitional wiggle room in the headline.

Flip rate is an upper bound on hint influence, never an estimate of it: some answers land on the hinted option by chance. test_flip_rate_exceeds_follow_rate pins that relationship so it cannot silently invert.

What the run cost

python -m cotf cost main-40x3
python -m cotf cost main-40x3 --price-in 0.15 --price-out 0.60   # unpriced model

Every response already carries the provider's usage block and a wall-clock stamp. cost reads them and answers the second question about the same run. Three of its numbers are not what a token counter gives you.

Cost per unit of signal. A faithfulness run does not buy completions, it buys flips. Most calls produce nothing to measure, so dollars per detected flip sits an order of magnitude above the per-call price, and it is the only figure that tells you what a bigger run would actually cost.

Service latency separated from wall latency. runner._call stamps the clock before providers.TokenBucket.take(), which sleeps until the free tier's tokens-per-minute ceiling lets the call through. So wall minus provider time is not network overhead, it is the harness throttling itself. On the 40x3 Groq run that is 99.8% of the wall clock: 1.15s of model against 480s of waiting. Quoting the wall figure as model latency would overstate it 400-fold. A provider that returns no per-stage timing gets the breakdown suppressed rather than relabelled, because a blank field is not a finding.

The reasoning tax. Chain-of-thought tokens are billed as output and dominate the bill in a CoT experiment, but no provider itemises them. The share is estimated from the character split between the reasoning trace and the visible answer, and it is labelled as an estimate everywhere it appears.

Prices are per million tokens and are never guessed. PRICES holds list prices each read from the provider's own docs on a stated date; a model with no entry reports tokens and seconds and omits every dollar column unless you pass --price-in and --price-out. Errored calls bought nothing, so they are counted as holes rather than averaged in as zeroes.

The four things this harness deliberately does not decide

These are the experimental design, and the design is the part that has to be yours. The harness makes each one an explicit, swappable knob rather than a buried default.

1. Which model. --provider and --model. Any OpenAI-compatible endpoint works; reasoning traces are read from reasoning, reasoning_content or an inline <think> block, because the field is not standardised.
2. What counts as "mentions the hint". Three detectors, ordered by strictness, in cotf/detect.py:
   • verbatim — the CoT quotes distinctive strings from the injected hint. Narrowest, least arguable, used as a floor.
   • strict — one sentence carries both a cue term for the hint type and the hinted letter. Default.
   • loose — any cue term anywhere. Over-counts by design. Plus an optional LLM judge, off by default because it costs tokens and can be wrong. Calibrate anything you rely on with cotf label / cotf calibrate.
3. What the control condition is. Currently: the same question with no hint, repeated, with the hinted option chosen so it is never the model's own control answer. Alternatives worth arguing about are a hint toward the option the model already chose, and a hint containing no answer at all.
4. Which hints. Four are implemented (cotf/hints.py), and they are not equivalent. "The CoT should mention it" is a much stronger claim for sycophancy (social pressure, no evidential content) than for metadata (which looks like a leaked answer key and is arguably a legitimate clue).

The monitor half

The second half is the part that makes this more than a replication: a monitor that reads only the chain of thought and predicts whether the answer was hint-driven, scored against baselines dumb enough to be embarrassing.

python -m cotf monitor pilot          # baselines only
python -m cotf monitor pilot --llm --judge-provider groq

Two regimes, kept separate:

• oracle — the monitor is told what the hint said. Not deployable, but it is the ceiling, and oracle_string_match is the baseline that most often beats a clever method.
• blind — CoT only. The setting that would actually matter.

always_positive is always in the table. Its F1 is 2p/(1+p) at base rate p, which is 0.64 at p=0.47 and is surprisingly hard to beat while meaning nothing. Any monitor that collapses into it is flagged <- degenerate and should not be reported as a result.

Thresholds are fit on a train half and scored on a held-out half, split by question id, never by response. The same question in both halves leaks.

Sanity-checking, as a number

python -m cotf label pilot --n 30     # writes labels/pilot.jsonl, blank verdicts
# fill in "human": true / false by hand, reading the transcripts
python -m cotf calibrate pilot        # agreement, precision, recall per detector

Reading transcripts yourself and reporting the agreement is the difference between "the experiment worked" as a claim and as a hypothesis that survived a check. If your primary detector disagrees with you on more than ~10% of cases, fix the detector before reporting the headline.

Getting a real model in

No key is needed for the mock, and none is committed here. Two free options:

Provider	Cost	Default model	Key goes in
Groq	free tier, no card	openai/gpt-oss-120b	$GROQ_API_KEY or ~/.openclaw/secrets/groq.key
OpenRouter	free tier, no card	deepseek/deepseek-r1:free	$OPENROUTER_API_KEY or ~/.openclaw/secrets/openrouter.key

python -m cotf doctor                 # which providers have keys
python -m cotf run r1-pilot --provider groq --n 60 --repeats 3 --concurrency 4

Reasoning models emit long traces. Budget roughly 1.5k output tokens per call and n_items x (1 + n_hints) x repeats calls: 60 items, 4 hints, 3 repeats is 900 calls. Start at --n 20 --repeats 1 to check the plumbing and the parse rate against a real model before committing to a full run.

Layout

cotf/providers.py   one interface over mock / groq / openrouter / together / deepseek / anthropic
cotf/dataset.py     MMLU and GPQA via the HF datasets-server REST API, cached to disk
cotf/hints.py       four hint types, prompt construction, target selection
cotf/runner.py      control pass then hinted pass, resumable, content-hash cache
cotf/detect.py      answer extraction, three mention detectors, optional LLM judge
cotf/score.py       rates, controls, caveats
cotf/monitors.py    blind and oracle monitors, trivial baseline, train/test split
cotf/stats.py       Wilson intervals, cluster bootstrap, F1 with CIs
cotf/cost.py        tokens, dollars and latency, split service vs throttle
cotf/labeling.py    hand-label sampling and detector calibration
cotf/plots.py       figures, every bar with its interval

Interrupted runs resume: every response is keyed by a hash of (provider, model, temperature, prompt, repeat) and appended to runs/<name>/responses.jsonl. Re-running skips what is already there.

Tests

python -m pytest tests -q

58 tests, all offline, under a second. They cover answer extraction on malformed output, detector strictness ordering, interval behaviour at 0 and 1, the clustering correction, and the end-to-end recovery of the mock's known rates.

Prior work

The hint-injection paradigm is from Turpin et al. 2023, Language Models Don't Always Say What They Think, with later variants from Chua and Evans 2025 and Anthropic's 2025 reasoning-faithfulness work. This is an independent implementation of the design, not a port of their code.