caps-emphasis: capitalised emphasis in prompts, measured on Baguettotron

[oaustegard]

Closes #45.

Capitalising a directive does not make Baguettotron obey it. Mid-sentence the effect is +0.003 log-odds, CI [−0.147, +0.168], t = 0.04. Measured as actual violations, every surface form matches the no-directive baseline at 42/43.

Answers to the five questions

Q1 — compliance. No effect. none, sentence case, NEVER, token-matched lowercase and ALL CAPS THROUGHOUT all violate 42/43; bold 41/43. The probability measure agrees more sharply: every directive raises the forbidden word by ~2.5 log-odds against no directive at all, reproducing the ironic rebound that Mann et al. and Rana report on models 7–20× larger.

Q2 — case or token count? Token count, to the extent it is anything. Capitalising a keyword costs a variable number of extra tokens in this tokenizer (do not → DO NOT is free; never → NE|VER costs one; under no circumstances costs four). Pooled by that cost, with several keywords per bin:

CAPS costs	keywords	effect	95% CI
+0 tokens	3	−0.100	[−0.229, +0.035]
+1 token	15	−0.147	[−0.203, −0.090]
+3 tokens	3	+0.104	[−0.030, +0.243]
+4 tokens	1	+0.299	[−0.000, +0.574]

When capitalising is free in tokens, the interval spans zero.

Q3 — dose-response. An inverted U, with the forbidden word's own case frozen so the manipulation stays a prohibition rather than becoming priming. Least rebound at 25–50% capitalised (−1.98); 100% (−2.26) is worse than 0% (−2.14). Twelve random word-subsets per level; between-draw spread 0.07–0.10 against between-level spread ~0.28.

Q4 — attention. Total mass onto the span scales with span token count, which a softmax over more keys guarantees and which is not evidence. Per token, bold and caps are near-identical (0.308 vs 0.294). Replaced with a per-layer causal knockout.

Q5 — base rate. 22,100 documents / 10.5M words of SYNTH. All-caps runs at 8.10 per 1,000 words but 96% are acronyms; caps-as-emphasis is 0.34 per 1,000. IMPORTANT 0 occurrences against 3,147 lowercase, REQUIRED 0 against 4,459, NEVER 5 against 957. Exactly one genuine capitalised instruction in the sample: NEVER give a baby honey. Markdown bold runs at 7.78/1,000 in 52% of documents, outnumbering emphatic caps 23:1.

The bold arm, which is where the interesting part went

Bold first appeared to suppress ~20× harder than caps, matching the 23:1 corpus ratio. That was wrong twice.

Most of the raw effect was entropy — bold raises next-token entropy more than any other marker (1.927 vs 1.451 for sentence case), and raw log P cannot separate suppressing one word from flattening everything. Normalised against a control token, bold's advantage is +0.277, not 20×.

What remains is register. 93% of SYNTH's bold spans sit in reasoning traces; user turns hold one bold span per 763,630 words. Moving the same bolded directive into the reasoning register reverses its sign:

	sentence case	bold	bold's effect
user turn	−2.557	−2.280	+0.277
reasoning register	−1.754	−1.993	−0.240

Bold helps where the corpus never puts it and hurts where the corpus always puts it — an off-distribution marker, not emphasis.

That comparison also surfaced the largest effect anywhere in the experiment, and it is not typographic: restating the constraint inside the reasoning register rather than the user turn cuts the rebound by ~0.8 log-odds, roughly 3× any surface-formatting effect measured.

Controls that changed conclusions

Added after an adversarial design review, each of which would otherwise have shipped a wrong number:

• Control-token normalisation — without it the bold headline was an entropy artifact.
• Frozen forbidden-word case in the dose sweep — at 100%, PARIS gets capitalised too, which would have produced the predicted peak-then-decline curve through priming alone.
• Sentence-case baseline rather than all-lowercase, so the contrast is not CAPS versus ungrammatical.
• Baseline-pressure strata — the case effect flips sign across them (+0.075 at ceiling, −0.193 at low), which an unstratified pool averages away.
• 26 keywords across 5 token-cost bins, so the cost is not a relabelling of which word it is.

Scope

A 321M model trained on synthetic reasoning traces is not evidence about frontier models and the effect sizes should not be carried anywhere. On Q1 and Q4 this is a replication under stricter controls rather than a discovery — Dillitzer et al. cover that ground across 13 models, and their finding that reasoning models show near-zero case sensitivity predicts this null. The dose-response, the within-tokenizer token-cost dissociation, the register contrast and the corpus base rate are where four searches found no prior work.

The teacher-forced frame forces the assistant turn past an empty <think> block, which PleIAs' model card says degrades performance. The generation arms are the check and they agree, but some of the null may belong to the frame rather than the model.

Contents

RESULTS.md, METHOD.md, PRIOR_ART.md, Q5_CORPUS.md, SYNTH_NOTES.md, ERRORS.md, recheck.py, plus the harness (caps_lib.py, render.py, conditions.py) and per-arm runners with their JSON artifacts.

ERRORS.md records two mistakes caught mid-run: an item screen that compared a whole word against a single token, and a knockout arm that silently produced NaN for its baseline because render() returned the uncased keyword. The second was cheap to catch only because span_range returns None rather than a (0, 0) a caller would happily index.

---

Generated by Claude Code