Code, data, and manuscript source for:
Mingguang Chen, Bo Qu, Licheng Wang. "The Calibration Floor: Format Repair Can Masquerade as Self-Correction at Small-to-Mid Scale." 2026. arXiv:2608.04355
One-line summary. Reported gains from LLM self-revision are often driven by an answer becoming extractable, not by the model's reasoning getting better. We decompose the observed accuracy delta into a content margin (both answers parseable — the only part attributable to reasoning) and two format margins (parseability changes), test the decomposition causally with grammar-constrained re-extraction, and show format effects dominate at capable model scale (4B-12B, and provisionally a ~55B-active frontier check) while genuine content-level change is concentrated — and often harmful — at floor scale (0.8B/2B).
proposal_option3_calibration_floor_v2.md # frozen pre-registered design
ANALYSIS_PLAN_OPT3.md # pre-registered analysis plan + amendment log
ENV.lock # exact environment/model revisions used
code/
cf_core.py # identity, floor criterion, offline gating, H2m gap, LOO
sim_trajectory.py # synthetic always-revise sandboxes
sim_validate.py # six instrument checks (no GPU / no LLM required)
make_validation_figures.py
pilot_trajectory.py # plumbing pilot (Qwen3.5-0.8B; not a hypothesis test)
run_stage2.py # Stage 2 batch runner (always-revise trajectories)
evaluate_offline.py # offline identity / floor / gating on frozen JSONL
evaluate_holdout.py # sealed dev/holdout protocol
check_extraction_gate.py # extraction/completion admission gate
decompose_margins.py # exact Delta_total = content + format_recover + format_loss
loo_analysis.py # leave-one-out floor prediction
floor_redo.py # nonparametric floor-pass criterion
gee_scale_contrast.py # clustered GEE scale-contrast model
tier_contrast_robustness.py# checkpoint-level robustness checks for the scale contrast
make_atlas_figures.py # Figures 4-5 (margin atlas, squeeze plane)
constrained_probe.py, analyze_constrained.py # causal control: grammar-constrained re-extraction
cas_grader_sensitivity.py # CAS-based MATH re-grading
run_ioe_replication.py # verbatim literature-protocol replication
run_frontier_api.py # OpenRouter frontier-model API client
run_gemma_ollama.py, probe_extraction.py, reparse_probes.py, stats_tests.py
data/
stage2_*.jsonl # raw always-revise trajectories (29 primary cells + frontier arm)
holdout/, probe/, constrained/, archive_prefixprompt/ # sealed holdout, probe, causal-control, and prompt-regime trajectory sets
margin_decomposition.json, stats_tests.json, loo_floor_prediction.json,
floor_redo.json, tier_contrast_robustness.json # derived results reported in the paper
draft/
main.md # manuscript source (14 sections + 4 appendices)
references.bib
pdf-header.tex
figures/ # Figures 1-5 as generated for the paper
No GPU or LLM required — these validate the identity/floor apparatus on synthetic sandboxes with known ground truth.
cd code
uv run --with numpy --with scipy python sim_validate.py
uv run --with matplotlib --with numpy --with scipy python make_validation_figures.pyRuns directly against the trajectory files in data/.
cd code
uv run --with numpy --with scipy python check_extraction_gate.py # 32 PASS, 6 FAIL (expected)
uv run --with numpy --with scipy python decompose_margins.py # -> ../data/margin_decomposition.json, §9.1/9.6
uv run --with numpy --with scipy python loo_analysis.py # -> ../data/loo_floor_prediction.json, §9.4
uv run --with numpy --with scipy python floor_redo.py # -> ../data/floor_redo.json, §9.3
uv run --with numpy --with scipy --with statsmodels python gee_scale_contrast.py # §9.2The data/*.json files in this repo are already the outputs of these
commands — rerun them to verify they reproduce byte-for-byte (up to floating
point / bootstrap-seed determinism), or inspect the JSON directly to audit
the reported numbers without rerunning anything.
cd draft
pandoc main.md --citeproc --bibliography=references.bib \
-s --embed-resources --standalone -o draft_v1.html
pandoc main.md --citeproc --bibliography=references.bib \
--pdf-engine=tectonic -H pdf-header.tex \
-V geometry:margin=1in -V fontsize=11pt -V colorlinks=true \
-o draft_v1.pdfData collection for the paper is finished; this is only for extending the study to a new model/task combination.
cd code
PYTHONUNBUFFERED=1 uv run --with mlx-lm --with mlx --with numpy --with scipy --with datasets \
python run_stage2.py \
--model mlx-community/Qwen3.5-4B-4bit \
--task gsm8k --n 400 \
--out ../data/stage2_qwen35_4b_gsm8k_new.jsonl --resume
# --task: gsm8k | math | triviaqa | arc | mmlu | commonsenseqa | truthfulqa_mc1 | humaneval | mbpp | pilot
uv run --with numpy --with scipy python evaluate_offline.py \
--traj ../data/stage2_qwen35_4b_gsm8k_new.jsonlENV.lock records the exact model revisions, quantization, and frozen study
constants (seeds, bootstrap replicates, gate thresholds) used to produce the
data in this repository; exact package versions for the ephemeral uv run
environments were not preserved (a stated reproducibility limitation), which
is precisely why the raw trajectories are included rather than only the
derived JSON — they make every reported statistic recomputable without
needing to re-run model generation.
- Code (
code/*.py) — MIT, seeLICENSE. - Manuscript, analysis plan, proposal, figures, and data
(
draft/main.md,draft/figures/,ANALYSIS_PLAN_OPT3.md,proposal_option3_calibration_floor_v2.md,data/) — CC BY 4.0, seeLICENSE-CC-BY.
@article{chen2026calibrationfloor,
title = {The Calibration Floor: Format Repair Can Masquerade as Self-Correction at Small-to-Mid Scale},
author = {Chen, Mingguang and Qu, Bo and Wang, Licheng},
year = {2026},
journal = {arXiv preprint arXiv:2608.04355},
url = {https://arxiv.org/abs/2608.04355}
}