Repository navigation
Fixes v1.1's length shortcut. On a frozen 200-row holdout, hard-vs-easy AUC rises from 0.8396 to 0.9794 (paired
bootstrap 95% lower bound of the gain +0.0906). Matched pairs ordered correctly go from 13/20 to 19/20. val2 AUC moves
from 0.8952 to 0.8966 and fresh100 AUC from 0.9908 to 0.9956.
Added
-
head_tailandencodeinswitchyard_decider/serve.py, with unit tests: inputs over 768 tokens keep the first 384
and last 384 tokens. -
benchmarks/holdout/: the frozen 200-row holdout (sha256967dc16e…c61357) and the deterministic generator that
reproduces it. -
training/retune/: augmentation builder, v1.2 fine-tune script, probe refit, holdout QA and leakage checks,
acceptance scoring with paired bootstrap, and an end-to-end latency harness. -
TRUNCoption intraining/train3.py(htby default,headreproduces v1.0 and v1.1). -
benchmarks/bench_llm.py: compares the served model with zero-shot OpenRouter chat LLMs and OpenRouter decision
models on the same validation sample. Reports AUC with bootstrap CI, routing accuracy, latency, tokens and cost
per decision, and has a spend cap. Results are inbenchmarks/results/openrouter_compare.json, the chart is
docs/pareto.png, and the write-up is in docs/benchmarks.md. These numbers were measured on v1.1.
Changed
- Weights: v1.2 is fine-tuned from v1.1 for 1 epoch at lr 1e-5 on 20k replay rows plus 4,653 label-preserving
augmentation rows, with the head+tail cut. The ridge probe was refit. Training took about 45 minutes on one RTX
5070 Ti. - The server reads up to 20,000 characters of
opening_task(was 3,000) and applies the head+tail cut instead of
keeping the first 768 tokens.predict.pyandtraining/embed.pyuse the sameencode. With v1.1 weights, scores
for inputs over 768 tokens change. v1.2 weights need this cut. - Docs and model card report v1.2 results. Latency is now measured end to end on long holdout texts (62.8 ms p50 for
v1.2, 64.0 ms for v1.1), so it is not comparable with the earlier short-prompt figure of about 25 ms.