Repository navigation
v0.3.0 — anti-leakage release heads
Sciev v0.3.0 — anti-leakage release heads.
noul and score release heads are retrained on the rho=0.25
evidence-corruption mixture: for each clean training row, 0.125
empty-evidence and 0.125 shuffled-evidence counterparts whose label is
forced to the rejection class. Choice is unchanged (fr_choice.pt) — no
anti-leakage choice head is released yet (same-question confounders are
covered in the manuscript as the alp25x experimental arm).
Why: the v0.2.x heads exploit option-side cues (empty-evidence
agreement 0.810 noul / 0.575 score). The alp25 heads keep evidence
controls at the rejection-class prior (0.667 noul, 0.336 score) while
scoring equal or better on evidence-grounded benchmarks:
- canonical SciFact noul: 0.464 -> 0.702 (+24 pts vs the leaky head)
- SciFact score proxy: 0.757 -> 0.757 (tie)
- elite clean acc: noul 0.907 (fr 0.931, itself partly leaked),
score 0.911 (fr 0.896)
Contents:
- fr_choice.pt — release choice head, unchanged (sha256 as v0.2.x)
- alp25_noul.pt / alp25_score.pt — anti-leakage release heads
(+ *.training_config.json provenance) - evals_v030.tar.gz — evaluation JSONs for the anti-leakage study:
alp25 heads on clean/empty/shuffled/SciFact arms, the rho ablations
(antileak_v2 full-strength, alp06), the choice anti-leakage arms
(alp25/alp25p/alp25cf/alp25x incl. teacher audits), and the paired
grounded-GPQA evaluations - sciev-0.3.0-{py3-none-any.whl,tar.gz} — same bits as PyPI
- sciev-v0.3.0-paper.pdf — current manuscript (in preparation for arXiv)
- MODEL_CARD.md, LICENSE, release-manifest.json, SHA256SUMS
Retrieval finding folded into the release narrative: PubMed BM25
evidence does not rescue GPQA (0.304 main vs 0.695 oracle) — honest
evidence reading matters more than topical retrieval.
Caveats: alp25 heads are single-run trainings; the elite clean-eval
accuracies of the leaky baseline are themselves inflated by option-side
cues; control ECE is high by construction (the head confidently rejects
rows labeled by reference agreement).