Skip to content

v0.3.0 — anti-leakage release heads

Choose a tag to compare

@alrobles alrobles released this 17 Oct 02:41
· 3 commits to main since this release

Sciev v0.3.0 — anti-leakage release heads.

noul and score release heads are retrained on the rho=0.25
evidence-corruption mixture: for each clean training row, 0.125
empty-evidence and 0.125 shuffled-evidence counterparts whose label is
forced to the rejection class. Choice is unchanged (fr_choice.pt) — no
anti-leakage choice head is released yet (same-question confounders are
covered in the manuscript as the alp25x experimental arm).

Why: the v0.2.x heads exploit option-side cues (empty-evidence
agreement 0.810 noul / 0.575 score). The alp25 heads keep evidence
controls at the rejection-class prior (0.667 noul, 0.336 score) while
scoring equal or better on evidence-grounded benchmarks:

  • canonical SciFact noul: 0.464 -> 0.702 (+24 pts vs the leaky head)
  • SciFact score proxy: 0.757 -> 0.757 (tie)
  • elite clean acc: noul 0.907 (fr 0.931, itself partly leaked),
    score 0.911 (fr 0.896)

Contents:

  • fr_choice.pt — release choice head, unchanged (sha256 as v0.2.x)
  • alp25_noul.pt / alp25_score.pt — anti-leakage release heads
    (+ *.training_config.json provenance)
  • evals_v030.tar.gz — evaluation JSONs for the anti-leakage study:
    alp25 heads on clean/empty/shuffled/SciFact arms, the rho ablations
    (antileak_v2 full-strength, alp06), the choice anti-leakage arms
    (alp25/alp25p/alp25cf/alp25x incl. teacher audits), and the paired
    grounded-GPQA evaluations
  • sciev-0.3.0-{py3-none-any.whl,tar.gz} — same bits as PyPI
  • sciev-v0.3.0-paper.pdf — current manuscript (in preparation for arXiv)
  • MODEL_CARD.md, LICENSE, release-manifest.json, SHA256SUMS

Retrieval finding folded into the release narrative: PubMed BM25
evidence does not rescue GPQA (0.304 main vs 0.695 oracle) — honest
evidence reading matters more than topical retrieval.

Caveats: alp25 heads are single-run trainings; the elite clean-eval
accuracies of the leaky baseline are themselves inflated by option-side
cues; control ECE is high by construction (the head confidently rejects
rows labeled by reference agreement).