feat: interscript-sadeed-eval — protocol re-scoring CLI - #177
Merged
Conversation
Third parties can now re-score any model's predictions on the public SadeedDiac-25 benchmark under the campaign's exact convention: windowed DER-CE via the vendored Misraj evaluator (public source, attribution preserved), paired bootstrap vs a reference predictions file, final_preds.jsonl or plain-text input. modal_distill's bootstrap now delegates to the shared pure home (sadeedbench.bootstrap). Validated end-to-end: scoring the G2b run's raw final_preds.jsonl against the benchmark parquet reproduces the published verdict exactly (4.8231 n=1200; delta 2.3717 [2.194, 2.555]).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The eval-harness CLI (the external-verifiability deliverable):
sadeedbench.scoring— windowed DER-CE via the vendored Misrajevaluator (public Sadeed source, attribution header preserved)
sadeedbench.bootstrap— the pure paired-bootstrap home;modal_distill delegates to it (one implementation)
interscript-sadeed-eval score --preds X --data bench.parquet [--key student] [--vs ref.jsonl] [--vs-key teacher]--vsadds the paired-bootstrap delta (candidate minus reference)Validation
Scoring G2b's raw final_preds.jsonl against the benchmark reproduces
the published verdict exactly: der_ce 4.8231 (n=1200), delta 2.3717
[2.194, 2.555], p=0 — the tool independently re-derives the headline
number from predictions + public data.
Tests
8 new (scoring ordering, None-pairwise bootstrap, determinism, CLI
formats incl. final_preds.jsonl + plain text + --vs); full suite 141
passed.