Judge criteria profiles (--criteria) + prompt provenance in metadata (#44)#47
Open
davanstrien wants to merge 4 commits into
Open
Judge criteria profiles (--criteria) + prompt provenance in metadata (#44)#47davanstrien wants to merge 4 commits into
davanstrien wants to merge 4 commits into
Conversation
…ata (#44) Replace the single hardcoded PAIRWISE_PROMPT with a CRITERIA_PROFILES dict. The "default" profile is the original prompt verbatim (byte-equal, pinned by a hash guard); a new "table-fidelity" profile keeps criteria 1-4 unchanged but swaps criterion 5's structure-neutrality for an explicit table criterion: tabular data must preserve row/column cell association, and a flattened table that breaks which value belongs to which row/column is a SIGNIFICANT error ranked just below completeness (markup style stays neutral — a well-aligned plain-text table is as good as markdown/HTML). - judge: CRITERIA_PROFILES + prompt_hash() helper; build_prompt and build_comparisons take a prompt_template param defaulting to the default profile (backward compatible for direct callers/tests). - cli: --criteria {default,table-fidelity} on the judge subcommand; threads the selected template through both build paths; logs the criteria + prompt hash at judge start. - publish: EvalMetadata gains criteria + prompt_hash, written into the metadata row and surfaced in the README details; _align_metadata_rows backfills the new columns as None for older rows. Tests: profile dict default is byte-equal to the old prompt; --criteria parses + rejects unknown; template threads to build_comparisons; metadata carries criteria + prompt_hash; old metadata rows align. 348 pass, ruff + ty clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codex review found one blocker + one prompt issue on the criteria-profiles work. 1. [BLOCKER] Criteria-mixing on incremental/refit runs. Judging a results repo under a different --criteria than its existing comparisons were scored with merged old- and new-rubric verdicts into one ELO board and republished the metadata mislabeled with only the current run's criteria (worst on the refit-only path: 100% old-criteria results published under the new label). Added a provenance guard in cmd_judge, applied wherever existing results are loaded (incremental AND refit): reads the last metadata row's criteria/prompt_hash (missing/None pre-#44 rows treated as the default profile — historically accurate, since default is byte-identical to the old prompt). On any mismatch it prints a clear error and exits without judging or publishing, pointing the user at --criteria <prev> or --full-rejudge. No warn-and-continue — mixed boards are impossible. 2. Table-fidelity prompt tensions. (a) The list said "in priority order" but table fidelity sat at position 5 with override language claiming just-below-completeness severity. Restructured so table fidelity IS criterion 3 (after completeness, before accuracy); accuracy→4, reading order→5; the markup-neutrality + bbox-tag-ignore text moved to a closing note. Position and severity now agree, no override wording. The DEFAULT profile is untouched (still byte-identical). (b) Fixed the tie line so it can't suppress ties on non-table pages ("...including, where tables are present, the same table structure..."). Tests: provenance-helper unit tests; guard integration tests (mismatch exits + nothing published, pre-#44 None rows treated as default so a default run proceeds, --full-rejudge bypasses, matching criteria proceeds); updated prompt-structure tests + new pinned table-fidelity hash. 395 pass, ruff + ty clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lets a user supply their own pairwise judge prompt from a file, alongside the
built-in --criteria profiles.
- --criteria-file PATH on the judge subcommand, in an argparse mutually
exclusive group with --criteria (which keeps its "default" when neither is
given). Forwarded through the bench subcommand the same way.
- The file is loaded and validated before the run: validate_prompt_template()
test-formats it with dummy strings and requires both {ocr_text_a} and
{ocr_text_b} and no other format fields — an unknown {field} or a stray
unescaped brace raises a clear error (suggesting {{ }} for literals) and the
run exits without judging or publishing.
- Recorded in metadata as criteria="custom:<filename>", with prompt_hash
computed from the file's template exactly as for a built-in profile.
- The criteria-mixing guard now compares on the prompt HASH (the true rubric
identity): the same custom file re-run matches, a different file (even same
basename) blocks, and a built-in prompt edited across versions is caught too.
Its error message handles the custom case — it tells the user to re-supply
the same file rather than pointing at a nonexistent --criteria name.
Tests: validate_prompt_template (valid/escaped/missing-placeholder/unknown-
field/stray-brace + built-ins pass); _resolve_criteria (built-ins, custom file,
invalid + missing file → DatasetError); parser mutual exclusion + defaults;
guard matches same-file rerun and blocks different content; bench forwards
--criteria-file. 415 pass, ruff + ty clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
|
Added per Daniel: Anyone benchmarking their own collection can now encode their own quality values without forking — per-collection leaderboards, per-collection criteria. |
….0 CI fix)
CI (Python 3.13.0) failed test_criteria_and_criteria_file_mutually_exclusive:
argparse mutually-exclusive groups with a defaulted `choices` argument in a
subparser don't detect the conflict on 3.13.0 (a real regression, fixed in
3.13.1) — so passing both flags parsed silently instead of erroring.
Drop the argparse mutex groups (judge + bench) and enforce exclusivity
explicitly in _resolve_criteria, so the conflict errors identically on every
Python version:
- --criteria default is now the None sentinel (was "default"), making
"explicitly passed" distinguishable from "unset"; _resolve_criteria maps
None → DEFAULT_CRITERIA.
- --criteria + --criteria-file together → DatasetError ("mutually exclusive").
- cmd_bench validates criteria up front (via _resolve_criteria) BEFORE launching
OCR jobs, so a conflict or a bad --criteria-file fails fast rather than after a
paid run; it forwards whichever flag was set (neither → judge's own default).
Tests updated for the sentinel default and the new error path; the conflict now
asserts DatasetError (judge) / cmd_bench raising before any phase runs (bench).
Gates green on BOTH Python 3.11 and 3.13 (incl. 3.13.0): 416 pass, ruff + ty
clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
|
CI failure root-caused and fixed: a CPython 3.13.0-specific argparse regression (mutually-exclusive group with a defaulted |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #44 (delegated implementation → Codex adversarial review → fixes applied).
Criteria profiles
CRITERIA_PROFILES = {default, table-fidelity}replaces the single hardcoded prompt. default is byte-identical to the previous prompt (pinned by a hash test —8d86832723b5— so provenance is continuous with every existing board).table-fidelitymakes preserving table row/column cell association criterion 3 (a flattened table that breaks cell relationships is a significant error); markup syntax stays neutral — a well-aligned plain-text table counts.--criteriaonjudge, forwarded bybench.Provenance
EvalMetadatagainscriteria+prompt_hash(sha256-12 of the template), written to metadata rows and surfaced on the card. Two boards judged under different rubrics are now distinguishable — which the MOH table-fidelity companion run needs.Review blocker fixed: criteria-mixing guard
Codex found that changing
--criteriabetween incremental runs on the same results repo would silently merge old- and new-rubric verdicts under the newest label (worst on the refit-only path: 100% old verdicts republished as the new rubric). Now: any criteria/prompt-hash mismatch with the repo's recorded provenance is a hard error before judging (pre-#44 rows count asdefault— historically accurate);--full-rejudgeis the sanctioned rubric-swap path. Mixed boards are impossible.Prompt fixes from the same review: table-fidelity's priority position now matches its severity (criterion 3, no override language), and the tie line can't suppress ties on non-table pages.
Known edge (fails safe, documented): a first-ever run under non-default criteria that dies after checkpointing but before any metadata row will be blocked on resume (guard sees no provenance → assumes default);
--full-rejudgerecovers. Clean fix = checkpoint-stamped provenance rows — noted on #43.Tests: 395 pass (+42 for this PR), ruff + ty clean.
Closes #44.
🤖 Generated with Claude Code