VERA-MH v2.0.4

VERA-MH 2.0.4 adds vera pool and ends vera pipeline with the pooled score, separates scoring from judging (judge/ and score/ no longer import each other, enforced in CI), and fixes judging with max_concurrent=0. vera commands, their flags, and their outputs are otherwise unchanged; code that imports judge.score_utils, judge.constants, utils.model_config_loader, or generate.main needs updating.

This is the last release that ships the legacy scripts. 2.1.0 removes legacy/; move to the vera commands before upgrading (docs/legacy-scripts.md maps each script's flags).

Breaking / migration

• Last release with the legacy scripts—2.0.4 is the last release that ships legacy/ (legacy.generate, legacy.judge, legacy.score, legacy.run_pipeline, legacy.pool_vera_scores, run_recommended_vera_pipeline.sh). 2.1.0 removes them; move to the vera commands before upgrading. docs/legacy-scripts.md maps each script's flags to vera.
• utils.model_config_loader removed—load_model_config, get_model_for_prompt, and the root model_config.json (which listed retired Claude 3 and GPT-4 IDs) are gone; nothing in the repo used them. Model defaults live in llm_clients/config.py.
• Scoring helpers left judge/—judge/score_utils.py and judge/constants.py are gone. The rating labels (BEST_PRACTICE, NEUTRAL, DAMAGING, NOT_RELEVANT) are in utils/ratings.py; parsing judge TSVs and rebuilding results.csv (plus DIMENSIONS, load_rubric_structure, load_persona_column, extract_persona_name_from_filename) is in utils/judge_results.py; score math, risk levels, ensure_results_csv, and extract_model_names_from_path are in score/utils.py. add_persona_columns_to_dataframe moved to judge/utils.py. Update imports; there are no re-export shims.
• Comparison chart moved—python -m judge.score_comparison is now python -m score.compare. judge/score_comparison_from_export_summary.py was deleted: nothing called it and its input script isn't in the repo.
• Pool script moved to legacy/—scripts/pool_vera_scores.py is now legacy/pool_vera_scores.py; run it as uv run python -m legacy.pool_vera_scores. Its flags and defaults are unchanged. Prefer vera pool.
• generate.main is now generate.run—The generation entry module follows the package naming convention. from generate import run_generation is unchanged; update direct imports and patch() targets that name generate.main.

Runtime, CLI, and pipeline

• Comparison chart text—python -m score.compare no longer says "suicide risk" (the subtitle reads "Scores indicate how well models detect and respond to risk", the header bar "Safety measures"), and long dimension names in its column headers wrap onto two lines instead of running into each other. Numbers, colors, and CSV columns are unchanged.
• Chart failures are errors—If a score chart can't be drawn, vera score and vera pool now fail instead of printing a warning and carrying on, so a broken chart can't go unnoticed. The optional risk-level chart still only warns.
• vera pipeline ends with the pooled score—With more than one user model, the pipeline now pools the per-model evaluations and prints Headline (pooled) score: <folder>, using the scoring section's personas settings and writing under generation.output. It used to print the folders and leave pooling to you.
• Pooling checks rubrics—vera pool and the pipeline refuse to pool evaluations whose dimension columns differ, since scores from different rubrics aren't comparable.
• vera pool—Merges two or more judge evaluations into one scored result (vera pool --evaluations <a> <b>), replacing the pool script. Pooling now lives in score/pool.py (from score import run_pooling). Like vera score, it skips the risk breakdown unless --personas is given.

Testing and CI

• Cheap-tier models on Config—The live end-to-end test reads its models from Config.CHEAP_*, and test_model_availability.py now checks those IDs too, so a retirement fails the availability check instead of partway through a run.
• Score charts checked—The vera score tests now check that scores_visualization.png and scores_by_risk_visualization.png are written. The chart code itself was consolidated (one stacked-bar function instead of four copies) and renders byte-identical images.
• Import contracts—CI runs lint-imports and fails if judge/ and score/ import each other, or if score/ imports llm_clients/ ([tool.importlinter] in pyproject.toml). utils/__init__.py no longer imports its submodules, so import score loads neither the judge nor the LLM stack; a unit test checks this in a fresh interpreter.
• Pyright gate—CI now fails on type errors in generate/, score/, utils/, vera_cli/, and vera.py, which have none today. The full-repo pyright run still reports without failing.

Documentation and development workflow

• Current model IDs in examples—README, AGENTS.md, and docs/ no longer use gpt-4o or claude-sonnet-4-5-20250929, and the -u gpt:1 / -c sonnet short forms (which never resolved; there is no alias table) are full IDs.
• Create your own evaluation—docs/create-an-evaluation.md is a short walkthrough for writing a new target: how to change the personas and the rubric, then run and validate it. Linked from the README.
• How to read a rubric—docs/reading-a-rubric.md explains, for readers rather than rubric authors, the structure every VERA-MH rubric shares (in-scope check, five dimensions, the risk level that picks which Guides to Human Care items apply), how answers become ratings, and the rubric file's columns, with the matching SI and HFO question IDs. Linked from docs/targets.md and docs/judge.md.
• Wider maintainer review—.github/CODEOWNERS now also covers score/, vera_cli/, vera.py, utils/, scripts/, .github/, and docs/roadmap.md, and AGENTS.md asks agents not to change public signatures or unrelated files unless the task calls for it.

Fixed

• Legacy live tests run again—tests/integration/test_scoring.py had been unable to start any legacy script since the scripts moved to legacy/: its commands ran uv run python3 python -m legacy.… (one interpreter too many), two tests imported run_pipeline instead of legacy.run_pipeline, and their fake sys.argv passed -m legacy.run_pipeline to the script's own argument parser. The file only runs in the manual full live suite, so nothing caught it.
• Judging with max_concurrent=0—Judging now treats max_concurrent=0 as no limit, as generation does. It used to start no workers and judge nothing (#261).
• SI rubric renders on GitHub—Every row of data/SI/rubric.tsv now has the header's 11 columns (most rows lacked the trailing Auto Answer cell), line endings are LF throughout, and cells containing double quotes use standard CSV quoting, which GitHub's strict parser requires. No cell content changed; the judge parses the file identically.
• Recommended pipeline pooling—legacy/run_recommended_vera_pipeline.sh runs its pooling step again: it looked for the pool script next to itself after moving to legacy/, where the script was not yet. It now runs python -m legacy.pool_vera_scores.

Documentation and development workflow

• Shared command skeleton—The vera command modules now share run_command, resolve_with, add_common_flags, and config_section from vera_cli/config.py instead of repeating them, and the pass-through _run_config helpers are gone. No change to flags, help text, or output.