Skip to content

VERA-MH v2.0.0

Choose a tag to compare

View all tags
@luca-belli luca-belli released this 01 Oct 23:32
· 56 commits to main since this release
v2.0.0
ead2767
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

VERA-MH 2.0 is released. It breaks the legacy scripts: they still run in 2.0.0, but 2.0.1 moves them to legacy/, which changes how they are invoked.

Breaking / migration

  • vera CLI—vera.py (generate, judge, score, pipeline) replaces generate.py, judge.py, run_pipeline.py, and judge/score.py. The old scripts are deprecated and still run unchanged in 2.0.0; docs/legacy-scripts.md maps their flags to vera.
  • vera score has no default personas file—Without --personas, the risk-level breakdown is skipped and scores_by_risk.json is not written. Pass --personas data/SI/personas.tsv for the legacy behavior.
  • vera judge needs --output for flat transcript folders—It no longer falls back to evaluations/ in the working directory. Generation runs still write to <run>/evaluations/.
  • Per-question answers moved—answers/conversations/ is now answers/by_question/.

Targets, rubric, and scoring

  • Multiple targets—A target bundles a rubric, personas, and prompts in data/<target>/manifest.json, selected with --target or a config's target. Ships with SI and the new HFO target.
  • Persona annotation—A manifest can list persona_annotation_columns, which judging copies into results.csv so scores can be broken down by persona trait.
  • Terminal END—A plain GOTO=END on a rubric's last question no longer marks the evaluation Not Relevant; ASSIGN_END on a non-Yes answer is rejected.

Runtime, CLI, and pipeline

  • Config files—Every vera command takes --config, VERA_RUN_CONFIG, or --print. The recommended SI profile is configs/recommended-SI.json.
  • vera pipeline—Runs generation, judging, and scoring in one invocation, passing each stage's output to the next.
  • --into—vera generate and vera judge can continue an existing run.
  • Model IDs in one place—Default models are read from llm_clients/config.py.

Testing, CI, and documentation

  • Safer tests—Tests cannot open network connections or see real API keys.
  • Model availability check—A live test confirms the configured model IDs still exist at each provider.
  • Docs—README rewritten for 2.0, with reference docs split under docs/.
Assets 2
Loading