VERA-MH v1.1.1

VERA-MH v1.1.1

VERA-MH v1.1.1 updates the recommended judge to GPT 5.4, supports latest models like Claude Opus 4.8, and adds tooling to turn judge output into more actionable improvement reports. The core scoring formula is unchanged from v1.1.

While this release includes incremental judge improvements, incremental work is underway to further strengthen alignment with human clinician evaluations.

Highlights

• GPT 5.4 is now the recommended judge model (replacing the v1.1 dual-judge setup of GPT-4o + Claude Sonnet 4.5). Internal inter-rater reliability (IRR) research found GPT 5.4 most aligned with human clinician ratings on a set of 50 newly simulated conversations.
• Improvement reports — new scripts/summarize_results.py turns results.csv into a Markdown breakdown of where an AI provider failed rubric standards, grouped by dimension and question.
• More stable Gemini judging via json_schema structured outputs, with better failure diagnostics in judge logs.
• Improved logging for judging where judge logs now contain metadata such as token usage.

v1.1.1 Scores

Recommended evaluation profile

Setting	v1.1.1 value
Personas	All 100 rows in data/personas.tsv
User agents	GPT 5.2 + Claude Opus 4.5 (100 conversations each)
Turns	30 per conversation
Judge	GPT 5.4 (gpt-5.4)

Run the full automated flow:

./scripts/run_recommended_vera_pipeline.sh <provider-agent>

Override the judge with VERA_JUDGE if needed (default: gpt-5.4). See scripts/run_recommended_vera_pipeline.sh for other environment variables.

What's new

Loading judge results

• Fix: read_results_csv() preserves persona risk level None when pooling or rescoring (pandas no longer coerces the literal string "None" to NA).

Improvement reports

After judging, summarize where standards were not met:

uv run python3 scripts/summarize_results.py \
  --results output/{YOUR_P_RUN}/evaluations/{YOUR_J_RUN}/results.csv \
  --rubric data/rubric.tsv \
  --out-stats output/{YOUR_J_RUN}/improvement_stats.json \
  --out-md output/{YOUR_J_RUN}/improvement_report.md

Runtime and reliability

• LLM params: unsupported model parameters are filtered before API calls. Claude Opus 4.8 rejects temperature; it is stripped via _unsupported_model_params.
• Gemini judges: LangSmith structured output now uses json_schema for improved stability.
• Judge logs: judge logs now include structured-output details, from tokens used to exceptions.

Automation

• scripts/run_recommended_vera_pipeline.sh — single GPT 5.4 judge (VERA_JUDGE); dual user-agent suites unchanged. Pooled output uses j_<judge>__p_... naming instead of j_pooled__....

Migration from v1.1

1. Judge model — If you used the v1.1 recommended dual-judge flow, switch to GPT 5.4 for new runs. Replace VERA_JUDGE_A / VERA_JUDGE_B with VERA_JUDGE.
2. Score labels — New score artifacts and comparison graphics are labeled v1.1.1; the underlying formula is the same as v1.1.
3. Pooling — Rescoring or pooling existing results.csv files benefits from the None risk-level read fix; no action required beyond upgrading.

Full changelog

See CHANGELOG.md for the complete list of changes.