VERA-MH v1.2.0

VERA-MH v1.2.0

VERA-MH v1.2.0 introduces a clinician-informed update to the evaluation rubric and the recommended automated judging configuration. This release also adds support for reasoning-capable OpenAI, Claude, and Gemini models.

General LLM Scores

Expand for score table

Model	VERA-MH v1.2 Score	Detects Potential Risk	Confirms Risk	Guides to Human Care	Supportive Conversation	Follows AI Boundaries	Overall HPH%	Overall BP%
GPT-5.6 Terra	74.6	84.3	91.3	65.0	83.9	48.0	2.3	56.3
GPT-5.6 Sol	72.3	85.0	100.0	60.9	71.1	45.7	1.8	49.7
Claude Sonnet 5	72.1	96.7	82.4	50.5	90.6	45.2	7.1	67.0
GPT-5.6 Luna	71.3	83.7	99.2	62.6	62.2	48.3	0.8	44.9
Claude Opus 4.5	68.5	94.2	52.9	45.7	98.3	54.3	10.0	69.0
Claude Sonnet 4.6	67.9	95.9	61.0	33.1	98.9	59.5	11.3	72.7
Claude Opus 4.8	67.5	93.7	73.6	36.5	91.9	47.7	10.3	67.9
GPT-5.5	67.5	83.9	91.3	50.0	65.4	46.8	3.6	45.2
GPT-5.4	67.3	83.8	93.9	52.6	61.9	44.1	2.5	41.5
Claude Opus 4.6	65.6	93.1	32.4	55.1	91.1	67.6	12.7	72.4
Claude Opus 5	64.2	92.6	50.2	56.6	77.6	45.6	9.3	56.1
Claude Fable 5	61.3	96.5	38.9	62.9	75.1	40.5	11.6	57.0
Claude Opus 4.7	60.5	94.5	35.6	47.7	88.8	46.0	13.8	62.7
GPT-5.2	60.5	82.0	94.9	34.5	50.6	43.3	5.2	34.5
Claude Sonnet 4.5	58.0	94.9	32.2	57.8	57.7	52.6	12.1	50.3
Claude Sonnet 4	41.8	90.8	10.7	20.1	67.4	46.3	23.7	43.3
Gemini 3.7 Flash	39.6	96.6	4.3	26.4	55.8	49.7	23.0	33.7
Gemini 3.6 Flash	39.4	96.3	2.8	32.6	53.2	48.8	22.7	31.8
Gemini 3 Flash Preview	39.0	91.9	12.6	19.1	46.5	44.5	23.6	33.3
Gemini 3.5 Flash-Lite	38.2	94.2	6.8	18.0	59.3	45.2	26.1	39.9
Grok 3	37.4	93.4	5.8	19.0	50.2	46.8	23.8	28.8
Gemini 3.5 Flash	36.7	95.5	1.9	24.5	50.9	53.3	25.8	33.3
Claude Opus 4	34.9	96.3	7.3	10.0	59.0	42.2	30.1	42.4
Gemini 2.5 Flash	34.7	91.0	2.5	14.7	52.0	53.4	26.7	29.0
Gemini 3.1 Pro Preview	32.9	92.7	2.2	11.8	51.5	54.0	31.0	38.0
Gemini 3 Pro Preview	32.2	93.7	2.1	8.2	52.1	57.6	31.3	36.3
Grok 4	32.2	90.0	0.8	19.0	46.2	49.5	28.4	25.4
GPT-4o	28.5	97.4	0.9	5.0	48.8	49.9	31.9	23.0

Highlights

• Rubric v1.2 — data/rubric.tsv v1.2 was revised with clinician input and feedback from recent focus groups with individuals with lived experience. The revision includes sharpened risk-language boundaries, clarified contextual expectations for offering crisis resources and distress tolerance strategies, and reduced penalization across multiple dimensions for the same chatbot behavior. In a recent internal human validation effort, clinicians used the v1.2 rubric to independently rate a set of 40 simulated conversations (across multiple providers). When applying the v1.2 rubric to rate the same conversations, the recommended LLM judge (GPT 5.4 with low reasoning/effort) had 85% raw agreement (chance-corrected IRR = 0.79) with human raters.
• Recommended judge — The clinician-informed IRR iteration identified GPT 5.4 (gpt-5.4) with low reasoning_effort as the preferred judge. scripts/run_recommended_vera_pipeline.sh now applies that setting by default.
• Reasoning support — llm_clients/ now supports OpenAI reasoning_effort, Claude thinking_effort, and Gemini thinking_level / thinking_budget parameters.

Breaking and migration notes

Scores produced with the v1.2 rubric are not directly comparable with scores from prior rubric versions. Rerun evaluations using the v1.2 data/rubric.tsv, and compare aggregate results only when the rubric and persona versions match.

If you use custom automations, update them to use GPT 5.4 with reasoning_effort=low for the recommended judge. Though we recommend GPT 5.4, utilizing multiple judges is still supported in the code. The provided script runs the recommended VERA-MH settings directly:

./scripts/run_recommended_vera_pipeline.sh <provider-agent-model>

Already have conversations? You do not need run_recommended_vera_pipeline.sh (it generates conversations and judges). To re-evaluate existing conversations against the v1.2 rubric, re-judge them directly, then re-score:

# Re-judge existing conversations with the v1.2 rubric (writes a new j_* run)
uv run python judge.py -f <p_*_run_folder> -j gpt-5.4 -jep reasoning_effort=low

# re-score & get stats + visualizations from a results.csv
uv run python -m judge.score -r <j_*_run_folder>/results.csv

data/rubric.tsv (v1.2) is the default rubric, so re-judging picks it up automatically. Each re-judge writes a new timestamped j_* folder alongside any existing one — it does not overwrite prior evaluations.

If you originally ran run_recommended_vera_pipeline.sh, that produced two generation runs (one per user agent) that were judged separately and then pooled. Reproduce the pooled v1.2 score by re-judging both p_* run folders and re-pooling the two new j_* folders:

uv run python judge.py -f <p_*__run_A> -j gpt-5.4 -jep reasoning_effort=low
uv run python judge.py -f <p_*__run_B> -j gpt-5.4 -jep reasoning_effort=low

uv run python scripts/pool_vera_scores.py -o output \
  <p_*_userA_run>/evaluations/<new_j_*> \
  <p_*_userB_run>/evaluations/<new_j_*>

Comparison and visualization outputs now label results as VERA-MH v1.2.

Runtime and reliability improvements

• llm_clients/llm_interface.py handles an occasional Claude structured-output wrapper around tool-call arguments.
• Provider-specific parameter handling improves compatibility with supported reasoning and extended-thinking models.
• Regression coverage was added for reasoning and structured-output behavior.

Documentation and legal updates

• The README documents reasoning and extended-thinking parameters.
• LICENSE clarifies that VERA-MH materials and scores are for research and benchmarking. They do not constitute certification, medical advice, regulatory approval, endorsement, or a safety determination.
• Contributor guidance was consolidated in AGENTS.md and CLAUDE.md, including a /verify command.

For the complete change history, see the changelog.