Refactor: decouple rubric logic from SI structure by luca-belli · Pull Request #186 · SpringCare/VERA-MH

luca-belli · 2026-08-05T16:56:19Z

Summary

This refactor decouples rubric parsing, navigation, and question-ID reporting from assumptions encoded for the shipped suicide-ideation (SI) rubric.

Treats question IDs as opaque strings rather than assuming they are integers, including when extracting failure reasons and building improvement summaries.
Requires every primary question row to declare its dimension explicitly.
Validates rubric navigation at load time: duplicate IDs, missing targets, malformed deterministic questions, and navigation cycles now fail early.
Replaces the judge's hard-coded SI control-question text with rubric-owned deterministic-answer metadata.
Moves assertions about the shipped SI rubric's exact dimensions, columns, and navigation into SI-specific integration tests while keeping generic parser behavior in unit tests.
Documents the supported rubric-bundle contract, authoring rules, CLI usage, and current compatibility boundary in docs/rubric.md.

This PR builds on the SI bundle and manifest work already merged in #180; it does not introduce data/SI/rubric_manifest.json or the data/SI/ directory layout.

Important rubric change: `Auto Answer`

data/SI/rubric.tsv now has an Auto Answer column, and Question 5 is marked true.

Q5 is a deterministic control-flow row: after the chatbot misses detected risk, the Confirms Risk dimension must be marked Not Relevant and evaluation must continue at Q9. Previously, LLMJudge recognized the exact English text of Q5 through a hard-coded SI-specific dictionary and skipped the LLM.

That behavior now belongs to the rubric itself. Marking Q5 with Auto Answer=true tells the generic judge to select its sole declared answer without an LLM call, which preserves the existing SI scoring and NOT_RELEVANT>>9 navigation while removing the coupling to Q5's wording. This matters because:

changing the rubric wording can no longer silently break a Python string match;
alternative rubric bundles can declare their own deterministic control rows without modifying judge code;
deterministic navigation remains deterministic instead of depending on an LLM response.

An auto-answer question must declare exactly one explicit answer. The clinical question and scoring text are unchanged; this PR adds explicit navigation metadata to the rubric schema.

Rubric-authoring impact

Every primary question row must now contain a non-empty Dimension value; blank dimension cells are only valid on continuation rows for additional answers.
Question IDs and GOTO targets may be non-numeric identifiers and are preserved as strings through result reporting.
Rubric loading now rejects duplicate question IDs, missing navigation targets, and cycles rather than allowing invalid graphs to reach evaluation.

Scope

Scoring and visualization still contain an SI-derived global dimension list. Consequently, a second rubric currently works end to end only when it preserves the existing five dimension names and VERA scoring semantics. The assumptions and setup steps are documented in docs/rubric.md.

Removing that remaining coupling requires persisting rubric dimensions (and a rubric fingerprint) with evaluation output and is tracked separately in TODO.

Test coverage

Adds unit coverage for opaque IDs, deterministic questions, missing targets, and cycle rejection.
Adds SI-specific structure and navigation coverage against the shipped rubric.
Adds an orchestration regression test proving that _ask_all_questions() processes Q9's NOT_RELEVANT>>23 route without requiring live LLM access.

luca-belli · 2026-08-05T17:10:39Z

+from judge.rubric_config import RubricConfig
+
+
+@pytest.mark.integration


Note: this is the equivalent of the removed tests below async def test_question_13_full_flow_integration

luca-belli · 2026-08-05T18:21:48Z

Why the old special-case tests were removed

TestSpecialCasesQuestionAnswersMatchRubric was coupled to the implementation this PR removes: it asserted that the exact English text of SI Question 5 appeared in the Python SPECIAL_CASES_QUESTION_ANSWERS dictionary and mapped to the exact rubric answer text.

The judge no longer recognizes deterministic questions by matching their wording. The rubric now declares that behavior through Auto Answer=true, so retaining the old test would require retaining the SI-specific dictionary and would defeat the purpose of this refactor.

The behavior is still covered at the new ownership boundaries:

tests/integration/rubrics/si/test_question_navigator.py::test_auto_answer_moves_to_next_not_relevant_dimension checks the shipped SI rubric: Q5 is deterministic, has exactly one answer, and routes through NOT_RELEVANT>>9 to Q9.
tests/unit/judge/test_llm_judge.py::test_auto_answer_skips_llm checks generic judge behavior: an auto-answer question selects its sole rubric answer without calling the LLM.
tests/unit/judge/test_rubric_config.py::test_invalid_auto_answer_declaration checks the schema invariant that an auto-answer question must declare exactly one explicit answer.

The former manually simulated Q9 NOT_RELEVANT test was also moved into the SI-specific test area and strengthened: tests/integration/rubrics/si/test_not_relevant_flow.py now calls _ask_all_questions() and verifies the actual Q9 NOT_RELEVANT>>23 orchestration without live LLM access.

In short, the exact string-coupling assertions were intentionally deleted; the rubric contract, generic auto-answer behavior, validation, and NOT_RELEVANT orchestration remain covered separately.

jgieringer · 2026-08-05T21:00:08Z

@@ -1,4 +1,4 @@
-Question ID	Dimension	Severity	Risk Type	Question	Examples	Answer	GOTO		Notes for Interpretability of GOTO logic
+Question ID	Dimension	Severity	Risk Type	Question	Examples	Answer	GOTO		Notes for Interpretability of GOTO logic	Auto Answer


The Notes for Interpretability of GOTO logic is more for human reading, right? So should Auto Answer be next to GOTO so it isn't accidentally lost if a developer is building a rubric?

Yes, those are for humans. And I think you might be right, i will double check the ordering doesn't matter

jgieringer

Just a couple comments. Thanks for cleaning all the things!

jgieringer · 2026-08-05T21:11:32Z

+`--rubrics` and `--rubric-manifest` are independent: the former selects the
+evaluation rubric, while the latter selects generation personas and their context
+template. There is currently no symbolic rubric-name shorthand, so pass the full
+manifest path.


I worry this could be confusing. Both use the same file?
I thought there would be one manifest file that then pointed to whatever rubrics/settings you wanted?

good point. I'd say this is a edge case that is not supposed to happen. This is only a temporary hack while creating the proper --target CLI so will be solved soon

#184 but it's branching from and to the wrong places

emily-vanark · 2026-08-06T14:47:20Z

+Judge existing conversations with the new bundle:
+
+```bash
+uv run python judge.py \


do we want to update this and the next example to run through vera.py?

I have a PR next to use the new CLI, which introduces a --target to avoid the hack of calling out the manifest explicitly

emily-vanark · 2026-08-06T14:53:42Z

+template. There is currently no symbolic rubric-name shorthand, so pass the full
+manifest path.
+
+When generating an improvement report, pass the new TSV explicitly so question


It would be cool if eventually we were smart enough to read the immutable config from the output folder, use that to find the right rubric, and pass that in to the summarize results. (but not necessary now)

thank you, that also raises a good point, that the config file should have sha of the files it uses, otherwise there might be silent failures.. adding to the TODO

emily-vanark · 2026-08-06T15:00:34Z

        )

    @classmethod
    async def load_bundle(cls, manifest_path: str) -> "RubricConfig":


this function name makes me think it will load all the files in the bundle, but the docstring says it only loads the rubric from the bundle... the code looks like it's loading all the rubric-necessary files from the bundle... so maybe its the docstring that could be edited for clarity?

updated to Load a rubric's configuration and metadata from a rubric bundle manifest.

Add note about including SHA of config files to avoid silent failures.

luca-belli · 2026-08-06T15:56:22Z

+template. There is currently no symbolic rubric-name shorthand, so pass the full
+manifest path.
+
+When generating an improvement report, pass the new TSV explicitly so question


thank you, that also raises a good point, that the config file should have sha of the files it uses, otherwise there might be silent failures.. adding to the TODO

luca-belli · 2026-08-06T15:58:54Z

+Judge existing conversations with the new bundle:
+
+```bash
+uv run python judge.py \


I have a PR next to use the new CLI, which introduces a --target to avoid the hack of calling out the manifest explicitly

…factor/decouple-si-rubric

…ERA-MH into refactor/decouple-si-rubric

refactor: decouple rubric logic from SI structure

15fdd2d

luca-belli requested a review from a team as a code owner August 5, 2026 16:56

luca-belli marked this pull request as draft August 5, 2026 17:06

luca-belli commented Aug 5, 2026

View reviewed changes

test: cover not relevant judge orchestration

e5e2309

clarifying docstring

4243e4a

luca-belli changed the title ~~refactor: decouple rubric logic from SI structure~~ Refactor: decouple rubric logic from SI structure Aug 5, 2026

docs: document compatible rubric requirements

8076f30

luca-belli marked this pull request as ready for review August 5, 2026 18:33

jgieringer reviewed Aug 5, 2026

View reviewed changes

jgieringer approved these changes Aug 5, 2026

View reviewed changes

emily-vanark reviewed Aug 6, 2026

View reviewed changes

Comment thread docs/rubric.md

emily-vanark reviewed Aug 6, 2026

View reviewed changes

Update TODO with config file SHA requirement

c71ef5e

Add note about including SHA of config files to avoid silent failures.

luca-belli commented Aug 6, 2026

View reviewed changes

luca-belli and others added 4 commits August 6, 2026 09:03

Update judge/rubric_config.py

c784e66

Merge branch 'feat/VERA_2.0' of github.com:SpringCare/VERA-MH into re…

837b71b

…factor/decouple-si-rubric

Merge branch 'refactor/decouple-si-rubric' of github.com:SpringCare/V…

ba05871

…ERA-MH into refactor/decouple-si-rubric

todo

fa4a15b

luca-belli merged commit 866241c into feat/VERA_2.0 Aug 6, 2026

		from judge.rubric_config import RubricConfig


		@pytest.mark.integration

		@@ -1,4 +1,4 @@
		Question ID Dimension Severity Risk Type Question Examples Answer GOTO Notes for Interpretability of GOTO logic
		Question ID Dimension Severity Risk Type Question Examples Answer GOTO Notes for Interpretability of GOTO logic Auto Answer

Uh oh!

Conversation

luca-belli commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Important rubric change: Auto Answer

Rubric-authoring impact

Scope

Test coverage

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luca-belli commented Aug 5, 2026

Why the old special-case tests were removed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jgieringer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

emily-vanark Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

luca-belli commented Aug 5, 2026 •

edited

Loading

Important rubric change: `Auto Answer`

emily-vanark Aug 6, 2026 •

edited

Loading