feat: wire --rubrics flag to RubricConfig.load_bundle()

[luca-belli]

Summary

Phase 0 wiring, for both generation and judging -- stacked on #177, which builds the shared foundation (utils/rubric_manifest.py + RubricConfig.load_bundle()) for both sides without wiring up any CLI flag. This PR is where both sides actually get connected to it. Base branch is #177's branch, not main.

Judging side:

• judge.py's --rubrics/-r flag was declared but never read; main() now loads args.rubrics[0] via RubricConfig.load_bundle() instead of hardcoding RubricConfig.load(rubric_folder="data")
• Default changed from data/rubric.tsv to data/rubric_manifest.json (the production manifest added in feat: add RubricConfig.load_bundle() for rubric bundle manifests #177)
• Passing more than one --rubrics value now prints a warning to stderr and uses only the first (multi-rubric support isn't implemented yet; kept the flag's list shape forward-compatible)
• run_pipeline.py has its own --rubrics flag that it forwards straight into judge.py's main() -- updated its default the same way so the full pipeline doesn't break on its own default
• Added tests/fixtures/rubric_manifest_multi_row.json, a second test bundle with genuinely different rubric content, used by a new end-to-end test proving --rubrics actually drives which bundle loads (not a hardcoded no-op)

Generation side (the counterpart that closes out Phase 0 -- see docs/architecture.md's Phase 0 entry, expanded in PR #170):

• generate.py gains --rubric-manifest <path>, which loads the same manifest's personas list (via utils.rubric_manifest.load_manifest_personas(), from feat: add RubricConfig.load_bundle() for rubric bundle manifests #177) and uses it as the persona file, instead of generate.py's previous fixed data/personas.tsv default
• Mirrors judge.py's "first entry wins, warn on extras" handling when a manifest lists more than one persona file
• run_pipeline.py gets the same --rubric-manifest flag for its generation step, alongside its existing --rubrics (judging) flag -- passing the same manifest to both attaches that run's personas to the rubric it's evaluated against

With this PR, a single rubric bundle manifest can drive both what gets generated and what it's judged against -- Phase 0 is complete on both sides of the pipeline, not just judging.

Rebased after #177 picked up a fix: #177 originally shipped load_manifest_personas() without resolving personas entries relative to the manifest's own folder (a bug, since docs/architecture.md requires manifest-relative resolution). That's now fixed in #177 directly, and this PR is rebased on top -- the one test here that depended on the old (buggy) behavior is updated to expect a manifest-relative resolved path instead.

Test plan

• uv run pytest -m "not live" passes (976 passed)
• New test: --rubrics <manifest> loads distinct rubric content for two different manifests (real RubricConfig.load_bundle(), no mocking of that call)
• New test: multiple --rubrics values warns on stderr and uses only the first
• New tests: generate.py --rubric-manifest selects personas from the manifest (resolved relative to the manifest's folder); omitting it keeps the default data/personas.tsv; a manifest with no personas raises ValueError; ConversationRunner forwards the resolved persona path into load_prompts_from_csv
• Updated existing unit/integration tests that asserted the old data/rubric.tsv default or mocked RubricConfig.load
• uv run ruff check / uv run ruff format --check clean
• uv run pyright clean (no new errors vs. base)

[jgieringer]

LGTM

[emily-vanark]

It's interesting how many shorter conversations we had, given that this was in the instructions... (not a blocker, just... noticing)

[emily-vanark]

I'm confused... I thought the PR text said we were replacing these hard-coded paths with the values from the rubric bundle... did I misunderstand?

[emily-vanark]

Or we're just falling back to the old default for now?

[luca-belli]

defaults are unchanged not to break code, but those options are not exposed, so we can point to a different persona file/prompt

[Copilot]

Pull request overview

Wires the existing CLI surfaces to the rubric bundle manifest foundation (from stacked PR #177), so a single manifest can drive both (a) which rubric the judge loads and (b) which personas generation uses, completing “Phase 0” end-to-end across generate.py, judge.py, and run_pipeline.py.

Changes:

• Judging: judge.py --rubrics now loads RubricConfig.load_bundle(args.rubrics[0]), defaults to data/rubric_manifest.json, and warns when multiple values are provided; run_pipeline.py default is updated accordingly.
• Generation: adds --rubric-manifest to generate.py and run_pipeline.py to source personas + persona context template from the manifest (with “first entry wins” warning behavior).
• Updates persona prompt rendering to a two-template contract ({persona_context} injected into the shared template), and adds/updates unit + integration tests plus a new fixture manifest to prove the CLI flags are live.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/unit/judge/test_judge_cli.py	Updates judge CLI tests for manifest default + load_bundle, adds end-to-end tests proving --rubrics drives rubric selection and warns on multiples.
tests/unit/generate_conversations/test_utils.py	Adapts unit tests for the new two-template persona prompt contract and adds negative tests for missing columns/templates.
tests/unit/generate_conversations/test_generate_cli.py	Adds tests ensuring generate.main() uses manifest-derived persona paths/templates and raises when personas are absent.
tests/integration/test_pipeline.py	Updates pipeline defaults/tests to the manifest-backed rubric default.
tests/integration/test_conversation_runner.py	Ensures custom persona paths/templates are forwarded into load_prompts_from_csv.
tests/fixtures/rubric_manifest_multi_row.json	Adds a second manifest fixture to verify distinct rubric bundles load differently.
run_pipeline.py	Updates --rubrics default/help to manifest and adds --rubric-manifest for generation persona selection.
README.md	Updates judge CLI documentation to reflect manifest-based --rubrics default/behavior.
judge.py	Implements manifest-backed rubric loading via RubricConfig.load_bundle() and warns when multiple --rubrics values are provided.
generate.py	Adds --rubric-manifest wiring to select persona TSV + persona context template from the manifest (with warnings/validation).
generate_conversations/utils.py	Makes persona context template required; renders schema-specific context into a shared prompt template.
generate_conversations/runner.py	Threads persona prompt/context-template paths through ConversationRunner into prompt loading.
data/persona_prompt_template.txt	Switches to {persona_context} placeholder to match the new two-template rendering model.

Suppressed comments (1)

generate_conversations/utils.py:81

• template.format(persona_context=persona_context) only provides persona_context, so any prompt template that also references TSV columns (e.g. {Name}) will raise KeyError. Passing the row fields into format() makes the function more robust and supports richer prompt templates while keeping the new two-template structure.

            persona_context = context_template.format(**row)
            row["prompt"] = template.format(persona_context=persona_context)
            data.append(row)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[emily-vanark]

We're... hard coding the old default values... in multiple places? (Here and the runner?)

[emily-vanark]

are we not passing the rubric bundle into the judge yet?

[emily-vanark]

Once again, I asked some questions that may just be reader confusion, but certainly not going to stop the PR train at this point!

[luca-belli]

Once again, I asked some questions that may just be reader confusion, but certainly not going to stop the PR train at this point!

To answer your questions: this is a stop-gap solution to not break previous code, and expose some of the vars we'll need soon.