Phase 1. Docs: define unified CLI architecture contract by luca-belli · Pull Request #189 · SpringCare/VERA-MH

luca-belli · 2026-08-07T00:08:35Z

Summary

define vera.py as the short parser/dispatcher and keep CLI support in small,
responsibility-focused vera_cli/ modules
define each manifest.json as one complete target containing rubric,
personas, and generation/judging prompts
make --target <name> whole-bundle selection while preserving explicit
--personas <name> and --rubric <name> component selection
require target expansion into canonical concrete paths before print,
persistence, or dispatch
document the transitional vera_cli.generate_command → generate.main
boundary
require deletion of root generate.py and creation of the permanent
top-level generate/ package to happen atomically later

CLI module contract

arguments.py: top-level parser only
*_arguments.py: per-command flags and CLI defaults
*_config.py: per-command canonical RunConfig resolution
*_command.py: thin adapters to importable Python functions
config.py and targets.py: focused shared input/manifest helpers

Design record

VERA CLI runtime wiring

Stack

depends on docs: define design record conventions #188
after docs: define design record conventions #188 merges, retarget this PR to feat/VERA_2.0

Validation

git diff --check
documentation-only PR; the full stacked implementation suite passes in Phase1: Feat: add unified vera generate command #191

emily-vanark · 2026-08-11T22:51:02Z

 - **Binds:** `vera.py`, `utils/config_schema.py`
 - **Prevents:** a merge/override mechanism between two config sources that would make the effective config ambiguous or order-dependent.
- **Rule:** [ADOPTED] For a given run, a piece of information (model selection/repeats, sampling knobs, persona/rubric lists, etc.) is supplied via `--config` JSON or via CLI flags, never both. Supplying the same information through both is rejected/errors — there is no silent merge. `--config` always resolves internally to the same canonical flag-set the CLI would have produced, and that resolved form is printed at run start. **`--sample <N>` is the one deliberate, named exception:** it MAY be combined with `--config`, since it's a debug-only smoke-test override (UC4) — it never sets information `config.json` itself carries, it only caps how much of the already-resolved persona/rubric/judge lists get used for this invocation, and it's never persisted into the run's own `config.json` artifact (AD-18) or any resumed state. No other flag gets this treatment; a future flag needs its own named exception here, not an implicit ride on `--sample`'s.
+- **Rule:** [ADOPTED] For a given run, a piece of information (model selection/repeats, sampling knobs, persona/rubric lists, etc.) is supplied via `--config` JSON or via CLI flags, never both. Supplying the same information through both is rejected/errors — there is no silent merge. Both forms resolve to the same canonical `RunConfig` before print, persistence, or dispatch. CLI-only presentation/execution controls such as `--debug` and `--print` do not supply run configuration and may accompany either form. **`--sample <N>` is the one deliberate behavior-altering exception:** it MAY be combined with `--config`, since it's a debug-only smoke-test override (UC4) — it never sets information `config.json` itself carries, it only caps how much of the already-resolved persona/rubric/judge lists get used for this invocation, and it's never persisted into the run's own `config.json` artifact (AD-18) or any resumed state. No other run-scoping flag gets this treatment; a future flag needs its own named exception here, not an implicit ride on `--sample`'s.


Is there a world where we'd want the option to put debug, print, or sample into the config that gets stashed with the run to keep track of how we ran that job?

Good question. I am not sure, but I think more information is better than less, so if the debug option is used, it should be reported in the config file as well

emily-vanark · 2026-08-11T23:11:04Z

 **Option B — CLI shorthand:**
 ```
-vera generate -c sonnet -u gpt:1 sonnet:2 --personas data/personas.tsv
+vera generate -c sonnet -u gpt:1 sonnet:2 --personas SI


If I read this as "run conversations once with gpt as the user model and twice with sonnet as the user model" is that correct? Are the :# parts mandatory?

yes this is correct! In general, I am explicit better than implicit, but we can say that if there is no number, the default is 1?

I am lazy, and would default to whatever is most likely, most of the time. So... yes, I'd probably say "if no number, default is 1", especially as that's consistent with the v1 usage.

emily-vanark · 2026-08-11T23:24:45Z

+persona files and persona prompt from that target's manifest.
+
+Generation behavior is also controlled at this input boundary. CLI invocations
+default to `--turns 3`, `--output output`, unlimited concurrency, no total-word


should we.. default to the recommended parameters? so 20 or 30 for the turns?

yeah good point. also I don't think that doc should say what the defaults are, since we might change them.

emily-vanark · 2026-08-11T23:29:45Z

-`--rubric`/`judging.rubrics[]` entries point at a [rubric bundle manifest](./architecture.md#rubric-bundle-manifest) (canonical definition), not a bare `.tsv` path.
+`--target` consumes the rubric and judging prompts from the selected complete
+[target manifest](./architecture.md#target-manifest). Explicit `--rubric SI`
+consumes only SI's rubric and judging prompts and does not select SI's personas


this seems like a distinction without a difference for judging?

emily-vanark · 2026-08-11T23:30:54Z

 - Internally, `--config` always resolves to the same canonical flag-set the CLI would produce, so there is exactly one resolved form regardless of input path. The tool prints this resolved form at run start for terminal/CI-log visibility (it does not write to the shell's own history — an opt-in `--print` flag emits the resolved flag-string with no execution, for a caller who wants to `eval` it into their own shell explicitly).
 - JSON, not YAML — robust when passed as a one-line env var or stdin payload with no escaping ambiguity.
- **Path fields inside `config.json` (`generation.personas`, etc.) resolve relative to `$ROOT`** — the directory containing `vera.py` — never relative to the current working directory the CLI was invoked from, and never relative to `config.json`'s own location. This is a single rule regardless of how the config arrives (`--config <file>`, `--config -`, or `VERA_RUN_CONFIG`), so a config's meaning never depends on where your shell happens to be or where you saved the file. This is distinct from the [rubric bundle manifest](./architecture.md#rubric-bundle-manifest), which deliberately resolves relative to *itself* instead, so a manifest folder stays portable across checkouts — `config.json` doesn't need that property, since it's checkout-specific by nature.
+- **Path fields inside `config.json` (`generation.personas`, etc.) resolve relative to `$ROOT`** — the directory containing `vera.py` — never relative to the current working directory or the config file. A [target manifest](./architecture.md#target-manifest) deliberately resolves its fields relative to its own directory so the complete target remains portable.


I wonder if this will be confusing to humans (having the two file paths resolve to different roots...). I suppose if they're having an agent help, it will keep things straight?

This was done to keep the manifest simplres (e.g. 'persona' : ['./persona_file.csv']), but I can see how it can be confusing. I'll add an example to make it clearer

emily-vanark · 2026-08-11T23:32:14Z

      {"name": "gpt-5", "repeats": 2}
    ],
-    "personas": ["data/personas_a.json", "data/personas_b.json"]
+    "personas": ["data/personas_a.json", "data/personas_b.json"],


wouldn't this likely be ["data/si/personas_a.json", "data/hfo/personas_b.json"] or similar? (the personas down a level in the data folder)?

ops yes thank you

emily-vanark

I'm not sure this is perfect, but it looks generally okay to me, and I figure we'll shake out anything we're missing, logic-wise, in the build and test phases.

luca-belli · 2026-08-12T20:50:06Z

I'm not sure this is perfect, but it looks generally okay to me, and I figure we'll shake out anything we're missing, logic-wise, in the build and test phases.

Thank you! The idea was go give some general rules, and apply those in the next PRs, and, as you noted, adjust as needed

docs: define unified CLI architecture contract

8caff01

luca-belli changed the title ~~docs: define unified CLI architecture contract~~ Phase 1. Docs: define unified CLI architecture contract Aug 7, 2026

luca-belli added 4 commits August 7, 2026 09:04

docs: define manifests as complete targets

29d8611

docs: resolve explicit inputs from targets

65fe3b9

docs: clarify transitional generation boundary

49607ca

docs: simplify unified CLI module structure

aa1611c

emily-vanark reviewed Aug 11, 2026

View reviewed changes

emily-vanark approved these changes Aug 11, 2026

View reviewed changes

Base automatically changed from docs/design-record-conventions to feat/VERA_2.0 August 12, 2026 20:42

luca-belli marked this pull request as ready for review August 12, 2026 20:56

luca-belli requested a review from a team as a code owner August 12, 2026 20:56

docs: resolve unified CLI review feedback

bbfb4e1

luca-belli merged commit 5c7aa15 into feat/VERA_2.0 Aug 13, 2026

luca-belli mentioned this pull request Aug 13, 2026

feat(cli): add unified vera.py CLI orchestrator wired to domain services #185

Closed

Uh oh!

Conversation

luca-belli commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

CLI module contract

Design record

Stack

Validation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emily-vanark left a comment

Choose a reason for hiding this comment

Uh oh!

luca-belli commented Aug 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

luca-belli commented Aug 7, 2026 •

edited

Loading