Add `assert-eval init` conversational config designer by tangym · Pull Request #110 · responsibleai/ASSERT

tangym · 2026-05-27T10:36:13Z

Summary

Adds assert-eval init — an AI-powered conversational config designer that interactively builds eval_config.yaml files through a multi-turn dialogue with the user. Also adds multi-preset judge support so generated configs can combine multiple judge presets.

What's new

`assert-eval init` command

A Click-based CLI command that launches a design agent to walk users through creating an evaluation config:

Conversational loop (_design_agent.py) — Three-action protocol (ask/propose/done) with validation-driven self-correction and draft preservation
LLM caller (_llm.py) — Synchronous litellm wrapper with chat_completion() and chat_completion_json()
Validation bridge (_validate.py) — Structural YAML validation without filesystem context; feeds errors back to the agent for self-correction. behavior is a required field.
Context builder (_context.py) — Assembles system messages with token-budget trimming from existing configs, eval specs, and examples. Threads seed_path for relative-path resolution and emits a token-budget warning when context is trimmed.
Atomic file writer (_emit.py) — Writes YAML via .tmp + atomic replace()
System prompt (prompts/init_system.md) — Design agent behavior prompt. Clarifies that policy_violation and overrefusal are always-included built-in judge dimensions, and asks the user about additional dimensions on top.

CLI options: --output, --describe, --from, --model, --non-interactive, --max-turns, --force, --dry-run

Default output: eval_config.yaml

Multi-preset judge support

pipeline.judge.preset now accepts a list of preset names. Dimensions from all presets are merged (with dedup), and inline overrides still take precedence.

Test coverage

44+ unit tests across 7 test files covering command, context builder, design agent, emitter, action parsing, and validation
9 integration tests (test_library_e2e.py) for multi-preset combining, dedup, inline overrides, empty list, and error cases

Commits (oldest → newest)

init: add module skeleton and CLI registration
init: add synchronous LLM caller module
init: add validation bridge for proposed configs
init: add context builder for design agent system prompt
init: implement design agent conversation loop
init: implement atomic file writer for generated configs
init: add system prompt and polish design agent UX
Add unit tests for p2m init module
feat(config): allow pipeline.judge.preset to accept a list of presets
test(config): cover multi-preset combining for judge dimensions
Remove leftover debug hook from conftest.py
Fix _emit.py: use replace() and correct docstring
Make behavior a required field in validation
Change default output filename to eval_config.yaml
Thread seed_path to system prompt and add token-budget warning
Merge remote-tracking branch origin/main into yemingtang/p2m-init
Rename p2m CLI references to assert-eval in init module
docs: use assert-eval naming in cli.md init section
docs: clarify policy_violation and overrefusal are built-in judge dimensions

Add p2m/init/ package with Click command definition exposing all CLI options (--output, --describe, --from, --model, --non-interactive, --max-turns, --force, --dry-run, etc.) and stub modules for the design agent loop and file emitter. Register the 'init' subcommand in p2m/cli.py.

Thin wrapper around litellm.completion for the init design agent. Provides chat_completion() for text and chat_completion_json() for structured JSON responses, reusing the existing _classify_llm_error pattern from model_client.py.

Structural validation of YAML proposals without requiring stage modules or filesystem context. Checks top-level keys, identifiers, behavior shape, pipeline stages, and reserved dimension names. Returns error list for LLM self-correction feedback.

Assembles the system message from the prompt template, CONFIG_REFERENCE schema, preset catalog, seed config, and CLI-provided hints. Applies token-budget trimming when the prompt exceeds model context thresholds.

Three-action protocol (ask/propose/done) with JSON parsing, markdown fence stripping, validation-driven self-correction, accept/refine/skip user flow, non-interactive single-shot path, KeyboardInterrupt draft preservation, and turn budget enforcement.

Normalizes YAML via roundtrip, writes to .tmp sibling file, then renames atomically. Preserves draft on failure. Respects --force flag.

- test_init_parse_action: action parsing, edge cases, error handling - test_init_validate: YAML validation, reserved names, identifiers - test_init_emit: atomic file writing, force overwrite, dry-run - test_init_context: system message assembly, token estimation - test_init_design_agent: design loop with mocked LLM - test_init_command: CLI integration via CliRunner 44 tests, all passing.

Combine dimensions from multiple judge preset groups. Accepts a single preset name (str) or a list of names. Dimensions are merged in order; later presets override earlier ones on dimension-name conflict. Inline dimensions under pipeline.judge.dimensions still take final priority. The new _parse_preset_names helper validates input (rejects non-string items, empty strings, and non-list/non-str types) and deduplicates repeated names (first occurrence wins) for deterministic merge order.

Add MultiPresetTest with 9 cases covering pipeline.judge.preset list syntax: single-element list equivalence, multi-preset dimension union, duplicate dedupe, inline override priority, empty list, and the four validation error paths (wrong type, non-string item, empty string item, unknown preset name).

Add p2m init usage examples, tips, and CLI reference to README, AGENTS.md, quickstart, writing-eval-specs, examples/README, and status-and-roadmap. Doc examples use --model azure/gpt-5.4.

Update default --model in _command.py and _context.py to use azure/gpt-5.4-mini. Add gpt-5.4/gpt-5.4-mini context window entries and fix _context_window_for() to strip azure/ prefix. Update all init tests to match.

Copilot

Pull request overview

Adds a new p2m init CLI workflow that uses an LLM-driven multi-turn dialogue to generate an eval config YAML, and extends config parsing to support combining multiple judge presets (merging/deduping dimensions with inline overrides taking precedence). The PR also includes extensive unit/integration tests and documentation updates describing the new command.

Changes:

Introduce p2m init implementation (prompt/context builder, LLM caller, design loop, validation bridge, atomic emitter) and wire it into the CLI.
Extend pipeline.judge.preset to accept a list of preset names and merge preset dimensions deterministically with inline overrides.
Add tests + docs covering p2m init usage and multi-preset judge behavior.

Reviewed changes

Copilot reviewed 25 out of 26 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
tests/test_library_e2e.py	Adds integration coverage for multi-preset judge preset merging/dedup/overrides.
tests/test_init_validate.py	Adds unit tests for the `p2m init` validation bridge.
tests/test_init_parse_action.py	Adds tests for JSON action parsing and fence stripping in the design agent.
tests/test_init_emit.py	Adds tests for atomic YAML emission/overwrite behavior.
tests/test_init_design_agent.py	Adds tests for the design loop (done/parse-error recovery/validation retry/budget exhaustion/LLM errors).
tests/test_init_context.py	Adds tests for system prompt assembly and token budget helpers.
tests/test_init_command.py	Adds tests for the `p2m init` Click command behavior.
tests/conftest.py	Introduces a pytest hook for debugging `caplog` behavior in specific tests.
README.md	Documents `p2m init` at a high level and links to CLI reference.
prompts/init_system.md	Adds the system prompt defining the ask/propose/done protocol and config guidance.
p2m/init/_validate.py	Adds schema-ish structural validation for proposed YAML during the init conversation.
p2m/init/_llm.py	Adds a synchronous LiteLLM wrapper with error classification behavior.
p2m/init/_emit.py	Adds atomic `.tmp` write + rename output for generated configs.
p2m/init/_design_agent.py	Implements the multi-turn conversational design loop and action protocol handling.
p2m/init/_context.py	Builds the system message with optional injected context and token-budget trimming.
p2m/init/_command.py	Defines the Click command/options for `p2m init`.
p2m/init/init.py	Introduces the `p2m.init` package marker.
p2m/config.py	Adds list-valued judge preset parsing and merges dimensions across presets.
p2m/cli.py	Registers the new `init` command with the top-level CLI.
examples/README.md	Mentions `p2m init` as an alternative to hand-authoring YAML.
docs/writing-eval-specs.md	Adds a tip pointing users to `p2m init` as a faster starting path.
docs/status-and-roadmap.md	Adds `p2m init` to the “Stable enough to try” list.
docs/reference/cli.md	Adds a full CLI reference section for `p2m init`.
docs/quickstart.md	Adds a “next steps” pointer to `p2m init`.
AGENTS.md	Updates contributor guidance to include `p2m init` as the fast path.
.gitignore	Adjusts venv ignore pattern and adds `.chainlit/`.

Drop the pytest_runtest_call hook and _spy machinery that was added during development debugging. Only the reusable _SpyHandler class is kept.

- Use str.replace() instead of re.sub() for plain-string replacement of the output filename placeholder. - Fix docstring to say 'Replace' instead of 'Emit' to match the method's actual behavior.

Reject configs with a missing behavior key rather than silently accepting them. The field is central to every eval spec.

Rename the default --output from eval.yaml to eval_config.yaml for consistency with the config reference and examples. Update the CLI docs and test assertions to match.

- Pass seed_path through _command.py → run_design_loop() → build_system_message() so the seed config participates in the system prompt's existing token-budget trimming logic. - Add a warning when the seed config in the user message exceeds 10% of the model's context window, without trimming it. - Export estimate_tokens and context_window_for as public aliases from _context.py for reuse by the design agent. - Improve LLM error handling: catch LLMAuthError, LLMInputError, and LLMProviderError with specific user-facing messages instead of a bare except.

# Conflicts: # AGENTS.md

…ensions Update init_system.md to make clear that policy_violation and overrefusal are always included as built-in dimensions (not from the safety-core preset), and reframe the judge question to ask what additional dimensions the user wants on top.

…esults - Restyle both incident_triage_simple and incident_triage_agent READMEs to match the PR #86 azure_doc_qa customer-facing format: top headline, what-this-demonstrates bullets, architecture diagram, quick-start, env-var table, judge-dimension list, expected-output file list. - Strip every reference to Agent Shield / ACS / agent_guarded.py / guardrails.yaml from both folders. Keep agent.py only. - Drop trade-off chart artifact + script reference. Drop all committed result snapshots (results are highly non-deterministic; the README now describes what artifacts to expect, not specific numbers). - Consolidate to a single eval_config.yaml per folder. Drop variant configs (baseline / naive_prompt / guarded / GEPA). Rebased onto current main (post #110 multi-preset judge dimensions). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…t_eval rename Merges 19 commits from main into the rename branch and resolves conflicts in favor of the assert_eval naming. Continues the rename so that the legacy `p2m` name is gone everywhere outside of frozen historical artifact snapshots. Conflict resolution: - All Python module docstrings: keep MIT license header from main + the rename branch's docstring text (which uses `ASSERT` for the project brand and `assert_eval` for the Python module name). - assert_eval/config.py: take main's multi-preset judge logic and rename `from p2m.library.loader` -> `from assert_eval.library.loader`. New rename work brought in by the merge: - `p2m/init/` (PR #110 conversational config designer) renamed to `assert_eval/init/` and all internal `from p2m.*` imports updated. - assert_eval/cli.py: `from p2m.init._command` -> `from assert_eval.init._command`. - New tests/test_init_*.py: all `p2m.init.*` / `p2m.cli` / `p2m.core.*` references and `p2m init` docstrings updated. - README.md, docs/quickstart.md, docs/status-and-roadmap.md, docs/writing-eval-specs.md, examples/README.md: `p2m init` -> `assert-eval init`. - .github/workflows/regression.yml: path filter `p2m/**` -> `assert_eval/**`, env var `P2M_AZURE_DEPLOYMENT` -> `ASSERT_AZURE_DEPLOYMENT`, and header comments updated. - uv.lock: package name `p2m-policy` -> `assert-eval` (matches pyproject.toml). Guard test: - tests/test_no_p2m_references.py walks every git-tracked file and fails if the legacy `p2m` name reappears outside the small allowlist of historical artifact snapshots (examples/incident_triage_agent/artifacts/results/) and the test file itself. Excluded (intentional, per the original rename convention): - examples/incident_triage_agent/artifacts/results/**/config.yaml - these are frozen records of past runs and must not be rewritten. Validated: - import assert_eval.cli, assert_eval.init._command, ... all succeed. - assert-eval --help works. - New no-p2m guard test passes and correctly fails when a `p2m` reference is reintroduced. - pytest tests (excluding viewer tests that require `npm ci`): 832 passed, 16 skipped, 0 failed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Refine init system prompt conversation flow Replace flat bullet list with structured numbered sections (1-5) covering system context, target type, behavior definition, test set generation, and judge configuration per minthigpen's PR #110 review. * Add section-coverage pacing guardrails to init prompt Smaller models (e.g. gpt-5.4-mini) were jumping to propose after only 2 questions, skipping Behavior Definition, Test Set Generation, and Judge Configuration entirely. Add three guardrails without changing minthigpen's 5-section structure: - Tighten the 'fewer rounds' clause to require verification of each remaining section rather than blanket permission to skip. - Add pacing instruction: acknowledge info from verbose answers but continue asking about uncovered sections. - Add a pre-propose self-check: verify all 5 sections have concrete answers before switching to propose. * Move section checklist into propose action as prerequisite gate Smaller models ignored the standalone 'Before proposing' section. Merge the 5-point checklist into the propose action definition itself, with stronger language: 'do NOT emit propose until every check is yes' and 'you MUST use ask instead'. * Don't ask for system prompt when target is callable Callable targets own their own prompt. Only ask for a system prompt when the user selects the model target type. * Make judge presets explicitly optional in init prompt Rewrite Section 5 (Judge Configuration) to: - Separate presets and custom dimensions into two explicit questions - State clearly that presets are optional and if declined, omit the preset: key from the YAML entirely - Add a 'Critical' paragraph reinforcing literal respect for user answers - Update checklist item #5 to say 'whether to include' instead of 'which' Fixes the issue where mini models added all judge presets (safety-core, grounding, tool-use, robustness, operational) even when the user explicitly said 'none'. * Increase default --max-turns from 20 to 30 With 5 question sections, propose/refine cycles, and potential validation retries for malformed YAML, 20 turns is too tight. 30 provides comfortable headroom without being unlimited.

tangym added 12 commits May 27, 2026 02:22

init: add synchronous LLM caller module

cede9ae

Thin wrapper around litellm.completion for the init design agent. Provides chat_completion() for text and chat_completion_json() for structured JSON responses, reusing the existing _classify_llm_error pattern from model_client.py.

init: add context builder for design agent system prompt

9594e31

Assembles the system message from the prompt template, CONFIG_REFERENCE schema, preset catalog, seed config, and CLI-provided hints. Applies token-budget trimming when the prompt exceeds model context thresholds.

init: implement atomic file writer for generated configs

69d1073

Normalizes YAML via roundtrip, writes to .tmp sibling file, then renames atomically. Preserves draft on failure. Respects --force flag.

init: add system prompt and polish design agent UX

2d88d0a

docs: add p2m init documentation across all doc files

660fb8a

Add p2m init usage examples, tips, and CLI reference to README, AGENTS.md, quickstart, writing-eval-specs, examples/README, and status-and-roadmap. Doc examples use --model azure/gpt-5.4.

chore: change default model from gpt-4.1-mini to azure/gpt-5.4-mini

4abac6c

Update default --model in _command.py and _context.py to use azure/gpt-5.4-mini. Add gpt-5.4/gpt-5.4-mini context window entries and fix _context_window_for() to strip azure/ prefix. Update all init tests to match.

AaronAspinwall123 requested a review from Copilot May 27, 2026 15:35

Copilot started reviewing on behalf of AaronAspinwall123 May 27, 2026 15:36 View session

Copilot AI reviewed May 27, 2026

View reviewed changes

tangym added 8 commits May 27, 2026 23:24

Remove leftover debug hook from conftest.py

a8afd3f

Drop the pytest_runtest_call hook and _spy machinery that was added during development debugging. Only the reusable _SpyHandler class is kept.

Fix _emit.py: use replace() and correct docstring

6c6b70f

- Use str.replace() instead of re.sub() for plain-string replacement of the output filename placeholder. - Fix docstring to say 'Replace' instead of 'Emit' to match the method's actual behavior.

Make behavior a required field in validation

7d4515a

Reject configs with a missing behavior key rather than silently accepting them. The field is central to every eval spec.

Change default output filename to eval_config.yaml

15fbfe8

Rename the default --output from eval.yaml to eval_config.yaml for consistency with the config reference and examples. Update the CLI docs and test assertions to match.

Merge remote-tracking branch 'origin/main' into yemingtang/p2m-init

9aaa575

# Conflicts: # AGENTS.md

Rename p2m CLI references to assert-eval in init module

0b235fc

docs: use assert-eval naming in cli.md init section

613c837

tangym enabled auto-merge (squash) May 28, 2026 00:10

tangym changed the title ~~Add p2m init conversational config designer~~ Add assert-eval init conversational config designer May 28, 2026

mmergawi approved these changes May 28, 2026

View reviewed changes

tangym merged commit bfcbcb8 into main May 28, 2026
4 of 5 checks passed

minthigpen reviewed May 28, 2026

View reviewed changes

Comment thread prompts/init_system.md

minthigpen approved these changes May 28, 2026

View reviewed changes

tangym mentioned this pull request May 29, 2026

Refine init system prompt: pacing, presets, and max-turns #134

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add `assert-eval init` conversational config designer - #110

Add `assert-eval init` conversational config designer#110
tangym merged 21 commits into
mainfrom
yemingtang/p2m-init

tangym commented May 27, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

tangym commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's new

assert-eval init command

Multi-preset judge support

Test coverage

Commits (oldest → newest)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tangym commented May 27, 2026 •

edited

Loading

`assert-eval init` command