Add assert-eval init conversational config designer

.gitignore

@@ -1,5 +1,5 @@
 # Environments and local config
-.venv/
+.venv*/
 .env
 .env.*
 !.env.example
@@ -28,6 +28,9 @@ artifacts/
 !viewer/src/routes/**/logs/
 !viewer/src/routes/**/logs/**
 
+# Chainlit local config
+.chainlit/
+
 # PyRIT data
 .pyrit_data/
 

AGENTS.md

@@ -75,6 +75,10 @@ source .venv/bin/activate
 python -m pip install --upgrade pip
 python -m pip install -e ".[otel,langgraph]"
 cp .env.example .env
+
+# Create a config interactively, or use an existing one
+assert-eval init --model azure/gpt-5.4
+# or run the flagship example directly
 assert-eval run --config examples/travel_planner_langgraph/eval_config.yaml
 ```
 
@@ -86,6 +90,10 @@ python -m venv .venv
 python -m pip install --upgrade pip
 python -m pip install -e ".[otel,langgraph]"
 Copy-Item .env.example .env
+
+# Create a config interactively, or use an existing one
+assert-eval init --model azure/gpt-5.4
+# or run the flagship example directly
 assert-eval run --config examples/travel_planner_langgraph/eval_config.yaml
 ```
 
@@ -94,12 +102,13 @@ assert-eval run --config examples/travel_planner_langgraph/eval_config.yaml
 ### Add an eval for a new agent
 
 1. Ask what target shape the developer has: framework agent, custom runtime, Python function, or hosted model.
-2. Create or adapt an eval spec markdown file.
-3. Add `context` describing the agent, tools, users, and constraints.
-4. Add `dimensions` only when systematic variation matters.
-5. Configure the target in `pipeline.inference.target`.
-6. Add judge dimensions with concrete descriptions and rubrics.
-7. Run `assert-eval run --config <path>`.
+2. **Fastest path:** run `assert-eval init --model <litellm-model>` (or `assert-eval init --model <litellm-model> --describe "..."`) to create a config interactively with an LLM assistant. Use `--from <existing_config>` to edit/extend an existing config.
+3. **Manual path:** Create or adapt an eval spec in `behavior.description` inside a YAML config.
+4. Add `context` describing the agent, tools, users, and constraints.
+5. Add `dimensions` only when systematic variation matters.
+6. Configure the target in `pipeline.inference.target`.
+7. Add judge dimensions with concrete descriptions and rubrics.
+8. Run `assert-eval run --config <path>`.
 
 ### Debug a failure
 

README.md

@@ -78,6 +78,22 @@ What the quickstart does:
 
 Start with the full walkthrough: [`docs/quickstart.md`](docs/quickstart.md).
 
+### Create your own config with `p2m init`
+
+Don't want to write YAML by hand? `p2m init` starts a conversational LLM assistant that asks about your agent, eval goals, and constraints, then proposes a complete config.
+
+`p2m init` needs an LLM to power the conversation. Pass `--model` with any [LiteLLM model string](https://docs.litellm.ai/docs/providers) and make sure the matching API key is set in your `.env` file (loaded by default) or environment:
+
+```bash
+p2m init --model azure/gpt-5.4
+# or skip the first question:
+p2m init --model azure/gpt-5.4 --describe "A customer-support chatbot with order-lookup and refund tools"
+# or edit/extend an existing config:
+p2m init --model azure/gpt-5.4 --from examples/travel_planner_langgraph/eval_config.yaml
+```
+
+See [`docs/reference/cli.md`](docs/reference/cli.md#design-a-config-interactively) for the full option reference.
+
 ## How it works
 
 ```text
@@ -148,6 +164,7 @@ Browse them with the CLI, the local viewer, or any JSONL tool. Nothing leaves yo
 - **Get started:** [`docs/quickstart.md`](docs/quickstart.md), [`docs/concepts.md`](docs/concepts.md)
 - **Targets:** [`docs/targets/`](docs/targets/) (overview), [`docs/targets/callable.md`](docs/targets/callable.md) (any agent), [`docs/targets/model-and-tools.md`](docs/targets/model-and-tools.md)
 - **Authoring:** [`docs/writing-eval-specs.md`](docs/writing-eval-specs.md), [`docs/reading-results.md`](docs/reading-results.md)
+- **Create a config:** `p2m init` — interactive config designer ([`docs/reference/cli.md`](docs/reference/cli.md#design-a-config-interactively))
 - **Reference:** [`docs/reference/cli.md`](docs/reference/cli.md), [`CONFIG_REFERENCE.md`](CONFIG_REFERENCE.md)
 - **AI assistants:** [`AGENTS.md`](AGENTS.md)
 - **Preview status:** [`docs/status-and-roadmap.md`](docs/status-and-roadmap.md)
@@ -158,6 +175,7 @@ Adaptive Eval is a customer preview / POC, not a GA service.
 
 Stable enough to try:
 
+- `p2m init` — conversational config designer
 - spec -> behavior categories -> test cases -> execute -> judge workflow
 - local artifact layout
 - `target.callable` with OTel trace capture (Phoenix/OpenInference for 33+ frameworks, or your own OTel SDK spans) — the recommended integration path

docs/quickstart.md

@@ -91,3 +91,14 @@ Read:
 - `metrics.json` for aggregate rates.
 
 For the agent graph itself, see [`docs/travel-planner-agent-flow.md`](travel-planner-agent-flow.md) or inspect [`examples/travel_planner_langgraph/agent.py`](../examples/travel_planner_langgraph/agent.py).
+
+## Next: create a config for your own agent
+
+Ready to evaluate your own agent? Use `p2m init` to design a config interactively instead of writing YAML by hand.
+Pass `--model` with any [LiteLLM model string](https://docs.litellm.ai/docs/providers) and make sure the matching API key is in your `.env`:
+
+```bash
+p2m init --model azure/gpt-5.4 --describe "My agent description"
+```
+
+See [`docs/reference/cli.md`](reference/cli.md#design-a-config-interactively) for the full option reference, or [`docs/writing-eval-specs.md`](writing-eval-specs.md) for guidance on authoring specs manually.

docs/reference/cli.md

@@ -2,6 +2,56 @@
 
 Adaptive Eval is CLI-first. All commands assume your virtualenv is activated (see the [README](../../README.md#quickstart-langgraph-travel-planner-any-agent-works-the-same-way) for setup).
 
+## Design a config interactively
+
+```bash
+assert-eval init
+```
+
+Starts a conversational LLM assistant that asks about your agent, eval goals, and constraints, then proposes a complete `eval_config.yaml`. Use this when you are setting up a new eval from scratch instead of editing YAML by hand.
+
+### Options
+
+| Option | Default | Description |
+|---|---|---|
+| `--output, -o` | `eval_config.yaml` | Output file path. |
+| `--describe` | — | One-line description of the system to evaluate (skips the initial question). |
+| `--from` | — | Seed from an existing config (edit/extend mode). |
+| `--behavior` | — | Use a built-in behavior preset name. |
+| `--judge-preset` | — | Use a built-in judge preset name. |
+| `--dimensions` | — | Hint dimension axes for the LLM to elaborate (e.g. `"user_role, language"`). |
+| `--model` | `azure/gpt-5.4-mini` | Model for the design agent (any [LiteLLM model string](https://docs.litellm.ai/docs/providers)). |
+| `--env-file` | `.env` | Dotenv file for credentials. |
+| `--non-interactive` | off | Single-shot mode (no conversation). |
+| `--max-turns` | `20` | Maximum conversation turns. |
+| `--force` | off | Overwrite existing output file. |
+| `--dry-run` | off | Print YAML to stdout without writing a file. |
+| `--no-color` | off | Disable colored terminal output. |
+
+### Examples
+
+```bash
+# Interactive session — the assistant will ask what you are evaluating
+assert-eval init --model azure/gpt-5.4
+
+# Skip the first question with a one-liner
+assert-eval init --model azure/gpt-5.4 --describe "A customer-support chatbot with order-lookup and refund tools"
+
+# Edit / extend an existing config
+assert-eval init --model azure/gpt-5.4 --from examples/travel_planner_langgraph/eval_config.yaml
+
+# Non-interactive: generate a config in one shot
+assert-eval init --model azure/gpt-5.4 --describe "RAG pipeline over internal docs" --non-interactive -o rag_eval.yaml
+
+# Preview the generated YAML without writing a file
+assert-eval init --model azure/gpt-5.4 --dry-run
+
+# Use an OpenAI model instead (requires OPENAI_API_KEY)
+assert-eval init --model gpt-4.1-mini
+```
+
+After `assert-eval init` writes the config, run the pipeline with `assert-eval run --config <path>`.
+
 ## Run a config
 
 ```powershell

docs/status-and-roadmap.md

@@ -4,6 +4,7 @@ Adaptive Eval is in customer preview. It is useful for design partners and struc
 
 ## Stable enough to try
 
+- `p2m init` — conversational config designer that creates eval configs interactively.
 - Spec-driven pipeline: spec -> behavior categories -> test cases -> execute -> judge.
 - Local artifact layout under `artifacts/results/`.
 - `target.callable` for any agent or multi-agent system with a Python entrypoint.

docs/writing-eval-specs.md

@@ -1,5 +1,12 @@
 # Writing Eval Specs
 
+> **Tip:** `p2m init --model <litellm-model>` can generate a complete eval config
+> interactively — including
+> the behavior spec, context, variations, and judge dimensions — by asking you
+> about your agent and eval goals. See the
+> [CLI reference](reference/cli.md#design-a-config-interactively). The guidance
+> below is for when you want to author or refine a spec by hand.
+
 An eval spec is the free-form text you put in `behavior.description` inside your
 YAML config. It describes the behavior_categories you want to measure. There is
 no separate markdown file — the spec, the context, the variations, and the

examples/README.md

@@ -21,6 +21,19 @@ assert-eval run --config examples\travel_planner_langgraph\eval_config.yaml
 assert-eval results status travel-planner-langgraph-v1 demo-1
 ```
 
+## Create your own config
+
+Use `p2m init` to design an eval config interactively instead of writing YAML by hand.
+Pass `--model` with any [LiteLLM model string](https://docs.litellm.ai/docs/providers) and make sure the matching API key is in your `.env`:
+
+```powershell
+p2m init --model azure/gpt-5.4-mini
+# or seed from an existing example:
+p2m init --model azure/gpt-5.4-mini --from examples\travel_planner_langgraph\eval_config.yaml
+```
+
+See the [CLI reference](../docs/reference/cli.md#design-a-config-interactively) for all options.
+
 ## Which example to start with
 
 | Goal | Example | Notes |

p2m/cli.py

@@ -510,6 +510,12 @@ def cli(ctx: click.Context, verbose: bool, quiet: bool, log_file: Path | None, o
     )
 
 
+# -- init (design an eval config with an LLM assistant) ---------------------
+from p2m.init._command import init  # noqa: E402
+
+cli.add_command(init)
+
+
 @cli.command(short_help="Run a pipeline from a YAML config")
 @click.option(
     "--config",

p2m/config.py

@@ -334,6 +334,35 @@ def _optional_str(value: Any, *, field_name: str) -> str | None:
     return stripped or None
 
 
+def _parse_preset_names(value: Any, *, field_name: str) -> list[str]:
+    """Accept a single preset name (str) or a list of preset names.
+
+    Returns a list of stripped, non-empty names preserving order. Duplicates are
+    removed (first occurrence wins) so callers can rely on deterministic
+    merge order. Returns [] when value is None or empty.
+    """
+    if value is None:
+        return []
+    if isinstance(value, str):
+        name = value.strip()
+        return [name] if name else []
+    if isinstance(value, list):
+        names: list[str] = []
+        seen: set[str] = set()
+        for index, item in enumerate(value):
+            if not isinstance(item, str):
+                raise ValueError(f"{field_name}[{index}] must be a string")
+            stripped = item.strip()
+            if not stripped:
+                raise ValueError(f"{field_name}[{index}] must not be empty")
+            if stripped in seen:
+                continue
+            seen.add(stripped)
+            names.append(stripped)
+        return names
+    raise ValueError(f"{field_name} must be a string or a list of strings")
+
+
 def _optional_float(value: Any, *, field_name: str) -> float | None:
     if value is None:
         return None
@@ -726,15 +755,23 @@ def parse_pipeline_config(raw: dict[str, Any]) -> PipelineConfig | None:
         if judge_enabled:
             model_raw = scorer_stage.get("model", default_model_raw)
             require(model_raw is not None, "pipeline.judge.model or default_model is required when judge is configured")
-            judge_preset_name = _optional_str(scorer_stage.get("preset"), field_name="pipeline.judge.preset")
+            judge_preset_names = _parse_preset_names(
+                scorer_stage.get("preset"),
+                field_name="pipeline.judge.preset",
+            )
             preset_dims: list[dict[str, Any]] = []
-            if judge_preset_name:
+            if judge_preset_names:
                 from p2m.library.loader import load_preset
-                preset = load_preset("judge_preset", judge_preset_name)
-                preset_dims = parse_judge_dimensions(
-                    preset.get("dimensions"),
-                    field_name=f"pipeline.judge.preset({judge_preset_name}).dimensions",
-                )
+                # Later presets override earlier ones on dimension-name conflict.
+                merged: dict[str, dict[str, Any]] = {}
+                for preset_name in judge_preset_names:
+                    preset = load_preset("judge_preset", preset_name)
+                    for dim in parse_judge_dimensions(
+                        preset.get("dimensions"),
+                        field_name=f"pipeline.judge.preset({preset_name}).dimensions",
+                    ):
+                        merged[dim["name"]] = dim
+                preset_dims = list(merged.values())
             inline_dims = parse_judge_dimensions(
                 scorer_stage.get("dimensions"),
                 field_name="pipeline.judge.dimensions",

p2m/init/__init__.py

@@ -0,0 +1 @@
+"""assert-eval init — conversational eval config designer."""