feat(verification): scaffold spec-vs-implementation matrix v0

[jakepresent]

Scaffolds docs/verification/ with the v0 spec-vs-implementation matrix.

What this is

The matrix answers one question per row: does the system actually do what we claim it does? It feeds the on-every-release regression and triages into build / cut / push-back decisions when the answer is no.

See docs/verification/README.md for schema and status definitions.

What's in v0

• README.md - schema, status values, how to add a row, weekly review cadence
• matrix.md - 12 rows seeded from the existing tests/regression/risks/travel_planner_*.md failure-mode lists, reversed into capability claims (7 quality + 5 safety)
• matrix.json - same data, machine-readable, for the regression harness to consume

Status

This is a draft PR intentionally. v0 is structural - the schema and seed rows are in, but:

• Spec IDs (TP-Q-001 etc.) are placeholders pending the canonical scheme from @changliu2
• All rows start in gap-build because no regression has run yet
• evidence column is empty across the board - populates as the harness lands and PRs reference rows
• Target dates assume the May 8 / May 15 / May 22 ladder from the Verification pillar task list

Followups (not in this PR)

• Re-key spec IDs to canonical scheme when locked
• First population pass against canonical specs (target May 8)
• Wire the regression harness to consume matrix.json (target May 8 PR open)
• Lock weekly triage cadence with @changliu2 (target May 8)

Part of the Verification pillar deliverables.

[changliu2]

@jakepresent — suggestion for 1.3 Target shapes: split AE-TGT-OTEL into two rows so the matrix actually pins the two distinct customer integration patterns we ship demos for. Source of truth for the framework list is examples/phoenix_auto_trace/.

spec_id	claim	status	evidence	owner	target_date
AE-TGT-OTEL-PHX	Phoenix/OpenInference auto-instrumentation (from phoenix.otel import register; register(auto_instrument=True)) feeds OTel traces to the judge for the 14 Azure-routable supported frameworks shipped as runnable demos: autogen, crewai, dspy, haystack, instructor, langchain, langgraph, litellm, llamaindex, openai, openai_agents, openai_router, pydantic_ai, smolagents (P0; LangGraph as the verified anchor)	gap-build	examples/phoenix_auto_trace/travel_*.py + eval_*.yaml; CI evidence will land via #28 once that merges	jakepresent	2026-05-22
AE-TGT-OTEL-CUSTOM	Custom OTel SDK instrumentation — manual tracer.start_as_current_span(...) spans following OpenInference semantic conventions — is captured by the same target.trace block; verified end-to-end against the multi-agent NeurOSan-style example (P0)	gap-build	examples/travel_planner_neurosan/agent.py + eval_config.yaml	jakepresent	2026-05-22

The remaining 7 demos in phoenix_auto_trace/ (anthropic, bedrock, google_adk, google_genai, groq, mistralai, portkey) target non-Azure providers — fine to leave out of the P0 PHX row since the eval credentials we standardize on are Azure. Our 3P budget request is still pending. They could become a P2 row if/when we want to certify multi-provider OTel ingest.

One implementation caveat to know about while drafting these rows: target.trace.backend is currently a free-form string with no enum validation (p2m/core/config_model.py:94), and the runtime treats every value identically — OTelTracedSession labels itself "trace_backend": "otel" regardless of YAML (p2m/core/otel_session.py:99-105). So both demos express as trace.backend: phoenix in YAML today; the PHX/CUSTOM distinction lives in customer code (auto-instrument vs hand-rolled spans), not the config value. Worth wording the row claims around the integration pattern, not the backend string, so the spec doesn't drift if we later validate backend as an enum.

[changliu2]

@jakepresent — yes, lock in the current scheme as canonical. The structure you picked is already stable on the right axis (architectural subsystems, not spec-doc section numbers), so re-keying against a hypothetical docs/adaptive-eval-spec.md later would actually be a regression — section numbers churn, subsystems don't.

Canonical ID convention (please add this to docs/verification/README.md, e.g. as a ## ID convention section):

Framework rows — AE-{SUBSYSTEM}-{IDENTIFIER}

• AE-PIPE-S{1..5} — pipeline stage claims (S1=Policy, S2=Seeds, S3=Rollout, S4=Judge, S5=Metrics)
• AE-PIPE-{TOPIC} — non-stage pipeline claims (e.g. AE-PIPE-ART for artifact layout)
• AE-CLI-{COMMAND} — CLI surface (e.g. AE-CLI-RUN, AE-CLI-FORCE, AE-CLI-RESULTS)
• AE-TGT-{SHAPE} — target shapes (e.g. AE-TGT-MODEL, AE-TGT-CALLABLE, AE-TGT-HTTP, AE-TGT-OTEL-PHX, AE-TGT-OTEL-CUSTOM)
• AE-X-{TOPIC} — cross-cutting concerns (e.g. AE-X-LITELLM)

Scenario rows — {SCENARIO}-{CATEGORY}-{NNN}

• {SCENARIO} — short scenario prefix (TP = travel planner; future scenarios pick their own 2–3 letter prefix and register it in README.md)
• {CATEGORY} — single-letter category (Q = quality, S = safety; add letters as new categories appear)
• {NNN} — zero-padded sequential number, never recycled (deleted rows stay deleted; new rows get the next number)

Stability rules:

• IDs survive row deletion (gap-cut) — never recycled to new rows.
• IDs are intentionally not anchored to spec-doc section numbers. docs/adaptive-eval-spec.md doesn't exist yet, and even when it lands, section numbers will renumber more often than subsystem boundaries shift.
• When you rename a row's claim text, keep the same ID. The ID identifies the verification slot, not its current wording.

Cleanup that goes with this (please bundle into the same PR if it's quick, or follow-up otherwise):

• Drop the "placeholder pending canonical schemes from @changliu2" caveats from matrix.md (top), docs/verification/README.md (final paragraph), and the PR body.
• Drop the docs/adaptive-eval-spec.md reference at the top of matrix.md — that doc doesn't exist; it would be a forward-looking link without a target. Replace with: "Framework rows seeded from the implementation surface in p2m/ (pipeline stages, CLI commands, target shapes); scenario rows seeded from tests/regression/risks/travel_planner_*.md."

[jakepresent]

Closing as stale after the release/rename work. The verification-matrix slice was useful planning context, but it is no longer current against the public ASSERT tree. We can reopen a fresh scoped PR if the verification matrix becomes active again.