Skip to content

docs: refresh aao-verified.mdx + conformance.mdx + comply-test-controller.mdx for (Sandbox) framing #4380

@bokelley

Description

@bokelley

Context

#4379 decided: `Verified (Sandbox)` replaces `Verified (Live)` as the higher tier. Two-tier model. `comply_test_controller` reclassified to dev/staging-only.

The doc surface still describes the old (Live)/canonical-campaign framing. Three pages need a coordinated update.

Pages to refresh

`docs/building/verification/aao-verified.mdx` (primary)

  • Replace `Verified (Live)` with `Verified (Sandbox)` throughout.
  • Update the "What each axis certifies" table:
    • (Spec): unchanged in meaning, but reword to say "storyboards pass somewhere — could be a test deployment, could be local dev. Attests wire-format conformance, not production tolerance."
    • (Sandbox): new framing — "Tested against your real production endpoint with `account.sandbox: true`. Attests your prod stack correctly honors the sandbox flag across the full storyboard suite — schema-valid responses, lifecycle transitions, error envelopes, no real-world side effects."
  • Drop the "lights up in 3.1" / "canonical campaigns" / "designated compliance account" / "attestation_verifier scope" / "eight observability checks" content — superseded.
  • Drop the seven-section deep-dive on canonical-campaign mechanics.
  • Keep: badge mark visuals, JWT envelope, registry surfaces, eligibility (active membership + API-access tier), revocation/grace semantics.
  • Add: cross-reference to #3755 as the schema gate that makes (Sandbox) enforceable.

`docs/building/verification/conformance.mdx`

  • Section "Storyboard conformance vs. AAO Verified" — replace (Live) references with (Sandbox).
  • Drop the "(Live) is opt-in, observed continuously over a 7-14 day rolling window on a designated compliance account, and orthogonal to storyboard conformance" paragraph.
  • Add: (Sandbox) means storyboards pass against the seller's registered `agent_url` with `account.sandbox: true` — same suite, real prod endpoint.

`docs/building/by-layer/L3/comply-test-controller.mdx`

  • Reposition controller as dev/staging-only: useful for adopters' own integration testing, NOT a production-time concept, NOT required for AAO grading.
  • Frontmatter / TL;DR update: "Implement this in your dev/staging environment to support deterministic local testing. AAO Verified (Sandbox) does not require it — the AAO runner drives your real production endpoint with sandbox-flagged traffic, no controller involved."
  • Keep the operational reference (the existing `force_` / `simulate_` / `seed_*` documentation is still correct for the dev-mode use case).
  • Add a "Why this is dev-only" callout up top explaining the boundary.

Minor sweep

  • `docs/building/verification/storyboards-vs-scenarios.mdx` (just shipped, #4373) — references "Verified (Spec)" in passing; check that storyboard semantics still align.
  • `docs/building/verification/validate-your-agent.mdx` — mentions sandbox briefly; verify nothing claims controller-required.
  • `docs/building/verification/compliance-catalog.mdx` — check for (Live) references.

Acceptance criteria

  • No remaining references to `Verified (Live)` in the four core pages
  • `comply_test_controller` page leads with "dev/staging only" positioning
  • (Sandbox) framing consistent: real prod URL + `account.sandbox: true` + full storyboard suite
  • Cross-references to #3755 (schema gate), #4028 (live-mode-denial storyboard), #4226 (UNKNOWN_SCENARIO grading) noted as the implementation underpinning where relevant
  • Mintlify nav unchanged (no new pages, no page renames)

Why now

The implementation work in #3755, #4028, #4226 will land in the coming days. Docs need to lead, not lag — adopters reading the current pages and building against a deprecated model is the failure mode this issue is trying to prevent.

Out of scope

  • Specs / schemas / runners (covered by the three implementation issues above)
  • Migration guide for existing (Spec)-earner agents moving to (Sandbox) — that's a separate "rollout" doc once we know the cutover plan

Metadata

Metadata

Assignees

No one assigned

    Labels

    claude-triagedIssue has been triaged by the Claude Code triage routine. Remove to re-triage.compliance-suitedocumentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions