Storyboards: distinguish schema-invalid vs business-rule negative paths

[bokelley]

Summary

The storyboard sample_request schema lint (landed in #2768) treats expect_error: true as a negative-test marker and skips schema validation on those steps. This covers two distinct negative-path categories that deserve separate treatment:

1. Schema-invalid negative path — the payload is intentionally malformed (missing required field, wrong type, etc.) to verify the agent's validation response. Examples: universal/error-compliance.yaml#missing_fields, universal/schema-validation.yaml#reversed_dates.
2. Business-rule negative path — the payload is schema-valid but violates a business rule or state-machine invariant. The agent rejects it, but for semantic reasons, not structural ones. Examples: invalid_transitions.yaml pause-canceled-buy, measurement_terms_rejected.yaml, governance_denied.yaml.

Current heuristic (isNegativeStep) lumps both categories together, which means shape drift on business-rule negative fixtures slips past the lint. A state-machine test that sends a schema-valid update_media_buy payload to a canceled buy would still be caught at runtime — but if someone introduces a typo that also makes the payload schema-invalid, the lint now silently accepts it because expect_error: true short-circuits validation.

Proposal

Introduce an explicit attribute to disambiguate:

steps:
  - id: pause_canceled_buy
    expect_error: true
    negative_path: business_rule    # schema-valid, agent rejects on business grounds
    # sample_request still schema-validated

  - id: reversed_dates
    expect_error: true
    negative_path: schema_invalid   # payload intentionally malformed
    # sample_request schema validation skipped

Defaults + migration:

• Default negative_path: schema_invalid (preserves current behavior — no regression on day 1)
• Authors of business-rule negative steps opt in by setting negative_path: business_rule
• Lint validates the sample_request when negative_path: business_rule
• Existing sample_request_skip_schema: true is a superset of schema_invalid — keep it as the explicit "skip validation, don't ask why" escape hatch

Why this matters

Flagged during the expert review of PR #2798 (from code-reviewer):

expect_error: true slightly over-broad for state-machine business-rule tests. Sites like state-machine.yaml 329/366/408, measurement_terms_rejected.yaml 92, governance_denied.yaml 166, and invalid_transitions.yaml 60/177/250 send schema-valid payloads and expect a business-rule error. Treating these as negative steps means shape drift on those specific steps would slip by the lint.

Today the cost is acceptable because business-rule fixtures have other lints covering them (scoping, branch-sets, context-entity). But as the suite grows, the latent coverage gap compounds.

Scope

• Update isNegativeStep in scripts/lint-storyboard-sample-request-schema.cjs to branch on negative_path
• Extend universal/storyboard-schema.yaml to document the new attribute
• Audit existing expect_error: true steps: tag them schema_invalid or business_rule as appropriate
• Lint enforces sample_request validation for business_rule steps (expect ~15-20 new assertions to fire on existing fixtures, most resolvable)
• Tests in tests/lint-storyboard-sample-request-schema.test.cjs expand to cover both code paths

Prior work

• feat(storyboards): lint sample_request against schema_ref with ratchet allowlist #2768 introduced the lint + isNegativeStep heuristic
• fix(storyboards): resolve 16-fixture actionable drift bundle (adcp#2763) #2798 landed expect_error: true as the canonical negative-test marker
• This issue tightens that heuristic without regressing existing behavior

[bokelley]

Triage

Outcome: Execute — draft PR opened

PR: #3226

What was built: negative_path attribute on storyboard steps. isNegativeStep() now branches on negative_path: business_rule (validate sample_request) vs schema_invalid (skip, the default — preserves current behavior). All 32 expect_error: true steps in the corpus are tagged (26 business_rule, 6 schema_invalid). The business_rule validation pass exposed fixture drift that is fixed in the same PR: 6 missing idempotency_key fields, a signal destination missing type: "platform", and pre-existing content-standards policy → policies array drift from #3147.

Classification of 32 steps:

• schema_invalid (6): idempotency/missing_key, deterministic-testing/unknown_scenario + missing_params, security/probe_unauth + probe_invalid_api_key + probe_invalid_oauth_token
• business_rule (26): all remaining (state-machine terminal enforcement, governance denial scenarios, invalid transitions, measurement terms, error-compliance date/version/structure, schema-validation temporal, brand/brand-rights identity + rights, signal governance)

Generated by Claude Code — triage session https://claude.ai/code/session_019dMkCmDw5ro7hzMQ6iTkKQ

---

Generated by Claude Code