Three storyboard-conformance gaps surfaced during the #3792 review (training-agent provenance enforcement). Filed as a single tracking issue; can be split into PRs as each is picked up.
1. Required-clean allowlist alongside the floor
.github/workflows/training-agent-storyboards.yml gates on min_clean_storyboards and min_passing_steps only — counts. That catches regressions (count drops) but not rebalancing: a future change could break load-bearing scenario A while a new scenario B passes, and the count stays flat. Examples of load-bearing scenarios that should never silently fail: creative_sales_agent/provenance_enforcement (the wire contract from #3468), signed_requests (auth conformance), error_compliance (rejection vocabulary).
Proposal: add a required_clean_storyboards: <list of IDs> gate alongside the floors. Failure of any listed scenario fails CI regardless of total count. Don't pin all 65 — that's just KNOWN_FAILING_STORYBOARDS inverted and gets noisy. Pin only the 5–10 contracts whose conformance is wire-load-bearing.
2. errors[*].code contains predicate in storyboard validations
The runner's field_value validator (in @adcp/sdk testing/storyboard/validations.ts) supports positional path access (errors[0].code) and [*] wildcard parsing in parsePathWithWildcards / resolvePathAll, but doesn't expose a wildcard-matching field_value check. Storyboards are forced into positional assertions like creatives[0].errors[0].code, which is fragile if a future implementation emits multiple errors per item or reorders them.
Proposal: add check: field_contains (or extend field_value to use parsePathWithWildcards when the path includes [*]) so storyboards can write creatives[0].errors[*].code value: PROVENANCE_DISCLOSURE_MISSING and pass as long as the code appears anywhere in the array. Documented compromise for now: per-creative error storyboards write the cascade order the seller emits, and consumers rely on stable ordering. Worked example in creative_sales_agent/provenance_enforcement.yaml.
3. Storyboard run in pre-commit (or pre-push)
scripts/with-timeout.sh 60 npm run test:unit && npm run test:test-dynamic-imports && npm run test:callapi-state-change && npm run typecheck is the current pre-commit chain. The Storyboards CI job is the gate on every PR that touches the training agent or compliance source — but it doesn't run locally on commit. #3792 demonstrated this: storyboard regressions were caught only in CI on the second push.
Proposal: add npm run storyboards (legacy + framework) to a pre-push hook (not pre-commit — they take ~3 min). Or a documented npm run precheck script committers can opt into. Don't gate every commit; do gate every push.
Reference
Surfaced in #3792's review (testing-expert pass). #3792 ships the spec contract + structural enforcement; these three are the conformance-rigor follow-ups, not blocking the PR.
Three storyboard-conformance gaps surfaced during the #3792 review (training-agent provenance enforcement). Filed as a single tracking issue; can be split into PRs as each is picked up.
1. Required-clean allowlist alongside the floor
.github/workflows/training-agent-storyboards.ymlgates onmin_clean_storyboardsandmin_passing_stepsonly — counts. That catches regressions (count drops) but not rebalancing: a future change could break load-bearing scenario A while a new scenario B passes, and the count stays flat. Examples of load-bearing scenarios that should never silently fail:creative_sales_agent/provenance_enforcement(the wire contract from #3468),signed_requests(auth conformance),error_compliance(rejection vocabulary).Proposal: add a
required_clean_storyboards: <list of IDs>gate alongside the floors. Failure of any listed scenario fails CI regardless of total count. Don't pin all 65 — that's justKNOWN_FAILING_STORYBOARDSinverted and gets noisy. Pin only the 5–10 contracts whose conformance is wire-load-bearing.2.
errors[*].code containspredicate in storyboard validationsThe runner's
field_valuevalidator (in@adcp/sdktesting/storyboard/validations.ts) supports positional path access (errors[0].code) and[*]wildcard parsing inparsePathWithWildcards/resolvePathAll, but doesn't expose a wildcard-matchingfield_valuecheck. Storyboards are forced into positional assertions likecreatives[0].errors[0].code, which is fragile if a future implementation emits multiple errors per item or reorders them.Proposal: add
check: field_contains(or extendfield_valueto useparsePathWithWildcardswhen the path includes[*]) so storyboards can writecreatives[0].errors[*].code value: PROVENANCE_DISCLOSURE_MISSINGand pass as long as the code appears anywhere in the array. Documented compromise for now: per-creative error storyboards write the cascade order the seller emits, and consumers rely on stable ordering. Worked example increative_sales_agent/provenance_enforcement.yaml.3. Storyboard run in pre-commit (or pre-push)
scripts/with-timeout.sh 60 npm run test:unit && npm run test:test-dynamic-imports && npm run test:callapi-state-change && npm run typecheckis the current pre-commit chain. The Storyboards CI job is the gate on every PR that touches the training agent or compliance source — but it doesn't run locally on commit. #3792 demonstrated this: storyboard regressions were caught only in CI on the second push.Proposal: add
npm run storyboards(legacy + framework) to a pre-push hook (not pre-commit — they take ~3 min). Or a documentednpm run precheckscript committers can opt into. Don't gate every commit; do gate every push.Reference
Surfaced in #3792's review (testing-expert pass). #3792 ships the spec contract + structural enforcement; these three are the conformance-rigor follow-ups, not blocking the PR.