feat: storyboard coverage and comply() scenario filtering#1985
Merged
feat: storyboard coverage and comply() scenario filtering#1985
Conversation
This was referenced Apr 8, 2026
bokelley
added a commit
that referenced
this pull request
Apr 8, 2026
Remove storyboard YAMLs (brand_rights, capability_discovery) and brand compliance track — these belong in #1985 which covers full storyboard coverage and comply() migration. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
bokelley
added a commit
that referenced
this pull request
Apr 8, 2026
* fix: storyboard validation failures against training agent - Add creative and account capability blocks to get_adcp_capabilities response so capability_discovery validation passes (#1990) - Increase training agent rate limit from 60 to 300 req/min to prevent cascading failures during bulk storyboard evaluation (#1991, #1992, #1994) - Add brand_rights and capability_discovery storyboard YAMLs (#1993, #1992) - Add brand compliance track to TRACK_SCENARIOS infrastructure (#1993) - Update storyboard test to accept core/ and brand/ schema_ref prefixes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address code review and security review findings - Update stale rate limit comment (60→300 req/min) in task TTL calculation - Add 'brand' to supported_protocols since training agent implements brand tools - Update test expectation for supported_protocols Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: training agent schema compliance for creative handlers - Add created_date and updated_date to list_creatives response (required by schema) - Add include_snapshot param to list_creatives tool schema and return snapshot_unavailable_reason when requested - Fix build_creative assets format: use object map with content field instead of array with html field (matches creative-manifest.json schema) - Add generative build mode (target_format_id only, no manifest/library) since training agent declares supports_generation: true - Fix preview_creative render format: use render_id, output_format, role, preview_url/preview_html per preview-render.json schema Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address code review and security review findings (round 2) - Fix preview_creative response: add response_type, preview_id, input fields per preview-creative-response.json schema - Fix preview_creative: reject invalid format_ids with INVALID_FORMAT error - Fix BuildCreativeArgs/PreviewCreativeArgs: type assets as Record (object map) matching creative-manifest.json schema, not Array - Cap target_format_ids at 50 to prevent response amplification - Remove non-schema synced_at field from list_creatives response Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: scope PR to training agent fixes only Remove storyboard YAMLs (brand_rights, capability_discovery) and brand compliance track — these belong in #1985 which covers full storyboard coverage and comply() migration. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: type training agent responses with @adcp/client types Import BuildCreativeResponse, ListCreativesResponse, PreviewCreativeResponse, and CreativeManifest from @adcp/client and use them as return types. The compiler now catches shape mismatches at build time: - Removed creative_id from CreativeManifest (not in schema) - Changed multi-format response from { results } to { creative_manifests } - buildHtmlAssets returns AdcpCreativeManifest['assets'] type Previously, handlers returned untyped object literals — the MCP SDK wraps them in content[0].text as opaque strings, so TypeScript couldn't validate the domain payload shape. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add five new storyboards and fix comply() performance for storyboard evaluations. Storyboard coverage: - capability_discovery: get_adcp_capabilities validation (#429) - campaign_governance_denied: hard denial with no escalation (#430) - campaign_governance_conditions: conditional approval flow (#430) - campaign_governance_delivery: delivery drift monitoring (#430) - creative_lifecycle: multi-format sync, list, build, preview (#431) Comply performance (#433): - extractScenariosFromStoryboard() extracts comply_scenario values from YAML - filterToKnownScenarios() validates scenario names against TRACK_SCENARIOS - Storyboard run endpoint now passes only referenced scenarios to comply(), reducing calls from 30+ to 2-5 per storyboard evaluation Track changes: - Add campaign_governance track with 5 scenarios - Add creative_lifecycle to creative track Closes #429, #430, #431, #433. Partial progress on #432. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Five new storyboards covering uncovered protocol domains: - social_platform: accounts, audiences, native creatives, events, financials - si_session: SI offering discovery, session lifecycle - brand_rights: identity, rights licensing, creative approval - property_governance: property list CRUD, delivery validation - content_standards: standards CRUD, calibration, delivery validation comply() migration from track-based to storyboard-based routing: - PLATFORM_STORYBOARDS maps each platform type to recommended storyboards - comply() accepts storyboards option (priority: scenarios > storyboards > tracks) - evaluate_agent_quality uses PLATFORM_STORYBOARDS when no explicit tracks - Heartbeat job uses storyboard routing when agent has a registered platform type - Compare endpoint now filters to storyboard scenarios Fixed comply_scenario annotations across all original storyboards: - Replaced phantom names (account_setup, governance_setup, media_buy_flow) with real TestScenario values (full_sales_flow, create_media_buy, etc.) - Added comply_scenario to all 7 storyboards that had none - 20/21 storyboards now have test coverage (brand_rights pending #1993) Validation against training agent filed as issues: - #1990: capability_discovery creative validation failure - #1991: creative_ad_server SSE/rate limit errors - #1992: 15 storyboards skip (agent tool discovery gap) - #1993: Brand protocol needs @adcp/client scenarios - #1994: Bulk validation needs backoff Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
bokelley
force-pushed
the
bokelley/client-storyboards
branch
from
fc5f3be to
b5ee24c
Compare
This was referenced Apr 8, 2026
The capability_discovery scenario fails with "Tools suggest unreported protocols: compliance (has: [comply_test_controller])" because the training agent exposes comply_test_controller but doesn't declare compliance in supported_protocols. Task retrieval failures (callToolStream over stateless HTTP) tracked separately in adcontextprotocol/adcp-client#442. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Picks up the serve() task store fix (adcp-client#443) and skip counting fix (adcp-client#441). Local validation shows 20/20 storyboards passing against the training agent. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…rsion) Keep storyboard routing + @adcp/client 4.22.1 from our branch. Add userAgent and outbound request logging from main. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
capability_discovery.yamlstoryboard coveringget_adcp_capabilities— the only task that had no storyboard coveragedenied,conditions,delivery) with a newcampaign_governancecompliance trackcreative_lifecycle.yamlcovering multi-format sync, library listing, build, and previewChanges
New storyboards (5):
capability_discovery.yaml— protocol-level capability introspectioncampaign_governance_denied.yaml— hard denial, no escalationcampaign_governance_conditions.yaml— conditional approval with binding conditionscampaign_governance_delivery.yaml— delivery monitoring with drift re-checkcreative_lifecycle.yaml— sync 3 formats, list/filter, build/previewComply performance fix:
extractScenariosFromStoryboard()extractscomply_scenariovalues from storyboard YAMLfilterToKnownScenarios()validates extracted names againstTRACK_SCENARIOScomply()Track changes:
campaign_governancetrack with 5 scenarioscreative_lifecycleto existingcreativetrackTest plan
protocol/refsextractScenariosFromStoryboardtested for dedup and empty casesCloses #429, #430, #431, #433. Partial progress on #432.
🤖 Generated with Claude Code