feat(training-agent): implement provenance enforcement (closes #3777)#3792
Open
feat(training-agent): implement provenance enforcement (closes #3777)#3792
Conversation
Bring the reference training agent up to the spec landed in #3468 so the creative_sales_agent/provenance_enforcement storyboard passes against it. handleGetProducts: - Overlay comply_test_controller-seeded products onto the response so storyboard-seeded creative_policy fields (provenance_required, provenance_requirements, accepted_verifiers) round-trip through get_products. Previously only handleCreateMediaBuy saw seeded fixtures. - New backfillProductDefaults fills in spec-required Product fields (name, description, publisher_properties, format_ids, pricing_options, reporting_capabilities and its required sub-fields) for fixture-seeded products that historically carried only the fields create_media_buy validation needed. Closes the response-schema gap exposed once seeded products began round-tripping through get_products. handleSyncCreatives: - Aggregate creative_policy across session-seeded products and run structural enforcement before persisting each creative. Emits per- creative failures with action: 'failed' + errors[]: PROVENANCE_REQUIRED PROVENANCE_DIGITAL_SOURCE_TYPE_MISSING PROVENANCE_DISCLOSURE_MISSING PROVENANCE_EMBEDDED_MISSING PROVENANCE_VERIFIER_NOT_ACCEPTED — buyer's verify_agent.agent_url cross-checked (canonicalized) against the seller's accepted_verifiers allowlist before any outbound call; off-list URLs are rejected without contacting them, closing the buyer-controlled-URL trust gap. - PROVENANCE_CLAIM_CONTRADICTED (truth-of-claim, requires calling get_creative_features against an on-list verifier) is out of scope for this pass — the structural codes are sufficient to exercise the wire contract. Storyboard cleanup: - Remove creative_sales_agent/provenance_enforcement from KNOWN_FAILING_STORYBOARDS — passes 5/5 in both legacy and framework. - Bump min_clean_storyboards (53→65) and min_passing_steps (388→444 legacy, 401→462 framework) in .github/workflows/training-agent-storyboards.yml. - Update the scenario fixture with a unique name/description so brief- mode scoring places it at products[0], and switch per-creative error assertions to field_value paths (the storyboard runner's error_code validator only inspects top-level errors[] but sync_creatives carries per-item failures inside creatives[]). Local conformance: 65/65 storyboards clean in both modes; 444 passing steps legacy / 462 framework — matching the new floors. Refs: #3468, #3777. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ion, storyboard rigor) Addresses all expert-review items from #3792's three-pass read (code-reviewer, security-reviewer, nodejs-testing-expert). Catalog safety + symmetric backfill (code-review): - Restrict backfillProductDefaults to seeded-product IDs only by folding it into overlaySeededProducts. The previous integration mutated nested objects (format_ids[], reporting_capabilities) on every product including catalog ones; getCatalog() returns a shallow copy whose nested fields alias the cached singleton, so mutations would leak across requests once a partial catalog product appeared. - Both handleGetProducts and handleCreateMediaBuy now go through the same overlay path, so backfill is applied symmetrically. Closes the asymmetric-backfill divergence the reviewer flagged. - Rename backfillProductDefaults -> backfillTrainingProductDefaults to make the training-only intent obvious at every call site. Sanitization (security-review): - Add sanitizeForError helper that strips C0/C1 controls and caps length; apply to buyer-controlled strings (verify_agent.agent_url, creative_id) before interpolating into TaskError.message and error.field. Defense against log/transcript poisoning by attacker- shaped values. - Confirmed via review (and grep): no outbound HTTP touches the buyer- supplied verify_agent.agent_url anywhere in the enforcement path. The trust invariant from #3468 holds. Helper documentation: - enforceProvenancePolicy gains a cascade-order docstring (storyboard assertions on errors[0] rely on stable ordering). - aggregateCreativePolicy gains a comment explaining the deliberate asymmetry: requirement booleans are intersected (most-restrictive wins), accepted_verifiers are unioned (allowlist semantics). Storyboard rigor (testing-review): - Add reject_no_provenance phase (PROVENANCE_REQUIRED) — the cheapest, most likely buyer mistake had zero coverage. - Add reject_missing_digital_source_type phase (PROVENANCE_DIGITAL_SOURCE_TYPE_MISSING). Fixture now requires digital_source_type so the policy gate fires. - Tighten accept-phase assertion from field_present to field_value allowed_values: [created, updated]. field_present would have silently passed on action: failed. - Comment the errors[0] indexing as a stable contract tied to enforceProvenancePolicy's cascade order, with a TODO to switch to a field_contains predicate when the SDK ships one (adcp#3803). - Comment the brief-mode coupling for products[0] selection. Workflow comment correction: - True origin/main baseline was 64 storyboards / 439 legacy steps / 457 framework steps — the previous 53/388/401 floors had drifted. New floors: 65 / 446 / 464, lifting only the new creative_sales_agent/provenance_enforcement scenario (six phases). Comment corrected so the next reader doesn't reverse-engineer the baseline. Truth-of-claim follow-up: - Add static/compliance/source/protocols/media-buy/scenarios/ provenance_truth_of_claim.yaml as a single-phase skeleton. - Register it in KNOWN_FAILING_STORYBOARDS pointing at adcp#3802 (the follow-up issue tracking PROVENANCE_CLAIM_CONTRADICTED implementation in the training agent). Conformance rigor follow-up: - adcp#3803 tracks: (1) required-clean storyboard allowlist alongside the floors, (2) errors[*] field_contains predicate in the storyboard validator, (3) storyboard run in pre-push hook. Local conformance both modes: 65/65 clean; 446 passing steps legacy / 464 framework — exactly matching the new floors. Refs: #3468, #3777, #3792. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Issue #3803 proposes three conformance-rigor follow-ups surfaced in this PR's review: Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings the reference training agent up to the provenance spec landed in #3468 so
creative_sales_agent/provenance_enforcementpasses against it. Closes #3777.Wire contract enforced:
get_productsround-tripscreative_policy.{provenance_required, provenance_requirements, accepted_verifiers}fromcomply_test_controller-seeded products (previously onlycreate_media_buysaw fixtures)sync_creativesruns structural enforcement against the session's effectivecreative_policybefore persisting; emits per-creative failures withaction: 'failed'+errors[]carrying:PROVENANCE_REQUIREDPROVENANCE_DIGITAL_SOURCE_TYPE_MISSINGPROVENANCE_DISCLOSURE_MISSINGPROVENANCE_EMBEDDED_MISSINGPROVENANCE_VERIFIER_NOT_ACCEPTED— buyer'sverify_agent.agent_urlcross-checked (canonicalized) against the seller'saccepted_verifiersallowlist before any outbound call. Off-list URLs reject without contacting them, closing the buyer-controlled-URL trust gap that drove the seller-publishes / buyer-represents / seller-confirms pattern in feat(provenance): add embedded_provenance and watermarks to provenance schema #3468.PROVENANCE_CLAIM_CONTRADICTED(truth-of-claim — requires callingget_creative_featuresagainst an on-list verifier) is out of scope for this pass; the structural codes are sufficient to exercise the wire contract end to end.Other changes
backfillProductDefaultsfills in spec-requiredProductfields (name,description,publisher_properties,format_ids,pricing_options,reporting_capabilities+ sub-fields) for fixture-seeded products. Historical fixtures only carried whatcreate_media_buyvalidation needed; once seeded products begin round-tripping throughget_products, missing required fields fail response schema validation. Defensive backfill so fixture-driven products serialize as validProductobjects without forcing every fixture to repeat boilerplate.creative_sales_agent/provenance_enforcementfromKNOWN_FAILING_STORYBOARDSinserver/tests/manual/run-storyboards.ts(passes 5/5 in both legacy and framework dispatches).min_clean_storyboards(53→65) andmin_passing_steps(388→444 legacy, 401→462 framework) in.github/workflows/training-agent-storyboards.yml.products[0]. Per-creative error assertions switched tofield_valuepaths (the runner'serror_codevalidator only inspects top-levelerrors[]butsync_creativescarries per-item failures insidecreatives[]).Test plan
Local run, both modes:
npx tsx server/tests/manual/run-storyboards.ts(framework) — 65/65 clean, 462 passing steps, 0 failuresTRAINING_AGENT_USE_FRAMEWORK=0 npx tsx server/tests/manual/run-storyboards.ts(legacy) — 65/65 clean, 444 passing steps, 0 failuresnpm run typecheck— cleannpm run test:unit— 849/849 pass, 45 files🤖 Generated with Claude Code