canonical-formats: implement hardened format_schema/platform_extensions fetch+cache helper

[bokelley]

Context

AdCP 3.1 defines a normative fetch contract for format_schema references and shares the same URI+digest shape with platform_extensions. The conformance/storyboard work is tracked in adcontextprotocol/adcp#4591 and adcontextprotocol/adcp#4623, but the reusable SDK helper belongs here so Python adopters do not hand-roll SSRF-prone schema fetching.

This is the Python SDK implementation tracker for the SDK half of adcontextprotocol/adcp#4623. The sibling TypeScript SDK tracker is adcontextprotocol/adcp-client#2063.

Required behavior

Add a reusable helper for resolving immutable uri@sha256:<digest> references used by format_schema and platform_extensions:

• HTTPS-only fetches.
• Public-network SSRF guard: reject loopback, link-local, RFC1918, CGNAT, cloud metadata, and RFC6761/special-use hosts.
• DNS rebinding defense by resolving and pinning, or re-validating the resolved address used for the request.
• Redirects disabled.
• 5 second default timeout, configurable by caller.
• 1 MiB streaming body cap, configurable by caller.
• Digest verification with sha256: plus 64 lowercase hex characters.
• Cache keyed by uri@digest, treating digest references as immutable.
• Negative-result handling that degrades gracefully for transient network/server failures but treats digest mismatch as a substitution-attack signal.
• Structured result/status that distinguishes resolved, unresolvable network failure, invalid schema, digest mismatch, and blocked unsafe URL.

For format_schema, also validate the fetched body as JSON Schema and enforce safe $ref behavior:

• Accept Draft 2019-09 and Draft-07 schema documents.
• $ref sandbox allows intra-document refs, same-origin refs, and trusted AgenticAdvertising.org mirror refs only.
• Reject off-origin refs, file://, http://, loopback/private/metadata refs, and excessive ref depth/count.
• Enforce schema-compile bounds such as ref depth <= 8, total refs <= 256, and a keyword/size bound to avoid compile-time DoS.

Acceptance criteria

• Public API is documented and usable by callers that need to resolve canonical format_schema and platform_extensions references.
• Tests cover digest match, digest mismatch, invalid JSON Schema, off-origin $ref, file:// $ref, http:// URL rejection, metadata/RFC1918 rejection, redirect rejection, timeout, and body-size cap.
• The helper is usable by adopters and any future Python storyboard harness without process-global mutable configuration.
• Error/result names are stable enough for conformance diagnostics and do not leak sensitive internal network details.

Related

• Phase 4 — platform_extensions / format_schema fetch+cache reference helper adcp#4623 — Phase 4 fetch/cache helper and conformance storyboards.
• Canonical-formats conformance storyboard track — gates 3.1 GA promotion rubric adcp#4591 — canonical-format conformance storyboard track.
• canonical-formats: implement hardened format_schema/platform_extensions fetch+cache helper adcp-client#2063 — TypeScript SDK sibling tracker.

[bokelley]

Triage

Classification: Feature request
Bucket(s): signing, validation, client (no matching repo labels exist — gap noted in run summary)
Status: ready-for-human
Milestone: v3.1

What the experts said:

• ad-tech-protocol-expert: Two concerns and a blocker. (1) Blocker — platform_extensions MUST follow the identical transport hardening path as format_schema; only the post-digest step differs (no JSON Schema validation for platform_extensions). A split path would undermine the weaker leg. (2) Concern — Issue proposes Draft-07 + 2019-09 but the normative spec (product-format-declaration.json) says Draft-07 + 2020-12; 2019-09 should be treated as invalid-schema. (3) Concern — digest format is ^sha256:[a-f0-9]{64}$ — lowercase only; implementation MUST reject uppercase hex, not silently normalize. Also: add a schema-compile-exceeded status code for keyword/size budget exhaustion; Cache-Control headers MUST NOT drive cache invalidation (immutability is asserted by the digest reference, not the HTTP header); negative results MUST NOT be cached keyed on uri alone.

• code-reviewer: One blocker plus four design concerns. (1) Blocker — jsonschema's RefResolver follows remote $ref URIs by default; a format_schema payload containing a malicious $ref would make an outbound SSRF call during schema compilation. Must use jsonschema 4.18+ referencing.Registry (closed, no remote resolution) or RefResolver with handlers={} and an empty store — schema_validator.py's existing file-rooted resolver MUST NOT be reused here. (2) Use bounded OrderedDict with LRU cap (≤256 entries) rather than an unbounded dict; document single-process scope explicitly. (3) Ship both fetch_canonical_schema() and async_fetch_canonical_schema() following the existing naming convention in signing/; sync-only callers (CLI, startup priming) cannot use async-only. (4) Trusted-mirror allowlist should be a frozenset[str] kwarg defaulting to {"schemas.adcontextprotocol.org"} — checked before the SSRF IP validation, not after. (5) Prefer separate uri: str, digest: str params + a parse_schema_ref(ref: str) -> tuple[str, str] splitter helper; the cache key is an implementation detail.

My take: This is a well-scoped, high-value addition — the existing signing infrastructure (ip_pinned_transport, idna, digest.py) provides most of the transport-layer building blocks. Two real blockers need implementation-level decisions before coding starts: the JSON Schema $ref→SSRF vector inside the validator, and the shared transport path for platform_extensions. Flagging for @bokelley to assign or authorize an implementer given the security surface.

Implementation brief (once owner is assigned):

• Module: src/adcp/canonical_formats/fetch.py; reuse build_async_ip_pinned_transport / build_ip_pinned_transport from src/adcp/signing/ip_pinned_transport.py — DNS rebinding defense is already built
• Digest: validate ^sha256:[0-9a-f]{64}$ strict regex (lowercase hex only — distinct from the base64 form in signing/digest.py); reject uppercase, no silent normalization
• API surface: fetch_canonical_schema(uri, digest, *, timeout=5.0, max_bytes=1_048_576, trusted_origins=frozenset({...})) + async twin; parse_schema_ref(ref) -> tuple[str, str] splitter
• Cache: bounded OrderedDict with 256-entry LRU cap; never invalidate on HTTP Cache-Control; never cache negative results by uri alone
• $ref sandbox: use closed referencing.Registry (jsonschema 4.18+) or RefResolver(base_uri="", referencing={}, handlers={}) — no remote resolution; apply depth ≤8 / count ≤256 / keyword/size limits before compile()
• Draft scope: Draft-07 + 2020-12 only; 2019-09 → invalid-schema
• Status codes (6): resolved, network-failure, invalid-schema, digest-mismatch, blocked-unsafe-url, schema-compile-exceeded
• Required checks: pytest tests/canonical_formats/ -v, mypy src/adcp/canonical_formats/, ruff check src/

---

Triaged by Claude Code. Session: https://claude.ai/code/cse_01AAhhfsqMBRJVeTS5abu41u

---

Generated by Claude Code

[bokelley]

Triage addendum — security-reviewer findings (arrived after initial comment):

Three additions to the implementation brief above:

1. $ref sandbox — scheme allowlist before resolver (blocker extension): jsonschema's RefResolver does not reject javascript: or data: scheme URIs. A schema with "$ref": "data:application/json,{...}" would resolve in-process, allowing attacker-supplied keywords in the compiled validator (compile-time DoS via deeply-nested not/if/then). The sandbox must reject any $ref whose scheme is not https (or the intra-document # fragment) before handing off to the resolver — the origin allowlist alone is not sufficient.

2. Multi-tenant result structure (new blocker): If ResolveResult (the cached value) carries any per-request fields — trace IDs, caller identity, request-scoped metadata — a cache hit for the same uri@digest from a second tenant would surface tenant A's context. The cached value MUST be strictly the parsed body + digest-verification outcome: no per-request fields. Error sentinels cached for digest-mismatch must likewise be sanitized of request context before storage.

3. Two implementation hygiene concerns: (a) Set trust_env=False on the httpx client — existing signing fetchers (jwks.py, revocation_fetcher.py) both do this to prevent HTTPS_PROXY from routing around the IP-pinned transport; missing it silently bypasses the SSRF + pin. (b) Reject http:// at the outermost entry point, before transport construction — scheme check before resolve_and_validate_host, not after.

The digest-mismatch terminal negative should be cached as a typed sentinel so repeat fetches for the same uri@digest fail immediately without a network round-trip (the mismatch is not transient).

---

Triaged by Claude Code. Session: https://claude.ai/code/cse_01AAhhfsqMBRJVeTS5abu41u

---

Generated by Claude Code