feat(adagents): publisher_domains compact form, revoked_publisher_domains, streaming fetch caps (closes #729)

[bokelley]

Summary

Implements the full scope of issue #729 (adcp#4504 — publisher_domains[] compact form on publisher_properties selectors) plus the scope-expansion item from the issue thread (top-level revoked_publisher_domains[] on adagents.json with a Reason enum), and rewrites the outbound fetch path with two-tier streaming size caps and an If-None-Match / If-Modified-Since cache.

Built in two commits:

1. 5160ab0b — feature implementation (schemas, generated types, validator, resolver, fetch rewrite, 24 new tests + 122 ported).
2. fcb967c1 — review fixups after running the diff through three expert subagents (protocol, security, code review).

What's in scope

Schema (cached, hand-patched for 3.0):

• publisher-property-selector.json gains optional publisher_domains[] (XOR with publisher_domain, by_id selectors excluded).
• adagents.json gains top-level revoked_publisher_domains[] with Reason enum.

The cached schemas will be overwritten on the next make regenerate-schemas once upstream 3.0.10+ ships and ADCP_VERSION is bumped — the runtime / test changes are independent of the cached schema text.

Runtime validation (validation/legacy.py):

• validate_publisher_properties_item enforces both XORs (selector and publisher_domain).
• validate_revoked_publisher_domain_entry enforces shape + Reason enum.
• validate_adagents plumbing fixed (was keyed off agents instead of authorized_agents — pre-existing typo that made the new XOR validator dead at the integration level until this PR).

Resolver (adagents.py):

• _fanout_publisher_properties expands compact entries one-per-domain, preserving order across mixed compact+expanded arrays.
• filter_revoked_selectors + _get_revoked_publisher_domains apply revocation precedence to both publisher_properties[] selectors and top-level properties[].

Fetch path (adagents.py):

• Rewritten onto httpx.AsyncClient.stream with two-tier caps (MAX_POINTER_BYTES=5MiB first hop, MAX_AUTHORITATIVE_BYTES=20MiB second hop).
• Content-Length pre-check + per-chunk running-total guard.
• follow_redirects=False so HTTP 30x cannot bypass the SSRF gate that protects authoritative_location (the only sanctioned cross-host delegation path).
• _validate_publisher_domain now calls the same SSRF gate as _validate_redirect_url (factored into _check_safe_host), so first-hop IP literals and internal hostnames (localhost, .local, .internal, metadata.google.internal, RFC 1918, link-local, multicast, reserved) are rejected.
• New AdagentsCacheEntry / AdagentsFetchResult types + fetch_adagents_with_cache send If-None-Match / If-Modified-Since; 304 → cache-lifetime refresh. Cache validators truncated above 256 bytes before persisting.

Expert review — closed in this PR

Severity	Finding	Resolution
Security · Critical	follow_redirects=True bypasses SSRF gate on HTTP 30x	Set False; only authoritative_location follows.
Security · Critical	First-hop publisher domain has no IP-literal gate	_check_safe_host called from _validate_publisher_domain.
Security · High	Etag/Last-Modified replayed unbounded	256-byte cap on persisted validators.
Security · Medium	except Exception + unbounded body in error	except json.JSONDecodeError, message truncated to 200 chars.
Protocol · Blocker	validate_adagents walked agents not authorized_agents	Fixed; integration validator now actually runs.
Code · Must-fix	Dead "304/" token in redirect-hop error	Removed.
Code · Must-fix	Dead is_redirect ternary in not_modified return	Removed; only first-hop 304 is reachable.
Code · Must-fix	Misleading "original retained" comment on _fanout	Comment fixed.
Code · Should-fix	_REVOCATION_REASONS drift risk vs generated enum	Drift-detection unit test added.
Code · Should-fix	Streaming-cap test passed even if running-total guard deleted	Multi-chunk test added.
Code · Should-fix	No mixed-compact-and-expanded test	Added.
Code · Should-fix	No last_modified-only cache test	Added.

Deferred follow-ups (tracked, not blocking)

• DNS pinning to harden against rebinding (security H1) — needs a custom httpx transport.
• Pydantic model_validator for selector XOR — datamodel-codegen can't emit allOf[not[required[both]]] + anyOf[required[either]], so the typed Pydantic surface is laxer than the JSON Schema. The runtime validator covers the dict-parsing path; typed consumers will need an explicit gate.
• Revocation filtering on the inline_properties branch — filter_revoked_selectors covers publisher_properties[] and the top-level properties[]; the inline path is currently unfiltered. Worth confirming spec intent before adding.
• AdagentsCacheEntry.body vs AdagentsFetchResult.data field-name inconsistency — defer to a follow-up rename to avoid bouncing the public API in this PR.

Test plan

• ruff check src/ tests/test_adagents.py — passes
• mypy src/adcp/ — 807 source files, no issues
• pytest tests/ — 4,821 passed, 34 skipped, 1 xfailed (no failures, no flakes)
• 146 adagents tests pass: 122 ported onto the streaming mock helper, 24 new across four classes (compact form, revoked domains, fetch-with-cache, size caps)
• CI green across Python 3.10–3.13

Notes for reviewers

• per-authorized_agents[] last_updated is intentionally omitted — the merged upstream spec PR does not carry that field, confirmed against the latest upstream schema. The original issue thread discussed it; the spec community removed it.
• The new public exports in adcp.__init__.py: AdagentsCacheEntry, AdagentsFetchResult, fetch_adagents_with_cache, filter_revoked_selectors. Public-API snapshot updated.

🤖 Generated with Claude Code