Proposal: CatalogSync client in @adcp/client for v3.1 beta (mirrors RegistrySync)

[bokelley]

Companion to the v3.1 catalog-sync cluster: #4761 (catalog_version conditional fetch), #4762 (get_signals wholesale), #4763 (per-agent change-feed). This issue asks for the SDK abstraction that makes those three protocol additions usable from a single call site, in time for the v3.1 beta SDK release.

Problem

The v3.1 catalog-sync cluster gives consumers (storefronts, federated marketplaces, registries) three new protocol primitives — wholesale signals enumeration, ETag-style conditional fetch, and a per-agent change feed. Used together, they let a consumer maintain a near-real-time priced mirror of any compliant agent's catalog. Used individually, each adds value.

But every consumer that adopts these will need to write the same code:

• Probe get_adcp_capabilities to discover which subset of the cluster the agent implements.
• Bootstrap via get_products(buying_mode: \"wholesale\") / get_signals(discovery_mode: \"wholesale\") (where supported).
• Cache catalog_version per (agent, account, filters) tuple; send if_catalog_version on subsequent calls; skip on unchanged: true.
• Subscribe to GET /catalog/events (where supported); apply events incrementally; re-bootstrap on catalog.bulk_change or RETENTION_EXPIRED.
• Register and verify HMAC webhooks (where supported); fall back to polling when subscriptions fail.
• Gracefully degrade for v3.0 agents that don't implement any of the above.

Each consumer rebuilding this from scratch is brittle (capability-detection bugs), inconsistent (different fallback strategies), and slow to adopt new spec features (every consumer must update independently). The @adcp/client SDK is where this abstraction belongs.

The precedent exists: specs/registry-change-feed.md already specs RegistrySync for the analogous problem at the registry level — bootstrap + poll + in-memory index + event emitter, with idiomatic clients in TypeScript, Python, and Go. CatalogSync is the same pattern applied to per-agent inventory (products and signals). #4763 already lists CatalogSync as Phase 2 of the change-feed proposal; this issue formalizes the API and asks for it in the v3.1 beta SDK rather than as a follow-on.

Goals

1. Consumers integrate catalog mirroring in one line — const catalog = new CatalogSync({ agentUrl, auth }), then await catalog.start(). No protocol-version branching in consumer code.
2. v3.0 agents work transparently — capability detection picks the best available strategy (manual-refresh-only, if the agent supports nothing beyond wholesale).
3. The SDK handles protocol mechanics (capability probing, conditional fetch, pagination, cursor management, retry/backoff, webhook HMAC, retention-expiry recovery). Consumers handle product policy (poll cadence, manual vs scheduled), persistence, and UX.
4. Ships in the v3.1 beta SDK release, not v3.1 GA. Beta consumers get the full integration shape from day one and can shake out the SDK during beta rather than retrofitting later.
5. TypeScript first (matches RegistrySync); Python and Go follow on the same release cadence as RegistrySync ports.

Proposed API

import { CatalogSync } from '@adcp/client';

const catalog = new CatalogSync({
  agentUrl: 'https://salesagent.example.com',
  auth: { /* per-agent credentials */ },
  // Optional product-policy knobs
  pollIntervalMs: 30_000,     // for change-feed mode; default 30s
  probeIntervalMs: 600_000,   // for catalog_version-only mode; default 10m
  webhooksEnabled: false,     // opt-in webhook registration; default false
  webhookEndpoint: 'https://consumer.example.com/hooks/catalog',
  webhookSecret: '...',       // HMAC secret if webhooks enabled
  persistCursor: true,        // persist change-feed cursor to disk for restart resumption
  cursorPath: '.adcp/catalog-cursor',
  onError: (err) => log(err),
});

await catalog.start();   // probes capabilities, picks strategy, bootstraps

// Unified read API — same shape regardless of mode
const products = catalog.products.list();
const signals  = catalog.signals.list();
const product  = catalog.products.get('prod_premium_ctv_us');
const filtered = catalog.products.search({ format_ids: ['display_300x250'] });

// Event stream — fires for change-feed events AND polled diffs in lower modes
catalog.on('product.created', (e) => { ... });
catalog.on('product.updated', (e) => { ... });
catalog.on('product.priced',  (e) => { ... });
catalog.on('product.removed', (e) => { ... });
catalog.on('signal.created',  (e) => { ... });
catalog.on('signal.updated',  (e) => { ... });
catalog.on('signal.priced',   (e) => { ... });
catalog.on('signal.removed',  (e) => { ... });
catalog.on('catalog.bulk_change', () => { ... });

// Manual force-sync — works in every mode
await catalog.refresh();

// Mode introspection — for consumer UI
catalog.mode;            // 'manual' | 'auto-poll' | 'live'
catalog.capabilities;    // resolved capability vector
catalog.lastSyncedAt;    // Date
catalog.lastEventAt;     // Date | undefined (live mode only)

// Graceful shutdown
await catalog.stop();

Mode selection (capability vector → strategy)

The SDK probes get_adcp_capabilities on start() and picks the highest-capability strategy the agent supports:

Capability profile	catalog.mode	Behavior
catalog_change_feed.supported === true	'live'	Wholesale bootstrap → poll GET /catalog/events. Optional webhook subscription if webhooksEnabled and webhooks_supported. on(...) fires per event.
catalog_version returned but no feed	'auto-poll'	Wholesale bootstrap → schedule if_catalog_version probes at probeIntervalMs. On version change: re-fetch and diff. on(...) fires per detected change.
Neither — v3.0 agent	'manual'	Wholesale bootstrap on start(). No background activity. refresh() triggers re-fetch. on(...) fires per detected change after refresh().

For mixed capabilities (e.g., agent supports products feed but not signals wholesale): per-entity-type mode resolution. catalog.products.mode === 'live' and catalog.signals.mode === 'manual' are independent. The top-level catalog.mode is the lowest of the two.

For v3.0 signals agents specifically: catalog.signals is empty by default — there's no protocol-conformant way to enumerate. The SDK surfaces this via a catalog.signals.queryable === false flag so consumer UI can render honestly ("queryable per-brief, not browsable").

What the SDK explicitly does NOT do

So the boundary is clear:

• No persistence beyond cursor + in-memory replica. Consumers needing durable catalog tables (multi-process, multi-host) consume on(...) events and write to their own storage.
• No product-policy decisions. Poll cadence, manual vs scheduled refresh, opt-in/opt-out of webhook registration — all exposed as knobs with sensible defaults; the consumer chooses values.
• No UX. The SDK exposes mode and capabilities; consumers render them.
• No auth or tenancy. The consumer supplies credentials and isolates per-tenant if needed.

This is the same boundary RegistrySync draws today.

Implementation notes

• TypeScript implementation in @adcp/client/src/catalog-sync.ts, alongside registry-sync.ts.
• Shared internals where possible: cursor persistence, HMAC verification, retry/backoff, webhook subscription lifecycle.
• In-memory indexes by product_id, signal_agent_segment_id, filterable secondary indexes (format_ids, channels, data_provider).
• Event emitter via the same pattern as RegistrySync (typed on(eventType, handler)).
• Cursor persistence optional (same default as RegistrySync: on for long-running processes, off for serverless).
• Python and Go ports follow the same API shape, on the same release cadence as RegistrySync ports.

What lands in v3.1 beta vs follow-up

v3.1 beta SDK (this proposal):

• TypeScript CatalogSync with manual, auto-poll, live modes.
• Capability detection, wholesale bootstrap, conditional fetch, change-feed polling.
• Read API (products.list/get/search, signals.list/get/search).
• Event emitter for all product.* / signal.* / catalog.bulk_change events.

v3.1 GA or follow-up (acceptable to defer):

• Webhook subscription lifecycle (HMAC, retries, suspension).
• Python and Go ports.
• Advanced query API beyond simple filters.

Webhook deferral is fine because consumers can stay in pure-polling live mode without webhooks and get acceptable latency (30–60s). Webhooks are a latency optimization, not a correctness primitive.

Open questions

1. Per-source instance vs multi-source manager. RegistrySync is a singleton (one registry). CatalogSync is naturally per-agent. Should the SDK also ship a MultiCatalogSync for consumers syncing N agents? Recommendation: defer. Consumers can manage their own Map<agentUrl, CatalogSync>; a multi-source wrapper is sugar that can land later without breaking the per-agent API.
2. Reconciliation on refresh(). Should refresh() fire diff events (product.priced, etc.) for changes detected between cached and fresh state, or only update the in-memory replica silently? Recommendation: fire events. Consumers using events as a write-trigger to their own storage expect parity between live mode and manual refresh.
3. Capability re-probing. Agents upgrade from v3.0 to v3.1 over time. Should the SDK re-probe get_adcp_capabilities periodically (e.g., daily) so a manual-mode instance auto-upgrades to auto-poll or live when the agent adds support? Recommendation: yes, configurable interval, default 24h. Consumer UI then shows the upgrade automatically.

Why ask now

The v3.1 beta cut is imminent. SDK consumers building integrations against the beta will write capability-detection and mode-switching code themselves if the SDK doesn't ship CatalogSync in the same release. Each of those integrations is a candidate for inconsistency or bugs that the protocol then has to support indefinitely. Shipping CatalogSync in beta — even with webhooks deferred — means every v3.1 consumer integrates the same way from day one.

We're committing to be one of those consumers (storefront / managed PSA), and our implementation collapses from ~6 steps (capability detection → manual sync → version cache → change-feed Temporal workflow → webhook lifecycle → SDK upgrade) to essentially two (configure SDK + persist its events) if the SDK ships this. Other v3.1 adopters will see the same multiplier.

Refs #4761, #4762, #4763. RegistrySync precedent at specs/registry-change-feed.md § Client SDK.

[bokelley]

Triage

Classification: Proposal (RFC)
Bucket(s): spec / protocol (no matching repo label found)
Status: deferred

What the experts said: No expert consultation — pre-classification defer. Title starts with "Proposal:" and the body explicitly positions this as Phase 2 of #4763, dependent on three open companion issues.

My take: CatalogSync is well-scoped and clearly the right SDK abstraction once the three protocol primitives land, but the spec work here can't proceed until #4761, #4762, and #4763 are accepted and merged. Deferring; will resurface via sweep when the blockers close.

Blocked-on: #4763, #4762, #4761 — resurfaces on merge

---

Triaged by Claude Code. Session: https://claude.ai/code/cse_01HudmGMiVCqRdk9R7xCxsuy

---

Generated by Claude Code