feat(webhook): PgWebhookDeliverySupervisor — Postgres-backed multi-worker delivery

[bokelley]

Closes #352

Summary

• src/adcp/webhook_supervisor_pg.py (new, ~800 LOC) — PgWebhookDeliverySupervisor, a Postgres-backed implementation of the WebhookDeliverySupervisor Protocol introduced in feat(decisioning): WebhookDeliverySupervisor + SQLAlchemy A2A stores example #348. Designed for multi-process/multi-worker deployments where the in-memory supervisor cannot provide durability.
• tests/test_webhook_supervisor_pg.py (new, ~650 LOC) — 28 unit tests using fully-mocked psycopg3 (no real database required).
• src/adcp/webhook_supervisor.py (patched) — Protocol send_mcp() was missing breaker_key and notification_type kwargs that existed on InMemoryWebhookDeliverySupervisor; fixed.
• src/adcp/webhooks.py (patched) — Re-exports PgWebhookDeliverySupervisor for discoverability (from adcp.webhooks import PgWebhookDeliverySupervisor).

Design highlights

Concern	Approach
Multi-worker job lease	FOR UPDATE SKIP LOCKED LIMIT 1 — connection held open during HTTP send; crash rolls back the lease
Circuit breaker	State stored in adcp_webhook_circuit_state; OPEN gates writes in send_mcp() without touching the queue
Half-open atomicity	INSERT … ON CONFLICT DO UPDATE … RETURNING — serializes concurrent success_count increments so exactly one worker triggers the CLOSED transition
Idempotency-key compliance	First attempt stores sent_body+idempotency_key in the queue row; retries call sender.resend(prev) to replay the exact same bytes
SQL injection	Table names validated with ASCII regex at construction time (identical to PgReplayStore pattern); all runtime values use parameterized queries
Optional dependency	Guarded by PG_AVAILABLE flag; uses the existing [pg] extra (psycopg[binary]>=3.1.0, psycopg-pool>=3.2.0)

Required DDL (included in create_schema())

CREATE TABLE IF NOT EXISTS adcp_webhook_circuit_state (
    breaker_key      TEXT PRIMARY KEY,
    state            TEXT NOT NULL DEFAULT 'closed',
    failure_count    INT  NOT NULL DEFAULT 0,
    success_count    INT  NOT NULL DEFAULT 0,
    last_failure_at  TIMESTAMPTZ
);

CREATE TABLE IF NOT EXISTS adcp_webhook_delivery_queue (
    id               BIGSERIAL PRIMARY KEY,
    url              TEXT        NOT NULL,
    task_id          TEXT        NOT NULL,
    status           TEXT        NOT NULL,
    task_type        TEXT,
    result           JSONB,
    token            TEXT,
    sequence_key     TEXT,
    breaker_key      TEXT,
    notification_type TEXT,
    attempt          INT         NOT NULL DEFAULT 0,
    max_attempts     INT         NOT NULL DEFAULT 3,
    sent_body        BYTEA,
    idempotency_key  TEXT,
    enqueued_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    process_after    TIMESTAMPTZ NOT NULL DEFAULT now(),
    status_col       TEXT        NOT NULL DEFAULT 'pending'
);
CREATE INDEX IF NOT EXISTS adcp_wdq_pending
    ON adcp_webhook_delivery_queue (process_after)
    WHERE status_col = 'pending';

CREATE TABLE IF NOT EXISTS adcp_webhook_delivery_log (
    id          BIGSERIAL   PRIMARY KEY,
    job_id      BIGINT,
    url         TEXT        NOT NULL,
    task_id     TEXT        NOT NULL,
    attempt     INT         NOT NULL,
    ok          BOOLEAN     NOT NULL,
    status_code INT,
    error       TEXT,
    logged_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);

What was tested

• ruff check — clean
• mypy src/adcp/ — clean
• pytest tests/test_webhook_supervisor_pg.py -v — 28/28 passed (fully mocked psycopg3, no DB)
• All 28 tests cover: construction validation, create_schema() DDL, circuit breaker gating, half-open timeout transition, FOR UPDATE SKIP LOCKED poll, success/failure/final-failure paths, resend-with-stored-body, sink error swallowing, log-write fault isolation, run_worker() lifecycle

Pre-PR expert review sign-offs

• code-reviewer (pre-PR): No blockers. Flagged one vacuous test (fetchone raise path unreachable in _log_attempt_via_conn) and one dead inner function — both fixed in the final commit.
• Core logic verified: FOR UPDATE SKIP LOCKED lease-holds-through-HTTP pattern, atomic UPSERT RETURNING for half-open, and safe-identifier validation all confirmed correct.

---

Triage-managed PR — opened by the Claude triage agent in response to issue #352. Implementation follows the PgReplayStore pattern established in src/adcp/signing/pg/replay_store.py.

---

Generated by Claude Code