spec(idempotency): rules 9 (concurrent retries) + 10 (downstream reconciliation); IDEMPOTENCY_IN_FLIGHT

[bokelley]

Summary

Closes the last two gaps the existing idempotency rules 1–8 left underspecified, both surfaced by a recent post on idempotency that walks through the failure modes a replay cache doesn't explain:

• Rule 9 — Concurrent retries / first-insert-wins. A second request carrying the same (authenticated_agent, account_id, idempotency_key) MAY arrive while the first is still executing. Sellers MUST resolve the race deterministically via INSERT … ON CONFLICT DO NOTHING on the scope tuple (with the canonical payload hash persisted at INSERT — not a sentinel — so same-key/different-payload mismatches are detectable before the first request completes), and MAY adopt one of two policies (behaving consistently): wait-and-replay (block the second request, return cached) or reject-and-redirect (return new IDEMPOTENCY_IN_FLIGHT with error.details.retry_after, bounded above by replay_ttl_seconds). In-flight row lifetime is bounded by the seller's declared per-task handler timeout.
• Rule 10 — Crossing service boundaries / downstream reconciliation. When a seller invokes a downstream system, "errors don't cache" (rule 3) is necessary but not sufficient. Sellers MUST adopt either write-claim-before-invoke or thread-buyer-key (where the threaded key is a per-downstream-provider HMAC derivative, not the buyer's raw key across trust boundaries, and not a single shared seller secret across all downstreams). Best-effort dedup on downstream response inspection is explicitly forbidden. Rule 10 is reviewer-graded (per parallel_dispatch_runner.reviewer_checks), not programmatically graded.
• New error code IDEMPOTENCY_IN_FLIGHT (held for 3.1 per wire-stability policy). Recovery: transient. Buyers MUST retry with the same key — minting a fresh one turns a safe retry into a double-execution race.

Storyboard coverage

static/compliance/source/universal/idempotency.yaml gains a concurrent_retry phase using two new cross-response check kinds (cross_response_count_distinct, cross_response_field_equal) that operate on the resolved response set across N parallel dispatches. The check kinds live in a separate cross_response_check_kinds enum (not authored_check_kinds) to prevent per-response-interpretation footguns; storyboard-schema.yaml documents the parallel_dispatch step block and the kinds' semantics. The runner contract is in static/compliance/source/test-kits/parallel-dispatch-runner.yaml; runners without parallel-dispatch support grade the phase not_applicable.

Wonderstruck (Python sales-agent) validation

Ran the new rules against the canonical Python sales-agent at wonderstruck.sales-agent.scope3.com:

• ✅ Rule 9 passes today via wait-and-replay. Two parallel create_media_buy calls with the same fresh key both returned media_buy_id: "4070698693", both ~5.5–6s — exactly one resource created. The new rule documents existing canonical behavior rather than inventing it.
• ✅ Rule 5 (key reuse with different payload) returns adcp_error.code: "IDEMPOTENCY_CONFLICT" correctly.
• ⚠️ Wonderstruck doesn't return replayed: true on the concurrent-replay response (rule 4 gap, not in scope here — tracked separately at Idempotency: missing replayed: true, lowercase validation_error, stripped replay payload, error.field in CONFLICT bokelley/salesagent#342).

Expert review

Ran through run-by-experts (4 reviewers in parallel) twice. Round 1 surfaced 1 convergent blocker + 4 single-reviewer must-fix items; all addressed in commit 404fec7322. Round 2 surfaced 0 convergent blockers + 3 single-reviewer must-fix items; all addressed in commit 4d98a0a198. Key fixes:

• Split cross_response_check_kinds out of authored_check_kinds (blocker: per-response-interpretation footgun)
• Bound in-flight row lifetime to declared handler timeout (locks-buyer-out footgun)
• Bound retry_after at replay_ttl_seconds (vacuous-wait footgun)
• Rule 9 hash-at-INSERT requirement (so same-key/different-payload is detectable mid-flight)
• Rule 10 per-downstream-provider HMAC derivation (not shared seller secret)
• Rule 10 strip buyer key from propagated downstream errors
• Rule 10 fail-closed on ambiguous downstream lookup
• Rule 10 raw-key clause narrowed to "across trust boundaries" (intra-tenant microservice forwarding permitted)
• Rule 10 explicitly flagged as reviewer-graded

Follow-up issues

• parallel_dispatch_runner: implement the storyboard runner contract for concurrent-retry tests (rule 9) adcp-client#1686 — parallel_dispatch_runner contract implementation (gates the new storyboard phase from running anywhere)
• @adcp/sdk middleware: return IDEMPOTENCY_IN_FLIGHT instead of SERVICE_UNAVAILABLE on the in-flight branch adcp-client#1687 — SDK middleware: swap SERVICE_UNAVAILABLE → IDEMPOTENCY_IN_FLIGHT on the in-flight branch (one-line wire-code update)
• capabilities.idempotency.in_flight_max_seconds: declare the in-flight bound so buyers can compute key lifetime #4406 — declare capabilities.idempotency.in_flight_max_seconds so buyers can compute retry budgets bounded by the seller's handler-timeout rather than the much-wider replay_ttl_seconds ceiling
• Idempotency: missing replayed: true, lowercase validation_error, stripped replay payload, error.field in CONFLICT bokelley/salesagent#342 — missing replayed: true envelope, lowercase validation_error, package-field drop on replay, field populated in IDEMPOTENCY_CONFLICT

Test plan

• node scripts/lint-error-codes.cjs, lint-error-code-drift.cjs, lint-storyboard-check-enum.cjs — clean
• node scripts/build-schemas.cjs, build-compliance.cjs — clean
• All 14 storyboard authoring lints — clean
• node --test tests/lint-storyboard-check-enum.test.cjs — 6/6 pass (after function rename)
• npm test — 35/36 pass; the one failure (TMP serve_window_sec) is pre-existing on main, not introduced by this PR
• Validated rule 9 behaviorally against Wonderstruck — first-insert-wins works, both dispatches converge on one media_buy_id

🤖 Generated with Claude Code