orb(relay): pull-mode forward errors are silently swallowed and unrecoverable — dedup guard blocks even manual redelivery

[JSONbored]

Summary

In pull relay mode — the mode edge-nl-01 runs — a transient D1 error while queueing a webhook for the self-host container causes the event to be lost with zero trace: no log, no metric, no retry row. The delivery-id dedup guard then rejects every recovery path, including an operator manually clicking "Redeliver" in GitHub. The review simply never happens.

Mechanism (verified at HEAD, 776c414)

1. The forward leg swallows every error. src/orb/webhook.ts:117-123:

export async function relayForward(env, args, fetchImpl = fetch): Promise<void> {
  try {
    const outcome = await forwardOrbEvent(env, args, fetchImpl);
    await persistRelayForwardOutcome(env, args, outcome);
  } catch {
    /* v8 ignore next -- fail-safe: a forward/persist error must never surface from the deferred task */
  }
}

No log. No counter. The receiver has already returned 202 to GitHub.

2. Only the push branch is protected. In forwardOrbEvent (src/orb/relay.ts:563-601), the try/catch that degrades to "failed" — and thereby persists to orb_relay_failures for the retry cron — wraps only the push path (:583-601). Outside it sit the enrollment SELECT (:563-568) and the entire pull branch:

if (row.relay_mode === "pull") {
  await enqueueRelayPending(env, {...});   // 4 D1 statements, unguarded
  return "queued";
}

Any D1 error there — contention, a near-cap write failure, a timeout — throws straight into the silent catch above.

3. The event is now in no recovery set. It exists in orb_webhook_events with status='received', but in neither orb_relay_pending nor orb_relay_failures. No cron scans for never-forwarded 'received' rows.

4. Redelivery is blocked. src/orb/webhook.ts:58-63:

if (existing && existing.status !== "error" && (existing.status === "processed" || existing.payloadHash === payloadHash)) {
  return c.json({ ok: true, deliveryId, eventName, status: "duplicate" }, 202);
}

The guard was copied from the review-app handler, where recording is processing. On the Orb, recording and forwarding are two separate legs — so a row that was recorded but never forwarded looks identical to one fully handled, and the re-forward never runs.

Why this matters here specifically

The hosted D1 has hit its 10 GB cap twice (2026-07-06 and 2026-07-26), and at the cap every write fails. That is precisely the condition that makes this path throw — so the silent-loss window is widest exactly when the system is already degraded. A lost pull_request event is eventually papered over by the open-PR sweep; a lost issue_comment command (@loopover review, pause, …) is silently ignored forever.

Requirements

1. No webhook event may be dropped without a durable record or, at minimum, an alertable log line.
2. A recorded-but-unforwarded event must be recoverable — by cron, by sweep, or by manual redelivery.
3. Re-forwarding must be safe to repeat (it already is: orb_relay_pending inserts ON CONFLICT DO NOTHING, and the container dedups by delivery-id).

Deliverables

• Wrap the pull branch and the enrollment SELECT so a throw returns "failed" rather than escaping — the event then flows into the existing orb_relay_failures retry machinery (isRelayFailureRetryTerminal already deletes the row once a retry lands "queued").
• Log and count in relayForward's catch. A silent catch {} on the last line of defence is the root cause; even if everything else is fixed, this should never be silent again.
• On a duplicate-suppressed redelivery of a forwardable event, re-run the forward instead of returning early — distinguishing "recorded" from "relayed" in the dedup decision.
• Add a bounded reconciliation scan for orb_webhook_events rows that are 'received', older than N minutes, and have no corresponding orb_relay_pending / orb_relay_failures / completed-forward record. Ride an existing cron tick.
• Log the per-installation cap eviction in enqueueRelayPending (src/orb/relay.ts:356-368), which currently deletes the oldest rows past RELAY_PENDING_MAX_PER_INSTALLATION with no log and without checking meta.changes. Its two siblings (pruneRelayPending :312-320, retryFailedRelays :507-515) already emit alertable *_dropped errors with samples — match them.
• Isolate retryFailedRelays per row (src/orb/relay.ts:490-538): it is documented as "never throws", but forwardOrbEvent can throw and the prune SELECT/DELETE at :494-503 are unguarded. A throw rejects the Promise.all, skips every later chunk in that tick, and — because finalizeRelayFailureRetryRow never ran — leaves the offending row's attempts unadvanced, so it stays first-in-batch and wedges the retry consumer's tail. Mirror the per-event isolation drainOrbRelayWithMonitor already uses.

Tests (must fail against current main)

• enqueueRelayPending throws ⇒ a durable failure row exists and an error is logged; the event is retried and eventually delivered.
• Enrollment SELECT throws ⇒ same.
• Redelivery of a recorded-but-unforwarded delivery-id ⇒ the forward runs (and is idempotent when repeated).
• The reconciliation scan finds a synthetic orphaned 'received' row and recovers it.
• Cap eviction emits a log with a sample.
• One poison row in retryFailedRelays does not prevent the rest of the batch from being retried, and its attempts advances.

Expected outcome

Every accepted webhook is either delivered, durably queued for retry, or loudly recorded as lost. "Silently vanished" stops being a reachable state.

[linear-code]

JSO-6535