fix(metrics): exclude review and schedule dwell from outbound terminal latency

[jiashuoz]

Summary

e2a_outbound_terminal_latency_seconds measured from messages.created_at, so any wait the caller deliberately asked for was recorded as e2a latency. A HITL hold sits in pending_review until a human (or the TTL sweep) resolves it — ApproveAndAccept updates the row in place and never touches created_at — so an approved hold's sample carried the reviewer's entire dwell. Every hold approved more than 5 minutes after drafting was an automatic miss against the "terminal within 5 min of acceptance ≥ 99%" SLO in docs/observability.md. Scheduled sends have the same defect: any send scheduled further out than the window missed it by construction.

This is the same class of error the webhook eventual-delivery SLI already avoids by excluding endpoint_failure — the target measures e2a's own behavior, and a reviewer taking an hour is not e2a being slow.

It burned the hosted error budget for real: on 2026-08-03 a reviewer clearing a queue of held messages put a cluster of them past the 300s bucket and paged the prod outbound acceptance→terminal latency alert at 11.46% against a 1% budget.

The fix introduces submissionAnchor(acceptedAt, scheduledAt, reviewedAt) — the latest of the three, i.e. the instant the message became eligible to submit. Both terminal emission paths use it (the send worker inline, and the terminal reconciler for stranded rows) so a message settled by the sweep is measured identically to one settled inline. Taking the latest of the gates mirrors pastRetryHorizon's existing max(accept, scheduled) reasoning.

Operational risk

Observability-only. No change to send, retry, or terminal outcomes, and no API or client surface is touched.

• Zero/nil scheduled_at and reviewed_at are inert, so an ordinary send still anchors at created_at and the SLI keeps reporting real pipeline latency. That path has an explicit regression test.
• SendJob.AcceptedAt and the 72h retry horizon are deliberately not changed. That is F2 in docs/design/hitl-ttl-async-send.md and stays open; an addendum there records the split so the two anchors don't get conflated later.
• The reconciler's query gains two nullable columns (scheduled_at, reviewed_at) on a query already reading that row — no new scan, no index implication.
• Rollback is a plain revert; nothing persists.

Once released, the hosted outbound_terminal_latency alert query needs no change — it becomes correct as written. Expect the reported late fraction on that SLI to drop for any deployment using HITL or scheduled sends, because those samples were never e2a latency to begin with.

Test plan

• go build ./... clean
• go vet clean on internal/outboundsend, internal/identity, internal/agent
• go test -short ./internal/outboundsend/ ./internal/identity/ ./internal/agent/ ./internal/telemetry/ — all pass
• New tests fail without the fix (verified by reverting the anchor to j.AcceptedAt); the unchanged-ordinary-send test correctly still passes
• Review dwell excluded on the sent arm and the failure arm, so the two terminal paths can't disagree about what "acceptance" means
• Reconciler and worker anchors asserted equal for the same row
• submissionAnchor table test: gate ordering, a gate before accept is inert, zero accept with a hold
• After merge: confirm on staging that a HITL-approved send lands in a low bucket rather than one reflecting the hold duration

🤖 Generated with Claude Code

https://claude.ai/code/session_016zA4AGZSEVk9jdBfyCkL1v

---

Generated by Claude Code