ZLT-3008: email send→deliver fan-out (DB lens primitives) + durable jobs overflow buffer

[agreenspan]

Reference implementation for ZLT-3008 — upgrading the template's email send path so the downstream Zealot/Tribe ports copy a correct example. The work landed as several composable pieces rather than the single "audience resolver" this description originally described: recipient targeting is now expressed through generic DB lens primitives instead of an email-specific resolveAudience/audience.ts (those were prototyped and then dropped — see commit history), and the send job was split into a planner + per-recipient delivery fan-out.

Note: this PR's scope grew past email. It also lands a generic jobs overflow buffer, the DB lens/hydrate primitives, email-template versioning, a CommunicationLog model, and an unsubscribe route. They're related (the email pipeline is the first consumer of each) but independently reviewable — the map below is by piece.

The pieces

1. Email send → deliver fan-out — apps/api/src/jobs/handlers/{sendEmail,deliverEmail}.ts

• sendEmail is now a planner: resolve entity + recipients via a lens, compute a sendKey + per-recipient idempotencyKey, upsert one CommunicationLog row per recipient, enqueue one deliverEmail each.
• deliverEmail is one recipient, one message: load the log row, settleTemplate (compose + render-error policy), suppression/deliverability check, claim the row (status: sending via conditional update), send, record sent/failed/suppressed/undeliverable.
• Idempotency authority is the DB (CommunicationLog.idempotencyKey @unique + the status guard), not the BullMQ jobId.

2. DB lens primitives — packages/db/src/lens/*, packages/db/src/hydrate/*, packages/shared/.../stableHash.ts

• fetchLens / includeFromLens / prune — a small, generic projection + hydration layer (a lens narrows the query, hydrates only the projected relations, prunes the result to the projection). Channel-agnostic; email is the first consumer.
• requireWhere fail-closed guard; stableHash backs the idempotency keys.

3. Durable jobs overflow buffer — apps/api/src/jobs/{outbox,handlers/drainOutbox}.ts, JobOutbox model, docs/design/jobs-overflow-buffer.md

• Transactional-outbox + leaky-bucket drain at the enqueueJob chokepoint: once Redis queue depth crosses a cap, adhoc enqueues spill to Postgres and a singleton cron meters them back. Keeps one-job-per-recipient (clean idempotent retries) without a Redis-memory blowup. Generic for all job types; the email fan-out is the first consumer.

4. Communication record + template versioning — CommunicationLog, apps/api/src/hooks/{emailVersioning,userEmailContact}/

• CommunicationLog is the per-recipient delivery ledger (status, sender polymorphism, provider id, deliverability cache).
• Email-template versioning hooks snapshot template/component edits into the audit log so a parent's latest snapshot pins its children's current snapshots.
• userEmailContact keeps an email Contact in sync on user create.

5. Unsubscribe — apps/api/src/routes/unsubscribe.ts, lib/email/unsubscribe.ts

• Signed (HMAC over BETTER_AUTH_SECRET) one-click unsubscribe, POST-only, RFC 8058 List-Unsubscribe / List-Unsubscribe-Post headers on non-system mail.

Removed — packages/email/src/targeting/ and apps/api/src/lib/resolveTargets.ts (replaced by the lens path; no remaining references).

Decisions worth a look

1. cc/bcc resolve through the same lens path and ride only single-message sends; on a fan-out they're dropped (you don't cc 40k separate sends). Flagging in case you want grouped-To semantics.
2. Composed MJML rides in each deliverEmail payload — fine at small N; a send-keyed short-TTL cache is the follow-up for large blasts (no cache primitive here yet).
3. Render-error policy via one probe-render per context (rule errors are template-level / recipient-independent), preserving the degrade/fallback/fail behavior without rendering the whole batch in the planner.
4. The overflow buffer is at-least-once — handlers must be idempotent; the email consumer's CommunicationLog status guard is the delivery-level dedup.

Known follow-ups

• docs/design/jobs-overflow-buffer.md is the original proposal and has drifted from the shipped schema (dedupeKey not collapseKey; no @@index([id]) — the uuidv7 PK is already the FIFO order; depth probe is waiting+active). The schema/code is canonical; doc to be reconciled (per @stevenolay's review note).
• Test coverage on the new send/deliver path and the overflow shutdown-drain / resolve-on-commit-rejection paths is thin.
• inScope (rebac scope gate), resolveSender/resolveFromAddress (per-brand Mailgun, Phase 5) are intentional @wip seams (COMM-005), not wired yet.

Validation

apps/api, packages/email, packages/db typecheck clean; the DB-free unit suites (render, stableHash, lens/hydrate, idempotency, unsubscribe) pass.

🤖 Generated with Claude Code

[agreenspan]

Framing this PR rests on, capturing the cc/bcc discussion:

The deliverable is a pure function of three inputs:

deliver = f(message/template, user, tenancy settings)

• template variant ← (slug, tenancy) — brand→default cascade (settleTemplate)
• sender / domain ← tenancy — per-brand Mailgun (Phase 5) — stubbed today (resolveFromAddress is global)
• relational cc/bcc ← tenancy — org admin / account owner — not derived yet
• suppress / send? ← (user, tenancy, message category) — notification prefs + unsubscribe — not applied yet
• deliverable address ← user — email, never userID

So context is just the handle to tenancy settings, and Recipient { user, context } stays minimal — nothing derived is stored on it. cc/bcc is deliberately not a field: relational cc/bcc derives from context; explicit cc/bcc is the single-message case handled in the planner.

This PR draws that boundary (resolve → group by context → settle template → combine with user per fanned deliver). The stubbed items above are outputs of the same function — named seams to fill, not new dimensions.

[stevenolay]

Review notes on the jobs overflow buffer (we're adopting it for Zealot)

Read through outbox.ts, handlers/drainOutbox.ts, enqueue.ts, the JobOutbox schema, and the tests. This is robust, production-grade work — we're going to adopt the overflow buffer as Zealot's queue-backpressure strategy (over chunking, with chunking + buffer as our documented fallback). A few notes for visibility.

What's especially well done — resolve-on-commit (a crash mid-flush tells the caller "not accepted" rather than silently dropping); the plain-createMany(skipDuplicates) vs keyed-upsert split with the explicit reasoning that skipDuplicates keeps first-commit-wins not latest; the flag lifecycle (NX set-once + TTL heartbeat-renew for the whole pass, EXPIRE no-op so the final renew never resurrects a cleared flag); depth = waiting + active excluding delayed (the standing cron floor); and the honest at-least-once contract. The shared flush/drain mutex + singleton drain is the right concurrency model.

1. Design-doc ↔ code drift

docs/design/jobs-overflow-buffer.md models JobOutbox with collapseKey + @@index([id]); the shipped schema (packages/db/prisma/schema/jobOutbox.prisma) uses dedupeKey and drops the index (uuidv7 id is time-ordered, so orderBy: { id: 'asc' } is the FIFO drain order). Worth reconciling the doc so the canonical example matches the code a downstream porter will copy.

2. MySQL port note (for the Zealot / Tribe ports)

@default(dbgenerated("uuidv7()")) is a Postgres-side function — MySQL has no uuidv7(), so we'll generate the id app-side (the uuidv7 npm pkg is already imported in enqueue.ts, so this is small). The @@unique([handlerName, dedupeKey]) NULL-distinct behavior (many fan-out rows coexist) and createMany({ skipDuplicates }) (→ INSERT IGNORE) both carry to MySQL. So it's a contained change, just flagging it.

3. Sustained-overload failure mode (accepted ceiling, just naming it)

Under arrival-rate > drain-throughput, JobOutbox grows between ticks — write volume stays low thanks to the write-behind batching, but row count climbs. warnIfOverflowStuck alerts on it, but that's an alert, not producer backpressure / load-shedding. This is the intended trade (move unboundedness from Redis-OOM to a cheaper-to-grow table), worth stating as the design's explicit ceiling in case a bounded-table or shed-load policy is wanted later.

4. Drain holds the shared mutex across the whole queue.add admit loop

You already flag this as "an accepted, self-recovering stall." For a large room (thousands), the sequential add-loop blocks that process's flushes for the whole pass; a chunked admit that released the mutex between sub-batches would shrink the window. Non-blocking nit.

5. Test coverage — strong, with a few gaps

Routing (adhoc/bypass/cron), resolve-on-commit, jobId dedup across flushes, superseding collapse (within-batch + cross-flush), and drain top-up/flag-clear are all covered. Not seen: the at-least-once replay (crash between queue.add and the row delete → re-admit next tick), the flushOutbox shutdown drain + retry, and the resolve-on-commit rejection path (flush throws → caller sees rejection). The first is timing-hard; the latter two look testable.

None of these are blockers — mostly doc reconciliation + port notes. Nice piece of infra.

[arongreenspan]

@stevenolay thanks for the close read on the overflow buffer — glad it's a fit for Zealot's backpressure. Walking your five points with where the code stands:

1. Design-doc ↔ code drift — you're right, and the schema is the canonical artifact. For porters copying this, trust the shipped jobOutbox.prisma, not the doc sketch:

• the field is dedupeKey (not collapseKey), and there is no dedupeKey ?? options.id fallback — a spilled job's lane is just dedupeKey ?? null;
• no @@index([id]) — the uuidv7 PK is already time-ordered, so orderBy: { id: 'asc' } is the FIFO drain order;
• depth probe is getJobCounts('waiting','active') (excludes delayed — that's the standing cron-repeat floor, not overflow pressure).

The design doc is the proposal that predates those final calls; reconciling it is the concrete follow-up from your note (now flagged in the PR description's Known follow-ups).

2. MySQL port (app-side uuidv7) — agreed, contained. JobOutbox.id uses Postgres dbgenerated("uuidv7()"); for the MySQL ports, generate the id app-side at the create site — the uuidv7 pkg is already imported in enqueue.ts for the spill jobId. The @@unique([handlerName, dedupeKey]) NULL-distinct behavior and createMany({ skipDuplicates }) → INSERT IGNORE both carry over. No template change needed.

3. Sustained-overload row growth — yes, that's the intended ceiling. The trade is explicit: move unboundedness off Redis (OOM, ungraceful) onto a cheap-to-grow Postgres table; warnIfOverflowStuck is the alert, not producer backpressure. A bounded-table / shed-load policy is a deliberate non-goal here — worth naming in the doc's open-questions, as you suggest.

4. Drain holds the mutex across the admit loop — accepted, self-recovering stall. It's documented as such at the top of drainOutbox.ts. A chunked admit that releases the mutex between sub-batches is the clean lever if a large room ever makes it bite; not pulled yet since it hasn't.

5. Test gaps — fair, all three are real. At-least-once replay is timing-hard; the shutdown-drain (flushOutbox retry loop) and the resolve-on-commit rejection path are both testable and worth adding. Tracking rather than blocking (noted in the PR description).

Net: 2–4 are intended design with the rationale above; 1 (doc reconciliation) and 5 (tests) are genuine follow-ups, not yet in the code.

---

Generated by Claude Code

[agreenspan]

@stevenolay — capturing where we landed on the email/comms direction after going through this PR against your two in-flight Zealot PRs (#1455 delivery, #1461 @zealot/email). Three buckets: decided, still open, and still-sketched-here.

Decided

• The architecture is shared and validated. Per-recipient fan-out; the durable log row — not the BullMQ jobId — is the idempotency authority (jobId is evicted by removeOnComplete under load). Your #1455 already converges on this independently, so we're not asking you to change the shape.
• Keep Zealot's NotificationLog. It is the equivalent of this PR's CommunicationLog — same per-recipient ledger. Widen it; don't add a parallel table.
• The log-row id is the single correlation identity end-to-end: it's the idempotency key where the provider accepts one, the verify-lookup key where it doesn't, and the reconciliation key. No separate DLQ — an unconfirmed row is the work-list.
• Per-adapter confirmation capability. An adapter is exactly-once-capable if it has either idempotentSend (provider replays the original response on a repeated key — Resend) or verify(correlationKey) → { status, providerMessageId } (Mailgun events / Twilio lookup). Absent both → at-most-once floor (Slack/SMS/webhook can't dedup a resend). Note the provider split: Zealot email is Mailgun, which has no send-idempotency key, so Zealot reaches exactly-once via verify, not via an idempotency key — don't go hunting for one.
• Don't ship dead enum values. CommunicationStatus stays lifecycle-only (queued/sending/sent/failed/suppressed/undeliverable). delivered/bounced/complained (and a deliveredAt) land with the webhook ingestion that writes them, not before. I removed the orphaned deliveredAt column here to match (the enum was already trimmed).
• camelCase enum values; status = lifecycle stage, reason = error. On the Zealot side this means standardizing NotificationLog's untyped VARCHAR status into a real Prisma enum, collapsing the DNS→email alias, and moving failure reasons (CONTENT_NOT_FOUND, USER_DATA_REPLACEMENT_FAILED) out of the status into errorMessage.
• Overflow buffer is adopted as Zealot's queue-backpressure strategy (your review note); route the fan-out through it before any large-brand flag.

Still working through

• Persisted-status migration sequencing (Zealot). Migrate NotificationLog.status VARCHAR → canonical Prisma enum and update the live Lambda's status constants now (one vocabulary from day one), vs. map at the boundary until the Lambda is retired. Leaning full-sweep.
• #1455 ↔ #1461 are not joined. Delivery still renders with the legacy replaceUserData; the @zealot/email MJML pipeline has no consumer. This PR runs them as one path (deliverEmail → settleTemplate → compose). Open question: when does Zealot delivery switch to consuming @zealot/email? settleTemplate's render-error policy (degrade/fallback/fail) ports at that integration point, not before.
• Ambiguous bucket representation — an explicit unconfirmed status, or a stale sending row swept by the reconciler.

Still unbuilt in this PR (don't treat as reference-complete)

• inScope → return true — the authorization gate (recipient ∈ scope of the entity) is a no-op pass-through, i.e. fail-open. COMM-005.
• resolveSender / resolveFromAddress → env-default stubs — sender identity is ignored; every mail sends from one DEFAULT_FROM_EMAIL. Multi-tenant sender branding isn't real yet. COMM-003.
• No delivery-event ingestion — no webhook consumer; delivered/bounced/complained deliberately not in the schema until that lands.
• recomposeCommunication → @wip — resend/preview replay-from-snapshot has no consumer (recomposeSnapshot, the live path, works).
• Custom (non-registry) send path — stubbed to the base User lens until the segment builder ships.
• sms/push/inApp channels — enum-only; email is the only built transport.
• cc/bcc — dropped on a fan-out (single-message sends only).
• Send-keyed MJML cache — not built; composed MJML rides in every deliverEmail payload (fine at small N).