Email outbox backend by N2D4 · Pull Request #1030 · hexclave/stack-auth

N2D4 · 2025-11-26T07:30:17Z

I'm sorry for whoever reviews this. I tried to split it up as much as possible so this is only the existing functionality migrated onto the new email outbox table; the new endpoints or UI changes are not here yet.

Tests not passing yet but already ready for reviews.

How to review:

The most important file is schema.prisma. It contains the explanation of the new EmailOutbox table, and all its constraints.
After that, check the updated emails.tsx. It writes to the EmailOutbox table.
Then, check email-queue-step.tsx. It contains the logic that renders, queues, and sends an email (alongside emails-low-level.tsx). This is called once every ~second (in a Vercel cron job in prod or a script in dev) and whenever a new email is sent (see the callers of runEmailQueueStep for more info).
Then, look at the updated endpoints. Would love to have some eyes on the ones that @BilalG1 wrote in particular.
Finally, everything else. Everything else should be relatively straight-forward, I think, mostly migrations.

The prompt that generated the first version (but note that the final version is somewhat different from what was generated, but it might help get some overview):

i want you to implement a new email sending backend infrastructure. apps/backend is the right folder

let's start with the database. For this purpose, I've already created an EmailOutbox model in the prisma schema. Note that, for now, canHaveDeliveryInfo is always false (we never fetch delivery info rn). Note that the EmailOutbox is on the global database, not the source-of-truth database (which means it's not in the same database as the project users, for example, and can't have foreign key relations to those). You must create a new migration for this new schema; you will have to manually write the migration SQL for it (can't use the Prisma command). After that, create another subsequent migration that migrates from the old SentEmail table to the new one.

Create a file in `lib` that has functions for the email outbox for listing all emails (returning the database objects), editing an email, resetting an email, deleting an email (which should only work under the same conditions as the retry one). This file should be SUPER clean, make sure to have many explained/documented functions with concise docs.

You also want to update emails.tsx to use the new email outbox. The usual current email-sending functions should instead of sending the email directly, create an EmailOutbox row in the database instead. It should also queue the email-queue-step below onto QStash so that it can start queuing itself repeatedly (this should probably be exported as a function so. Keep the existing function implementations somewhere for your own reference until you've implemented the email queue step endpoint down below, as you'll need to implement the email queue step endpoint down below with them. Note that if you send from a templtae to many users, each user is its own row in the email outbox system. Feel free to update the signature of the send email functions to match the information that is required in EmailOutbox, just update all the callers.

Also, update the `email-rendering.tsx` file to export a new function that allows you to render many email templates/drafts/themes at once, as long as they are for the same tenancy ID. They should all be in a single call to Freestyle.

Note that difference between InternalError and ExternalError in the EmailOutbox schema. Internal errors contain all the information, but maybe have sensitive implementaiton-speciifc data that should not exposed to the end user who sent the email. External errors are guaranteed to be safe to expose to the end user who sent the email. There is already a similar handling that's implicit in the emails.tsx file, so take some of its logic and make it more explicit (for both rendering & sending server errors, make functions that take internal full error objects and returns an external error object, with just a string like "Internal server error" if it's an unknown error). 

Also, create the following endpoints (please look at other endpoints in the backend to see how they're implemented, and make sure the security is good):

- GET /api/latest/emails/delivery-info: This endpoint returns some stats on the emails. It should return the number of emails sent in the last hour, last day, last week, and last month, and how many of those emails (per hour, day, week, and month) were marked as spam or bounced. It should also return the "capacity rate": the number of emails that can be sent per second. It should be calculated like this: `max(max(<the number of emails sent in the last week> * 4, 200) * penaltyFactor, 5) / 7 / 60 / 60`, where penaltyFactor is a number between 0 and 1 depending on what % of emails were bounced or marked as spam (I will build a precise metric for this later, for now you can think about it and make your own).
- GET /api/latest/emails/outbox: List all emails in the outbox, paginated like the users endpoint. Should return a nice type for the individual rows which should at least contain the status and simple status, and probably some other useful fields that you think should be displayed to the user in the UI, but not all those from the database (if you're not sure whether it's needed, don't include it, we can include it later). The function which converts DB EmailOutbox objects to the API type should be separate from the other stuff as we will use it elsewhere too.
- PATCH /api/latest/emails/outbox/{id}: Edit an email in the outbox, if the conditions for it are met. Only some fields can be edited, see the Prisma schema for an explanation
- POST /api/latest/emails/outbox/{id}/cancel: Cancel an email in the outbox, if the conditions for it are met
- POST /api/latest/emails/outbox/{id}/retry: Retry an email in the outbox, if the conditions for it are met. See the Prisma schema for an explanation
- POST /api/latest/internal/email-queue-step: This is a QStash receiver that is constantly called by QStash in an infinite loop. Make sure that it is super clean and easy to understand. Feel free to split it up into multiple functions and files as needed.
- - First thing it should do is add a call to itself back into the QStash queue, with Flow Control with a flow control key equal to the URL, a rate limit of 1 per second, and a parallelism limit of 1.
- - Then, it should acquire a Postgres exclusive lock, to make sure that only one email-queue-step can be running at a time (make sure this is easy to understand and free of data races). It should be released automatically at the end of this endpoint (think about how we can make it that there's no deadlock if the endpoint fails for whatever reason, eg. the server it's running on dies).
- - Now, it should get the timestamp of the last execution of this endpoint (which is stored in some metadata table in the global database), and calculate the precise time delta in seconds. The new time should be updated in the metadata table. (Make sure that it's not negative, which can happen due to clock skew, so make sure to handle that by just not doing anything if the new time is before the last execution time).
- - Then, it should fetch all the email outbox rows that do not have a renderedByWorkerId, and set renderedByWorkerId to a random UUID and startedRenderingAt to now (it should do this in an atomic statement so that an email is never rendered by multiple workers at the same time). It should then use email-rendering.tsx to render the email, using the previously created batch-by-tenancy function (you need to group the email outbox rows by tenancy ID for this). Once done, it should (again, in a single atomic statement) check whether the renderedByWorkerId is still set to the same UUID (this is to make sure there are no race conditions), and if so, set the appropriate fields in the EmailOutbox row.
- - Next, it should run a SQL query that updates all rows with scheduledAt < now() and isQueued = false to set isQueued = true.
- - Now, it should fetch a deduplicated list of all tenancy IDs of all email outbox rows that have isQueued = true and startedSendingAt being null, and set startedSendingAt to now.
- - For each tenancy ID in the list:
- - - Calculate the capacity rate. Multiply it by the time delta since the last endpoint execution. Use stochastic rounding to round to the nearest integer.
- - - For that tenancy, fetch at most that many email outbox rows that have isQueued = true and startedSendingAt being null, and set startedSendingAt to now. Fetch those rows that have the highest priority (see the Prisma schema for an explanation).
- - - Then, asynchronously in parallel, process each email that needs to be sent. Use the email-sending functions to send the emails, and update finishedSendingAt and canHaveDeliveryInfo (recall it's always set to false for now) for each row when the email is sent. Before you send the email, check whether the user still exists in the DB, and use the same notification category logic from the send-email endpoint (extract it into a helper function) to find out which users have already unsubscribed and should not receive the email, and set skippedReason accordingly if necessary. If the email encountered an error while sending, set the sendServerError-ish fields in the same atomic statement. `waitUntil` each promise (to make sure that they continue running in the background). Ensure that a bug when handling one of the emails does not mess with the other emails. Don't use `.catch`, ever, use try-catch blocks, and use `captureError("<some-machine-readable-location-id>", ...)` to capture all unknown errors aggressively.


While you're at it, we now have some new features like scheduling, add those to the send-email endpoint.

Make sure to make VERY comprehensive E2E tests for ALL the edge cases.

The code should be ULTRA clean and NEVER repetitive, feel free to create helper functions and files as needed. Definitely make sure everything is clean and easy to understand. When there's some logic that is used in multiple places, create a helper function for it. In the end, write a human-readable README.md file in the email endpoint folder that explains how the email infrastructure works, meant to be read by a technical human (it should be VERY concise and understandable!)

Note

Migrates email to an async outbox pipeline with queueing/rendering, delivery stats, updated APIs, and extensive tests.

Database/Migrations:
- Add EmailOutbox schema with generated columns, constraints, and indices; migrate data from SentEmail and drop it.
Backend/Logic:
- Implement outbox-driven flow: enqueue via sendEmailToMany, batch render (Freestyle), queue, and send (low-level SMTP/provider) with robust error handling and skip reasons.
- Add periodic processor: runEmailQueueStep and internal endpoint /api/latest/internal/email-queue-step with locking, delta-based capacity, and stochastic quotas.
- Extract low-level email sender, batched renderer, and delivery stats calculator.
APIs:
- Update POST /api/latest/emails/send-email (per-user enqueue, variables JSON, priority/scheduling, notification category validation).
- Add GET /api/latest/emails/delivery-info (hour/day/week/month stats and capacity).
- Switch internal email listing to outbox-backed data shape.
SDK/Template:
- Expose getEmailDeliveryStats and useEmailDeliveryStats; server interface method added.
Tests:
- Comprehensive E2E for queue step, delivery stats, unsubscribe, drafts/templates, variables, all-users sends, and edge cases.
- Test helpers improved (mailbox wait utilities, auth fast signup).
Infra/Tooling:
- Add Vercel cron for email queue step; dev script to run queue locally.
- Docker deps: add Drizzle Gateway; tweak Postgres and compose; CI workflow renamed to build/run.
- Misc refactors (ESM config, Next/Sentry adjustments), README for email pipeline.

^{Written by Cursor Bugbot for commit 313c229. This will update automatically on new commits. Configure here.}

Summary by CodeRabbit

New Features
- Asynchronous Email Outbox pipeline (queuing, rendering, scheduling, priority, drafts) and internal queue runner
- Delivery statistics API with capacity metrics
- Send-email API: draft/template/html support, high-priority flag, and results now return user_id
- Unsubscribe handling and two‑pass template rendering; unsubscribe links
Bug Fixes
- Improved deliverability checks, error reporting, retries and skip conditions
Documentation
- Email infrastructure README added
Tests
- Extensive e2e and unit tests for email flows and delivery-info endpoint

_{✏️ Tip: You can customize this high-level summary in your review settings.}

coderabbitai

Actionable comments posted: 0

🧹 Nitpick comments (1)

apps/backend/src/auto-migrations/index.tsx (1)

104-196: Avoid sleeping while holding the advisory xact lock (and a DB connection)

await wait(500) runs inside the Prisma transaction, so it holds pg_advisory_xact_lock(...) and the transaction connection for the extra 500ms, which can amplify contention/timeouts when multiple workers race migrations. Prefer committing first, then sleeping before the next retry.

       await options.prismaClient.$transaction(async (tx) => {
+        let requestedRepeatDelayMs: number | null = null;
         log(`  |> Preparing...`);
         await tx.$executeRaw`
           SELECT pg_advisory_xact_lock(${MIGRATION_LOCK_ID});
         `;
@@
                 if (res[0].should_repeat_migration) {
                   log(`  |> Migration ${migration.migrationName} requested to be repeated. This is normal and *not* indicative of a problem.`);
-                  await wait(500);  // give the database a chance to catch up with everything else that's happening
+                  requestedRepeatDelayMs = 500; // delay after commit to avoid holding locks/connections
                   // Commit the transaction and continue re-running the migration
                   return;
                 }
@@
         newlyAppliedMigrationNames.push(migration.migrationName);
         shouldRepeat = false;
-      }, {
+        if (requestedRepeatDelayMs != null) {
+          // keep shouldRepeat=true, but sleep outside of the transaction
+        }
+      }, {
         timeout: 80_000,
         maxWait: 30_000,
       });
+      // If we returned early to repeat, wait here after commit/connection release.
+      if (shouldRepeat) {
+        await wait(500);
+      }
     }
   }

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between e3decb1 and c7fc0a0.

📒 Files selected for processing (2)

apps/backend/prisma/migrations/20251020183000_migrate_sent_email/migration.sql (1 hunks)
apps/backend/src/auto-migrations/index.tsx (2 hunks)

🚧 Files skipped from review as they are similar to previous changes (1)

apps/backend/prisma/migrations/20251020183000_migrate_sent_email/migration.sql

🧰 Additional context used

📓 Path-based instructions (5)

**/*.{ts,tsx}