[world-postgres] Add a producer role that enqueues without a runner or startup recovery - #3969
Open
Ehco1996 wants to merge 1 commit into
Open
[world-postgres] Add a producer role that enqueues without a runner or startup recovery#3969Ehco1996 wants to merge 1 commit into
Ehco1996 wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: 633b58d The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
|
@Ehco1996 is attempting to deploy a commit to the Vercel Labs Team on Vercel. A member of the Team first needs to authorize it. |
Ehco1996
force-pushed
the
feat/world-postgres-producer-role
branch
from
September 4, 2026 09:48
795f1c2 to
2a2c3cc
Compare
…r startup recovery `queue()` awaits `start()`, and `start()` both starts a graphile-worker runner and calls `reenqueueActiveRuns()`. A process that only ever enqueues therefore gets both, and in the common "API process starts runs, worker process executes them" topology both are unwanted: the API's runner claims due jobs and fails their HTTP delivery whenever the worker has no ready endpoint, and every API replica replays every active run on startup, including runs a live peer is executing. Add a `role` option to `PostgresWorldConfig`, its vocabulary declared once as a zod enum in `config.ts`, defaulting to `worker` so existing behaviour is unchanged. `createWorld()` resolves the role once — option, then `WORKFLOW_POSTGRES_ROLE`, then the default — and hands it to `createQueue()`, so nothing else reads the environment. Under `producer`, `start()` still creates the worker utils, migrates the schema, and runs the pg-boss job migration — so a producer can enqueue into a fresh database — but never calls graphile's `run()` and never re-enqueues active runs. `queue()` is untouched, and `close()` already tolerates a null runner, so it ends only what was opened. This removes N of the N+1 startup-recovery sources behind vercel#3119 and vercel#3758 without touching the recovery predicate itself. Closes vercel#3968 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Ehco <ehco@mewtant.io>
Ehco1996
force-pushed
the
feat/world-postgres-producer-role
branch
from
September 5, 2026 00:22
2a2c3cc to
633b58d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A process that only ever enqueues cannot avoid getting a consumer.
queue()begins withawait start(), andstart()both starts a graphile-worker runner and callsreenqueueActiveRuns(). In the common "API process starts runs, worker process executes them" topology — where only the worker compiled the"use workflow"/"use step"code and serves.well-known/workflow/v1/*— both of those are unwanted in the API. Its runner claims due jobs and then fails their HTTP delivery whenever the worker has no ready endpoint (a rolling update, for instance), manufacturing failures the worker's own runner never produces, because that one dies together with its executor and leaves the job waiting. And every API replica runs full startup recovery, so N replicas rolling means N re-enqueues of every active run on top of the worker's own.This adds
role?: 'producer' | 'worker'toPostgresWorldConfig, defaulting to'worker'— today's behaviour, no change for existing users.WORKFLOW_POSTGRES_ROLEis read lazily as a fallback, matching how the package's other options pick up environment configuration when it is selected throughWORKFLOW_TARGET_WORLD. Under'producer',start()still creates the worker utils, runsworkerUtils.migrate(), and runs the pg-boss job migration, so a producer can enqueue into a fresh database; what it never does is call graphile'srun()(none of thesetupListeners/startRunnerWhenExecutorIsReady/deferRunnerStartpaths) orreenqueueActiveRuns().queue()is untouched, andclose()already tolerates a null runner, so it ends only what was opened. Nothing about the recovery predicate changes: this removes N of the N+1 recovery sources behind #3119 and #3758 for this topology, and leaves the remaining one — the worker's own — exactly as it is.How did you test your changes?
Unit tests in the two existing world-postgres suites, which already mock
graphile-worker, so "no runner" is asserted directly onrun()rather than inferred from a timeout.src/reenqueue.test.ts(World level):starts a graphile runner and recovers active runs in the default role— pins the unchanged default.does not start a graphile runner in the producer role—run()not called,migrate()still called.does not re-enqueue active runs in the producer role— two active runs listed, noaddJob.still enqueues messages in the producer role—world.queue(...)lands aworkflow_flowsjob, still no runner.reads the producer role from WORKFLOW_POSTGRES_ROLE.ignores an unrecognized WORKFLOW_POSTGRES_ROLE— falls through toworker.closes only what a producer opened—runner.stop()not called,workerUtils.release()called once.src/queue.test.ts(queue level):never starts a runner in the producer role, even when a local executor is reachable— a real loopback HTTP server is listening andPORTis set, so the unfixed code starts the runner synchronously insidestart(); the producer starts none, enqueues fine, and the server receives no request.All six new cases fail against unmodified
mainand pass with the change.Commands run locally:
pnpm exec biome ci --max-diagnostics=200— exit 0 (warning count unchanged; the only warning in the touched files is the pre-existingcreateTaskHandlercomplexity one).node scripts/check-no-unrun-tests.mjs— pass.node scripts/check-changesets.mjs— pass, 1 pending changeset, 1 package released.tsc --noEmitinpackages/world-postgres— clean. (pnpm turbo typecheckcould not run end to end here: the dependency graph builds@workflow/swc-plugin, which needs a Rust toolchain with thewasm32-unknown-unknowntarget that this machine does not have.)vitest run src/inpackages/world-postgres— 4 files, 35 tests passed.vitest run src -t "packages/world-postgres/README.md"inpackages/docs-typecheck— 3 samples pass, including the new one.bun ./scripts/lint.tsindocs— 0 errors.test/*.test.tsintegration suites could not run: they start a Postgres testcontainer and there is no container runtime available here. They are untouched by this change, which adds no storage or SQL code.Docs:
roledocumented alongsidequeueConcurrencyandapplicationManagedShutdowninpackages/world-postgres/README.md(options table, environment-variable table, and a short "Producer-only processes" section), indocs/content/docs/v5/configuration/worlds.mdx, and asWORKFLOW_POSTGRES_ROLEindocs/content/worlds/v5/postgres.mdx.Closes #3968
PR Checklist - Required to merge
pnpm changesetwas run to create a changelog for this PRgit commit --signoffon your commits)@vercel/workflowin a comment once the PR is ready, and the above checklist is complete🤖 Generated with Claude Code
PostgresWorldRoleSchemais exported as a value from the package index alongside thePostgresWorldRoletype, so a consumer can passrole: PostgresWorldRoleSchema.enum.producerinstead of spelling the literal.