fix(world-postgres): abort stalled HTTP delivery on shutdown#3064
Merged
VaguelySerious merged 4 commits intoJul 24, 2026
Merged
Conversation
🦋 Changeset detectedLatest commit: 2342b1e The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
|
@joeyhotz is attempting to deploy a commit to the Vercel Labs Team on Vercel. A member of the Team first needs to authorize it. |
joeyhotz
force-pushed
the
agent/world-postgres-graceful-shutdown
branch
from
July 23, 2026 11:19
926a5fe to
856e2c6
Compare
joeyhotz
force-pushed
the
agent/world-postgres-graceful-shutdown
branch
from
July 23, 2026 11:41
856e2c6 to
2a98e77
Compare
joeyhotz
marked this pull request as ready for review
July 23, 2026 11:47
Contributor
Author
|
Pinging @vercel/workflow |
VaguelySerious
left a comment
Member
There was a problem hiding this comment.
LGTM, doing another AI pass before approving
VaguelySerious
left a comment
Member
There was a problem hiding this comment.
AI review: blocking issues found
Forward Graphile Worker task cancellation to HTTP delivery so graceful shutdown can release a stalled request through Graphile native failure handling. Wait for the runner to finish before closing dependent resources, and let applications with broader lifecycle cleanup manage shutdown. A shutdown abort consumes the attempt Graphile recorded on claim. It creates no replacement row and retries only when the existing attempt budget permits. Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Allow the standard package target to opt into application-managed shutdown without a custom World module. Document the environment variable and cover both its enabled and default behavior. Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
joeyhotz
force-pushed
the
agent/world-postgres-graceful-shutdown
branch
from
July 24, 2026 00:42
120943f to
2ed986b
Compare
Contributor
Author
|
Thanks @VaguelySerious, attended to the feedback. @vercel/workflow |
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
VaguelySerious
approved these changes
Jul 24, 2026
github-actions Bot
added a commit
that referenced
this pull request
Jul 24, 2026
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Contributor
|
Backport PR opened against |
This was referenced Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
We observed this failure in a self-hosted deployment with a short termination grace period:
SIGTERMwhile Graphile Worker was delivering a step to the local Workflow HTTP route.helpers.abortSignal, butworld-postgresdiscarded the signal, so its HTTP request kept waiting.world.close()did not finish before the platform sentSIGKILL. The PostgreSQL job remained locked until Graphile's four-hour stale-lock recovery window.The same missing abort forwarding exists on current
main/ the v5 beta.This PR forwards
helpers.abortSignaltofetch, including the response-body read. If graceful shutdown aborts a stalled delivery, the task rejects into Graphile's existing failure path. Graphile unlocks the same PostgreSQL row; the shutdown path inserts no replacement job.Graphile records an attempt when it claims a row, before the HTTP call:
So this is a graceful lock-release fix, not guaranteed continuation of a final-attempt job. The adapter deliberately does not refund the attempt: aborting the client cannot prove that the server handler stopped, and replaying it as though the first delivery never happened could duplicate partial work.
The lifecycle changes are limited to what that shutdown requires:
world.close()stops the queue, waits forrunner.promise, then closes the streamer and internally owned pool.runner.promiseif Graphile.s automatic shutdown started first.world.close()call does not poison later cleanup attempts; a retry performs a fresh ordered shutdown.WORKFLOW_POSTGRES_APPLICATION_MANAGED_SHUTDOWN=1enables application-managed shutdown for the standard package target;applicationManagedShutdown: trueprovides the same opt-in for programmatic Worlds. The default remainsfalse.The application-managed mode prevents a second lifecycle race: Graphile's default handler re-sends the termination signal as soon as its worker pool stops, which can end the process before application-owned HTTP, database, or telemetry cleanup finishes.
SIGKILLcan still preempt cleanup, and HTTP handlers must remain safe for at-least-once execution.How did you test your changes?
Added loopback HTTP tests for aborting:
Added lifecycle tests for:
stop()returns early or reports that shutdown already started.world.close()after a transient cleanup failure.Verification:
pnpm --filter @workflow/world-postgres test(163 tests, including Testcontainers PostgreSQL conformance tests)pnpm --filter @workflow/world-postgres typecheckpnpm --filter @workflow/world-postgres buildpackages/world-postgres/README.md(2 tests)pnpm changeset status --since=upstream/mainA real PostgreSQL repro confirmed the attempt behavior above with pinned Graphile Worker 0.16.6 and current 0.17.3 (
maxAttempts: 1becomes unlocked/terminal;maxAttempts: 2becomes unlocked/retryable). Graphile 0.17.3 does not provide an attempt-neutral cancellation primitive.Docs Preview
The fork preview is pending Vercel Labs authorization. Direct page links will be added when the deployment is available.
PR Checklist - Required to merge
pnpm changesetwas run to create a changelog for this PRgit commit --signoffon your commits)@vercel/workflowin a comment once the PR is ready, and the above checklist is complete