Skip to content

fix(world-postgres): abort stalled HTTP delivery on shutdown#3064

Merged
VaguelySerious merged 4 commits into
vercel:mainfrom
joeyhotz:agent/world-postgres-graceful-shutdown
Jul 24, 2026
Merged

fix(world-postgres): abort stalled HTTP delivery on shutdown#3064
VaguelySerious merged 4 commits into
vercel:mainfrom
joeyhotz:agent/world-postgres-graceful-shutdown

Conversation

@joeyhotz

@joeyhotz joeyhotz commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Description

We observed this failure in a self-hosted deployment with a short termination grace period:

  1. A revision received SIGTERM while Graphile Worker was delivering a step to the local Workflow HTTP route.
  2. Graphile stopped taking new jobs and began its graceful-shutdown wait.
  3. After that wait, Graphile aborted the task's helpers.abortSignal, but world-postgres discarded the signal, so its HTTP request kept waiting.
  4. world.close() did not finish before the platform sent SIGKILL. The PostgreSQL job remained locked until Graphile's four-hour stale-lock recovery window.

The same missing abort forwarding exists on current main / the v5 beta.

This PR forwards helpers.abortSignal to fetch, including the response-body read. If graceful shutdown aborts a stalled delivery, the task rejects into Graphile's existing failure path. Graphile unlocks the same PostgreSQL row; the shutdown path inserts no replacement job.

Graphile records an attempt when it claims a row, before the HTTP call:

State when shutdown starts Result
Delivery finishes during Graphile's grace period Normal completion
Delivery is aborted and attempt budget remains Same row is unlocked and becomes eligible after Graphile's normal retry backoff
Delivery is aborted on its final attempt Same row is unlocked but is terminal

So this is a graceful lock-release fix, not guaranteed continuation of a final-attempt job. The adapter deliberately does not refund the attempt: aborting the client cannot prove that the server handler stopped, and replaying it as though the first delivery never happened could duplicate partial work.

The lifecycle changes are limited to what that shutdown requires:

  • world.close() stops the queue, waits for runner.promise, then closes the streamer and internally owned pool.
  • It still waits for runner.promise if Graphile.s automatic shutdown started first.
  • A runner completion error cannot prevent queue, streamer, and pool cleanup; Graphile has already logged the worker failure.
  • A failed world.close() call does not poison later cleanup attempts; a retry performs a fresh ordered shutdown.
  • WORKFLOW_POSTGRES_APPLICATION_MANAGED_SHUTDOWN=1 enables application-managed shutdown for the standard package target; applicationManagedShutdown: true provides the same opt-in for programmatic Worlds. The default remains false.

The application-managed mode prevents a second lifecycle race: Graphile's default handler re-sends the termination signal as soon as its worker pool stops, which can end the process before application-owned HTTP, database, or telemetry cleanup finishes.

SIGKILL can still preempt cleanup, and HTTP handlers must remain safe for at-least-once execution.

How did you test your changes?

Added loopback HTTP tests for aborting:

  • While waiting for response headers.
  • While reading a partial response body.
  • Without creating a replacement job in either path.

Added lifecycle tests for:

  • Waiting for Graphile.s completion promise when stop() returns early or reports that shutdown already started.
  • Continuing queue, streamer, and pool cleanup when the runner completion promise rejects.
  • Queue → streamer → internally owned pool close ordering.
  • Retrying world.close() after a transient cleanup failure.
  • Default, environment-configured, and programmatic application-managed shutdown behavior.
  • Caller-owned pools remaining open.

Verification:

  • pnpm --filter @workflow/world-postgres test (163 tests, including Testcontainers PostgreSQL conformance tests)
  • pnpm --filter @workflow/world-postgres typecheck
  • pnpm --filter @workflow/world-postgres build
  • Executable documentation samples in packages/world-postgres/README.md (2 tests)
  • Biome checks over the changed TypeScript files
  • pnpm changeset status --since=upstream/main

A real PostgreSQL repro confirmed the attempt behavior above with pinned Graphile Worker 0.16.6 and current 0.17.3 (maxAttempts: 1 becomes unlocked/terminal; maxAttempts: 2 becomes unlocked/retryable). Graphile 0.17.3 does not provide an attempt-neutral cancellation primitive.

Docs Preview

The fork preview is pending Vercel Labs authorization. Direct page links will be added when the deployment is available.

PR Checklist - Required to merge

  • 📦 pnpm changeset was run to create a changelog for this PR
  • 🔒 DCO sign-off passes (run git commit --signoff on your commits)
  • 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

@changeset-bot

changeset-bot Bot commented Jul 23, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 2342b1e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@workflow/world-postgres Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

@joeyhotz is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

@joeyhotz
joeyhotz force-pushed the agent/world-postgres-graceful-shutdown branch from 926a5fe to 856e2c6 Compare July 23, 2026 11:19
@joeyhotz joeyhotz changed the title fix(world-postgres): gracefully stop active jobs fix(world-postgres): abort stalled HTTP delivery on shutdown Jul 23, 2026
@joeyhotz
joeyhotz force-pushed the agent/world-postgres-graceful-shutdown branch from 856e2c6 to 2a98e77 Compare July 23, 2026 11:41
@joeyhotz
joeyhotz marked this pull request as ready for review July 23, 2026 11:47
@joeyhotz
joeyhotz requested review from a team and ijjk as code owners July 23, 2026 11:47
@joeyhotz

Copy link
Copy Markdown
Contributor Author

Pinging @vercel/workflow

@VaguelySerious VaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, doing another AI pass before approving

Comment thread packages/world-postgres/HOW_IT_WORKS.md Outdated
Comment thread .changeset/gentle-jobs-stop.md Outdated

@VaguelySerious VaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found

Comment thread packages/world-postgres/src/index.ts Outdated
joeyhotz added 3 commits July 24, 2026 08:41
Forward Graphile Worker task cancellation to HTTP delivery so graceful shutdown can release a stalled request through Graphile native failure handling. Wait for the runner to finish before closing dependent resources, and let applications with broader lifecycle cleanup manage shutdown.

A shutdown abort consumes the attempt Graphile recorded on claim. It creates no replacement row and retries only when the existing attempt budget permits.

Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Allow the standard package target to opt into application-managed shutdown without a custom World module. Document the environment variable and cover both its enabled and default behavior.

Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
@joeyhotz
joeyhotz force-pushed the agent/world-postgres-graceful-shutdown branch from 120943f to 2ed986b Compare July 24, 2026 00:42
@joeyhotz

Copy link
Copy Markdown
Contributor Author

Thanks @VaguelySerious, attended to the feedback. @vercel/workflow

@joeyhotz
joeyhotz requested a review from VaguelySerious July 24, 2026 00:48
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
@VaguelySerious
VaguelySerious merged commit cdb3db4 into vercel:main Jul 24, 2026
64 of 105 checks passed
github-actions Bot added a commit that referenced this pull request Jul 24, 2026
Signed-off-by: Joey Hotz <joeyhotz1@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

Backport PR opened against stable: #3082. Merge conflicts were resolved by AI — please review carefully. (backport job run)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants