Skip to content

feat: refill execution slots eagerly and add self-hosted Supabase guide#620

Open
jumski wants to merge 1 commit intomainfrom
feat/slot-aware-backpressure
Open

feat: refill execution slots eagerly and add self-hosted Supabase guide#620
jumski wants to merge 1 commit intomainfrom
feat/slot-aware-backpressure

Conversation

@jumski
Copy link
Copy Markdown
Contributor

@jumski jumski commented Apr 24, 2026

Slot-aware batch polling and graceful shutdown improvements

Concurrency-aware polling

The batch processor now checks available execution slots before polling for new messages. If no slots are free, it waits for one to open rather than polling and discarding work. The poller's poll() method accepts an optional limit parameter so it fetches only as many messages as there are open slots, preventing over-fetching when the queue is partially saturated.

Slot notification mechanism

ExecutionController exposes an availableSlots getter and a waitForSlot() method. When a task finishes, it calls notifySlotWaiters() to wake any processor waiting for capacity. This allows a new poll cycle to begin as soon as a slot frees up — before the slowest task in the previous batch completes — rather than waiting for the entire batch to drain.

Shutdown signal propagation

Worker now accepts an optional requestShutdown callback. When the worker begins stopping, it calls this callback before aborting its internal controller, allowing the platform adapter's shared abort signal to be triggered first. This ensures pollers, executors, and handler contexts all see the abort signal promptly. SupabasePlatformAdapter.stopWorker() is updated to drain the worker before closing the SQL connection, and a new requestShutdown() method is added to the PlatformAdapter interface.

Test coverage

  • Integration test verifying that a "refill" task starts after a fast task frees its slot but before the slow task from the same batch finishes, for both queue workers and map-flow workers.
  • Integration test confirming that worker.stop() aborts the handler's shutdownSignal and completes within a tight time bound.
  • Unit test asserting that stopWorker calls abort, then worker.stop, then sql.end in that order.

Documentation

Added a Self-Hosted Supabase deployment guide covering Postgres image upgrades for pgmq compatibility, EDGE_WORKER_DB_URL configuration via Supavisor, manual function deployment, and migration application. Updated the manual installation guide to reflect that the Control Plane function is no longer required and that self-hosted deployments are a primary use case.

@changeset-bot
Copy link
Copy Markdown

changeset-bot Bot commented Apr 24, 2026

🦋 Changeset detected

Latest commit: 34180f2

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 5 packages
Name Type
@pgflow/edge-worker Patch
@pgflow/core Patch
@pgflow/dsl Patch
@pgflow/client Patch
pgflow Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Copy link
Copy Markdown
Contributor Author

jumski commented Apr 24, 2026

This stack of pull requests is managed by Graphite. Learn more about stacking.

@nx-cloud
Copy link
Copy Markdown

nx-cloud Bot commented Apr 24, 2026

View your CI Pipeline Execution ↗ for commit 34180f2

Command Status Duration Result
nx run edge-worker:test:integration ✅ Succeeded 4m 23s View ↗
nx run edge-worker:e2e ✅ Succeeded 48s View ↗
nx affected -t verify-exports --base=origin/mai... ✅ Succeeded <1s View ↗
nx affected -t build --configuration=production... ✅ Succeeded <1s View ↗
nx affected -t lint typecheck test --parallel -... ✅ Succeeded 21s View ↗

☁️ Nx Cloud last updated this comment at 2026-04-24 15:13:16 UTC

@jumski jumski force-pushed the feat/slot-aware-backpressure branch from 86c664b to 34180f2 Compare April 24, 2026 15:05
@jumski jumski deployed to preview April 24, 2026 15:13 — with GitHub Actions Active
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant