Skip to content

feat: multi-process prefork worker pool#33

Merged
pratyush618 merged 5 commits intomasterfrom
feat/prefork-worker-pool
Mar 21, 2026
Merged

feat: multi-process prefork worker pool#33
pratyush618 merged 5 commits intomasterfrom
feat/prefork-worker-pool

Conversation

@pratyush618
Copy link
Copy Markdown
Collaborator

Summary

  • Prefork worker pool: queue.run_worker(pool="prefork", app="myapp:queue") spawns child Python processes with independent GILs for true CPU parallelism
  • Rust PreforkPool: implements WorkerDispatcher trait, dispatches serialized jobs over stdin pipes, reads results from stdout pipes
  • JSON Lines IPC: ParentMessage (job/shutdown) and ChildMessage (success/failure/cancelled/ready) over stdio
  • Child process: python -m taskito.prefork <app_path> — imports app, builds task registry, executes tasks in a read-execute-write loop
  • Least-loaded dispatch: jobs sent to the child with fewest in-flight tasks
  • Graceful shutdown: sends shutdown message to children, waits with timeout, kills if needed

Architecture

Parent: Scheduler → PreforkPool → stdin pipes → Children
Children: stdin → execute task → stdout → Reader threads → result_tx → Parent

Test plan

  • cargo check all feature combos
  • cargo test --workspace — 43 Rust tests pass
  • uv run python -m pytest tests/python/ -v — 352 pass, 9 skipped
  • Prefork child process starts and responds to jobs manually
  • pool="prefork" validates app= is required
  • pool="thread" (default) unchanged

@pratyush618 pratyush618 merged commit 5c9a025 into master Mar 21, 2026
10 checks passed
@pratyush618 pratyush618 deleted the feat/prefork-worker-pool branch March 31, 2026 17:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant