Skip to content

docs(design): multi-producer WorkQueue for multi-root transfers#3648

Merged
oferchen merged 1 commit into
masterfrom
docs/multi-producer-workqueue-1382
May 5, 2026
Merged

docs(design): multi-producer WorkQueue for multi-root transfers#3648
oferchen merged 1 commit into
masterfrom
docs/multi-producer-workqueue-1382

Conversation

@oferchen
Copy link
Copy Markdown
Owner

@oferchen oferchen commented May 5, 2026

Summary

Design note for #1382 covering a multi-producer extension to the bounded
delta WorkQueue so multi-root transfers (oc-rsync foo/ bar/ baz/ dst/)
and parallel directory enumeration (#1573) can fan in to one shared queue.

Key sections

  • Problem statement, citing the serial multi-root loop at
    crates/engine/src/local_copy/executor/sources/orchestration.rs:79
    and the SPMC contract at
    crates/engine/src/concurrent_delta/work_queue/bounded.rs:11-21.
  • Three multi-producer designs (Clone, Arc-wrapped, producer pool)
    evaluated against today's crossbeam-backed channel.
  • Recommended API: with_producers(n) returning a Vec<WorkQueueSender>
    that surfaces the producer-count contract in the type system.
  • Ordering via per-producer sequence ranges, with the atomic-counter
    fallback already validated by the existing test
    multi_producer_requires_atomic_sequence_coordination.
  • Backpressure, drain_parallel interaction (auto-close on last sender
    drop), migration plan for multi-root, --files-from, and future async
    producers, plus risks (deadlock, sequence collisions, fairness,
    adaptive policy).

Wire-compat invariants

Zero impact. WorkQueue is internal engine coordination only; no
protocol bytes change. The wire-protocol receive path keeps its
single-producer contract documented in
crates/engine/src/concurrent_delta/multi_producer_audit.rs:8-19.

Test plan

  • Doc lint: 536 lines (within the 400-650 target).
  • git diff origin/master..HEAD confirms only
    docs/design/multi-producer-workqueue.md is added.
  • CI: docs-only change, no code surface modified.

Design note for #1382. Surveys the current SPMC contract on the bounded
delta WorkQueue, evaluates three multi-producer designs, and recommends a
Clone-based fan-in API (with_producers(n)) that surfaces the
producer-count contract as a Vec of N senders. Covers ordering via
per-producer sequence ranges, backpressure, drain_parallel interaction,
and migration for multi-root transfers, --files-from + walker
concurrency, and future async producers. Wire-compat impact: zero
(channels are internal coordination only).
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label May 5, 2026
@oferchen oferchen merged commit 95c0f9b into master May 5, 2026
8 checks passed
@oferchen oferchen deleted the docs/multi-producer-workqueue-1382 branch May 5, 2026 09:59
oferchen added a commit that referenced this pull request May 18, 2026
Design note for #1382. Surveys the current SPMC contract on the bounded
delta WorkQueue, evaluates three multi-producer designs, and recommends a
Clone-based fan-in API (with_producers(n)) that surfaces the
producer-count contract as a Vec of N senders. Covers ordering via
per-producer sequence ranges, backpressure, drain_parallel interaction,
and migration for multi-root transfers, --files-from + walker
concurrency, and future async producers. Wire-compat impact: zero
(channels are internal coordination only).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant