Replace spawn-on-demand pool servicing with a persistent scheduler actor

[leynos]

Summary

Replace the pool scheduler's Arc<Mutex<SchedulerState>> plus atomic service guard and transient tokio::spawn calls with one long-lived scheduler task that owns the fairness and waiter state.

This is the final structural implementation slice for ADR 013 proposal #638. It follows the stable slot graph in #645 and should consume the scoped permit-selection API from #646.

Current behaviour

The current scheduler:

• shares SchedulerState through a synchronous mutex;
• uses AtomicBool::is_servicing as a distributed task-lifecycle guard;
• calls kick from acquire and lease-drop paths;
• spawns service_waiters when the guard transitions from idle;
• may spawn a task that immediately discovers no waiter and exits;
• divides shutdown, waiter selection, restart recovery, and capacity notification among several caller contexts.

This topology makes one conceptual state machine look like shared mutable data. It also complicates #550's bounded-waiter admission and leaves adversarial interleavings from #593/#535 difficult to test directly.

Proposed architecture

Create one scheduler task during pool construction. It owns SchedulerState directly and receives commands through a Tokio channel.

A representative protocol is:

enum SchedulerCommand<S, P, C> {
    Acquire {
        handle_id: u64,
        reply: oneshot::Sender<Result<PooledClientLease<S, P, C>, ClientError>>,
    },
    DeregisterHandle {
        handle_id: u64,
    },
    CapacityAvailable,
    Shutdown {
        reply: oneshot::Sender<()>,
    },
}

The exact types may differ, but the protocol and ownership must remain explicit.

Ownership split

Keep scheduler control and physical slot storage separate:

• Arc<PoolCore> from Introduce PoolCore and index-based pooled leases #645 owns fixed slots and shared physical resources;
• a cloneable SchedulerHandle owns only the command sender and synchronous handle-ID allocation if retained;
• SchedulerTask owns the receiver, SchedulerState, fairness rotation, waiter counts, and shutdown state;
• the task may own Arc<PoolCore>, but PoolCore must not own the command sender in a way that creates a strong cycle.

Dropping the final public scheduler sender must eventually close the mailbox, terminate the task, and release the task's pool-core reference.

Responsibilities of the actor

Admission

• reject new acquire commands after shutdown begins;
• enforce the waiter bound and/or acquisition deadline required by [security][low] Client pool admits unbounded blocked waiters under backpressure #550;
• return a typed error when admission fails;
• prune cancelled oneshot replies so abandoned callers do not consume waiter capacity indefinitely.

Fairness

• preserve FIFO arrival order for live admitted waiters;
• preserve round-robin turns among logical handles;
• handle deregistration idempotently;
• never grant a lease to a cancelled or deregistered waiter.

Capacity

• attempt an immediate grant when an acquire command arrives and capacity exists;
• wait for capacity without spawning a task per lease release;
• use Race borrowed pool-slot permits without per-slot Arc cloning #646's slot-index/permit selection;
• treat duplicate CapacityAvailable messages as harmless hints, not extra permits.

Shutdown

• reject later acquires;
• resolve every queued waiter with ClientError::disconnected or the chosen typed shutdown error;
• cancel or drop any pending permit-selection future safely;
• acknowledge explicit close() only after scheduler state has drained and the actor can terminate;
• terminate on mailbox closure even when callers omit explicit close.

Public API behaviour

Preserve existing WireframeClientPool, PoolHandle, and PooledClientLease APIs where practical.

WireframeClientPool::handle() may remain synchronous by allocating a logical ID outside the actor. The implementation must prove command ordering for acquire and deregistration from one logical handle. A registration acknowledgement is acceptable if the public API change is justified, but do not quietly assume cross-sender global ordering that Tokio does not promise.

WireframeClientPool::close(self) becomes an awaitable scheduler shutdown rather than shutdown(); yield_now(); drop(self).

Drop behaviour

PooledClientLease::drop cannot await. It should:

• release its OwnedSemaphorePermit by ordinary drop;
• send or coalesce a non-blocking capacity hint to the actor;
• tolerate an already-closed scheduler mailbox during shutdown;
• avoid cloning a large ownership graph;
• never spawn a task.

If permit release can be observed without a per-drop command, for example through a shared Notify polled by the actor, that is acceptable if it preserves the single-owner scheduler state and has no lost-wakeup race.

Integration with existing issues

• Simplify client pool waiter state transitions #535: reuse its state-transition simplification inside the actor or close it as superseded with an explicit explanation.
• Add pool poison recovery integration coverage #539: reassess whether scheduler poison-recovery integration remains relevant once scheduler state is no longer behind std::sync::Mutex; retain slot-bookkeeping scope if still applicable.
• [security][medium] Cancelled pooled client operations can return dirty sockets as reusable #548: do not grant reuse of a dirty physical connection merely because capacity returned. Cancellation safety remains a separate invariant.
• [security][low] Client pool admits unbounded blocked waiters under backpressure #550: implement bounded waiter admission in the actor rather than adding another shared counter outside it.
• Mutation testing: client pool lifecycle and slot rotation survivors #593: preserve and extend shutdown, rotation, and waiter behavioural tests.

Acceptance criteria

• Exactly one scheduler task is created per client pool.
• Repeated acquire/drop operations create no additional scheduler tasks.
• SchedulerState is owned by the actor and is not behind Arc<Mutex<_>>.
• The is_servicing atomic/spawn-restart protocol is removed.
• Acquire, deregister, capacity, and shutdown transitions flow through an explicit command or wake protocol.
• FIFO and round-robin semantics match the existing public contract.
• Waiter admission is bounded according to [security][low] Client pool admits unbounded blocked waiters under backpressure #550.
• Cancelled acquire futures are pruned and receive no lease.
• Blocked waiters resolve promptly during shutdown.
• close() waits for scheduler shutdown rather than yielding speculatively.
• Dropping all public pool/control handles without close() terminates the actor and releases PoolCore.
• Lease drop never calls tokio::spawn.
• No strong-reference cycle retains the pool.
• [security][medium] Cancelled pooled client operations can return dirty sockets as reusable #548's dirty-socket invariant is preserved.
• Establish runtime ownership and task-churn baselines #639 benchmarks record uncontended/contended latency, allocations, and task-count changes.

Tests

Add deterministic state-machine and runtime tests covering:

• immediate uncontended grant;
• FIFO ordering under saturation;
• round-robin ordering across several logical handles;
• cancelled waiter before and during grant;
• handle deregistration with queued waiters;
• bounded waiter rejection;
• capacity release racing with acquire and shutdown;
• explicit close with blocked waiters;
• implicit shutdown when all senders drop;
• actor task count remaining one under a long acquire/drop loop;
• no PoolCore retention after final handle/lease drop.

Use loom or a small pure scheduler-state model for event-order races that cannot be forced reliably with wall-clock Tokio tests.

Non-goals

• Moving socket request/response I/O into the scheduler actor.
• Dynamically resizing the slot collection.
• Replacing Tokio channels or semaphores.
• Weakening cancellation safety to simplify capacity accounting.

Dependencies

• ADR 013 proposal ADR 013 proposal: Single-owner client-pool scheduler and slot graph #638.
• Baseline issue Establish runtime ownership and task-churn baselines #639.
• Pool core Introduce PoolCore and index-based pooled leases #645.
• Scoped permit selection Race borrowed pool-slot permits without per-slot Arc cloning #646.

References

• Epic Epic: Align Tokio task boundaries with ownership boundaries #635
• ADR 011 proposal ADR 011 proposal: Runtime ownership and task-lifetime boundaries #636
• ADR 013 proposal ADR 013 proposal: Single-owner client-pool scheduler and slot graph #638
• Simplify client pool waiter state transitions #535
• Add pool poison recovery integration coverage #539
• [security][medium] Cancelled pooled client operations can return dirty sockets as reusable #548
• [security][low] Client pool admits unbounded blocked waiters under backpressure #550
• Mutation testing: client pool lifecycle and slot rotation survivors #593
• Establish runtime ownership and task-churn baselines #639
• Introduce PoolCore and index-based pooled leases #645
• Race borrowed pool-slot permits without per-slot Arc cloning #646
• src/client/pool/scheduler.rs
• src/client/pool/client_pool.rs
• src/client/pool/handle.rs
• src/client/pool/lease.rs

[coderabbitai]

🔗 Related PRs

#540 - Approve the actor and codec-driver boundary (10.1.3) [merged]
leynos/pg-embed-setup-unpriv#157 - Add verification tests for lifecycle invariants (#152) [merged]
#552 - Align roadmap footnotes with the roadmap grammar [open]
#612 - Kill mutation-testing survivors and widen scaffolding excludes (#566, #567, #569, #570, #593, #595, #598) [merged]
leynos/cuprum#232 - docs: mark #125 FD-ownership hazard resolved in roadmap 8.1 (#125) [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!