Skip to content

[Bug] Scheduler shrink can orphan I/O started by retired workers during drain #244

Description

@Coldwings

Bug Description

Code review found a scheduler resize/lifecycle race in runtime::scheduler::set_thread_count(). When shrinking the worker pool, a worker whose io_context().pending_count() == 0 is stopped and joined. However, worker_thread::run() enters a drain phase after running_ becomes false and continues resuming queued coroutine tasks on the retiring worker.

If one of those drained tasks starts a new async I/O operation, it binds to the retiring worker's io_context. The worker can then exit, leaving that newly submitted I/O without a polling worker. The suspended coroutine may never resume.

Relevant code paths:

  • include/elio/runtime/scheduler.hpp: shrink path calls workers_[i]->stop() when pending_count() == 0, then redistribute_tasks().
  • worker_thread::run() drain phase continues calling run_task(handle) after running_ is false.
  • I/O awaitables bind to runtime::worker_thread::current()->io_context().

Environment

Please complete the following information:

  • Elio Version: 0.5.2 (CMakeLists.txt project version)
  • Compiler: Not compiler-specific; found by code review
  • OS: Linux
  • Kernel Version: Not kernel-specific
  • Build Type: Not build-type-specific
  • CMake Version: Not applicable

Steps to Reproduce

This was found by code review. A likely runtime scenario is:

  1. Start a scheduler with more than one worker.
  2. Queue a coroutine on a worker that has no pending I/O yet.
  3. Shrink the scheduler so that worker is retired.
  4. During the worker's shutdown drain phase, resume the queued coroutine.
  5. The coroutine starts a new async operation such as co_await tcp.read(...) or co_await time::sleep_for(...).
  6. The retiring worker exits; the new operation is attached to an io_context that is no longer polled.

Minimal Reproducible Example

No executable reproducer yet. Sketch of the shape:

#include <elio/elio.hpp>

using namespace elio;

coro::task<void> starts_io_during_retire() {
    // Needs to be resumed during the retiring worker's drain phase.
    co_await time::sleep_for(std::chrono::milliseconds(10));
}

int main() {
    runtime::scheduler sched(2);
    sched.start();
    sched.go(starts_io_during_retire);
    sched.set_thread_count(1);
    sched.shutdown();
}

A robust test likely needs worker affinity or instrumentation to force the task onto the worker being retired.

Expected Behavior

Shrinking the scheduler should not leave tasks or I/O operations attached to an io_context that no worker will continue polling.

Actual Behavior

The current drain phase can execute queued tasks on a retiring worker after running_ is false. Those tasks can submit fresh I/O to the retiring worker's io_context, after the shrink path already decided there was no pending I/O to drain.

Error Messages

None. This is a potential hang/silent leak rather than a direct error path.

ASAN/TSAN Output

Not available. This was found by code review, not by sanitizer execution.

Additional Context

The shrink path already has a separate draining-worker mode for workers that have pending I/O. The problematic case is when pending_count() == 0 before stop(), but drain-phase task execution creates new pending I/O afterward.

Possible approaches:

  • Do not resume arbitrary queued tasks on a worker being retired; redistribute queued tasks before stopping it.
  • Add a stop mode for shrink that drains/destroys or migrates queue entries without running user coroutine bodies on the retiring worker.
  • Make I/O awaitables refuse to bind to a worker that is in shutdown/draining retirement mode and reschedule to an active worker instead.

Related Issues

  • None found in currently open issues.

Possible Solution

// Conceptual direction only:
// 1. Mark worker as retiring / not accepting local execution.
// 2. Stop stealing from it and stop external scheduling to it.
// 3. Move queued handles to active workers before joining.
// 4. Keep the special I/O-draining path only for operations that were already pending.

Checklist

  • I have searched existing issues to avoid duplicates
  • I have provided all requested information available from code review
  • I have included a minimal reproducible example (if possible)
  • I have tested with the latest version of Elio
  • I have included sanitizer output (for memory/threading issues)

Metadata

Metadata

Labels

bugSomething isn't working

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions