Bug Description
Code review found a scheduler resize/lifecycle race in runtime::scheduler::set_thread_count(). When shrinking the worker pool, a worker whose io_context().pending_count() == 0 is stopped and joined. However, worker_thread::run() enters a drain phase after running_ becomes false and continues resuming queued coroutine tasks on the retiring worker.
If one of those drained tasks starts a new async I/O operation, it binds to the retiring worker's io_context. The worker can then exit, leaving that newly submitted I/O without a polling worker. The suspended coroutine may never resume.
Relevant code paths:
include/elio/runtime/scheduler.hpp: shrink path calls workers_[i]->stop() when pending_count() == 0, then redistribute_tasks().
worker_thread::run() drain phase continues calling run_task(handle) after running_ is false.
- I/O awaitables bind to
runtime::worker_thread::current()->io_context().
Environment
Please complete the following information:
- Elio Version: 0.5.2 (
CMakeLists.txt project version)
- Compiler: Not compiler-specific; found by code review
- OS: Linux
- Kernel Version: Not kernel-specific
- Build Type: Not build-type-specific
- CMake Version: Not applicable
Steps to Reproduce
This was found by code review. A likely runtime scenario is:
- Start a scheduler with more than one worker.
- Queue a coroutine on a worker that has no pending I/O yet.
- Shrink the scheduler so that worker is retired.
- During the worker's shutdown drain phase, resume the queued coroutine.
- The coroutine starts a new async operation such as
co_await tcp.read(...) or co_await time::sleep_for(...).
- The retiring worker exits; the new operation is attached to an
io_context that is no longer polled.
Minimal Reproducible Example
No executable reproducer yet. Sketch of the shape:
#include <elio/elio.hpp>
using namespace elio;
coro::task<void> starts_io_during_retire() {
// Needs to be resumed during the retiring worker's drain phase.
co_await time::sleep_for(std::chrono::milliseconds(10));
}
int main() {
runtime::scheduler sched(2);
sched.start();
sched.go(starts_io_during_retire);
sched.set_thread_count(1);
sched.shutdown();
}
A robust test likely needs worker affinity or instrumentation to force the task onto the worker being retired.
Expected Behavior
Shrinking the scheduler should not leave tasks or I/O operations attached to an io_context that no worker will continue polling.
Actual Behavior
The current drain phase can execute queued tasks on a retiring worker after running_ is false. Those tasks can submit fresh I/O to the retiring worker's io_context, after the shrink path already decided there was no pending I/O to drain.
Error Messages
None. This is a potential hang/silent leak rather than a direct error path.
ASAN/TSAN Output
Not available. This was found by code review, not by sanitizer execution.
Additional Context
The shrink path already has a separate draining-worker mode for workers that have pending I/O. The problematic case is when pending_count() == 0 before stop(), but drain-phase task execution creates new pending I/O afterward.
Possible approaches:
- Do not resume arbitrary queued tasks on a worker being retired; redistribute queued tasks before stopping it.
- Add a stop mode for shrink that drains/destroys or migrates queue entries without running user coroutine bodies on the retiring worker.
- Make I/O awaitables refuse to bind to a worker that is in shutdown/draining retirement mode and reschedule to an active worker instead.
Related Issues
- None found in currently open issues.
Possible Solution
// Conceptual direction only:
// 1. Mark worker as retiring / not accepting local execution.
// 2. Stop stealing from it and stop external scheduling to it.
// 3. Move queued handles to active workers before joining.
// 4. Keep the special I/O-draining path only for operations that were already pending.
Checklist
Bug Description
Code review found a scheduler resize/lifecycle race in
runtime::scheduler::set_thread_count(). When shrinking the worker pool, a worker whoseio_context().pending_count() == 0is stopped and joined. However,worker_thread::run()enters a drain phase afterrunning_becomes false and continues resuming queued coroutine tasks on the retiring worker.If one of those drained tasks starts a new async I/O operation, it binds to the retiring worker's
io_context. The worker can then exit, leaving that newly submitted I/O without a polling worker. The suspended coroutine may never resume.Relevant code paths:
include/elio/runtime/scheduler.hpp: shrink path callsworkers_[i]->stop()whenpending_count() == 0, thenredistribute_tasks().worker_thread::run()drain phase continues callingrun_task(handle)afterrunning_is false.runtime::worker_thread::current()->io_context().Environment
Please complete the following information:
CMakeLists.txtproject version)Steps to Reproduce
This was found by code review. A likely runtime scenario is:
co_await tcp.read(...)orco_await time::sleep_for(...).io_contextthat is no longer polled.Minimal Reproducible Example
No executable reproducer yet. Sketch of the shape:
A robust test likely needs worker affinity or instrumentation to force the task onto the worker being retired.
Expected Behavior
Shrinking the scheduler should not leave tasks or I/O operations attached to an
io_contextthat no worker will continue polling.Actual Behavior
The current drain phase can execute queued tasks on a retiring worker after
running_is false. Those tasks can submit fresh I/O to the retiring worker'sio_context, after the shrink path already decided there was no pending I/O to drain.Error Messages
None. This is a potential hang/silent leak rather than a direct error path.
ASAN/TSAN Output
Not available. This was found by code review, not by sanitizer execution.
Additional Context
The shrink path already has a separate draining-worker mode for workers that have pending I/O. The problematic case is when
pending_count() == 0beforestop(), but drain-phase task execution creates new pending I/O afterward.Possible approaches:
Related Issues
Possible Solution
Checklist