[Feature Request] Run-to-Completion (RTC) Worker Loop Support

[MalikHou]

[Feature Request] Run-to-Completion (RTC) Worker Loop + User Hooks + Optional io_uring Backend

Background

We run brpc in a per-thread-per-core deployment where worker threads are CPU-pinned and also interact with a high-performance I/O stack (SPDK or io_uring). In this model, there are scheduling “gaps” inside the worker loop (e.g., after running a batch of runnable bthreads and before the worker blocks in epoll_wait). Those gaps are ideal to execute small, non-blocking work on the same worker thread to preserve locality and avoid cross-thread handoff.

Concrete examples of work we want to run in these gaps:

• Harvest/poll SPDK completions (poll-mode style)
• Drain io_uring CQEs and advance state machines
• Convert completions into runnable units and push them back to the worker’s local queue

Today, integrating these cleanly requires extra polling threads, periodic timers, or cross-thread scheduling, which increases tail latency and adds contention.

Problem

1. No first-class “run user logic on the current worker thread” hook points

   • Users cannot reliably execute a lightweight function at well-defined points in the worker loop (before/after waiting, between batches, etc.).

2. No supported way to reschedule derived work back into the local run queue

   • For completion-driven workloads, the critical requirement is: harvest completions on the worker thread and immediately enqueue follow-up work to the same worker’s local queue. Global queues/cross-thread wakeups increase contention and latency.

3. epoll-centric waiting is not ideal for io_uring/SPDK-centric stacks

   • For io_uring, batching submit/completion is the natural model and can reduce syscalls.
   • For SPDK, polling is the natural model and requires a bounded busy loop or hybrid strategy.

Requirements

R1. Run-to-Completion (RTC) style worker execution

Provide an opt-in worker model that repeatedly:

• runs runnable bthreads until a budget is reached,
• runs registered “maintenance” hooks (budgeted),
• waits for events (blocking/polling/hybrid),
• repeats.

R2. epoll polling / hybrid mode

Support configurable event-wait behavior for the epoll backend:

• blocking (default): epoll_wait(timeout > 0)
• polling: epoll_wait(timeout = 0) in a bounded busy loop
• hybrid: poll for N iterations or T microseconds, then fall back to blocking

R3. User hook registration on worker threads

Provide an API to register callbacks that execute on the current brpc worker thread at defined hook points. Hooks must be:

• non-blocking (no sleeping, no long syscalls)
• budgeted (time/ops limited)
• safe to run frequently

R4. Ability to enqueue work back to the worker’s local queue

Hooks must be able to take harvested completions and schedule follow-up work back to the worker’s local run queue (not global), e.g.:

• enqueue an existing bthread/task handle locally
• or submit a closure that is executed by the same worker

R5. Optional io_uring backend replacing epoll

Support selecting an io_uring event backend as an alternative to epoll (opt-in), to:

• reduce syscalls via batching
• align with per-thread-per-core (one ring per worker)
• enable future enhancements (multishot, buffer rings) without forcing epoll