time: add alternative timer for better multicore scalability by ADD-SP · Pull Request #7467 · tokio-rs/tokio

ADD-SP · 2025-07-19T08:50:37Z

Review guide

Please checkout Reduce contention in timer driver #6504 for the lock contention issue.
Please checkout RFC: Delayed cancellation of timer #7384 for high-level design.
Design document
Benchmarks

Signed-off-by: ADD-SP qiqi.zhang@konghq.com

ADD-SP · 2025-07-19T13:31:47Z

tokio-util/tests/time_delay_queue.rs

 }

-#[tokio::test]
-async fn reset_later_after_slot_starts() {


Note

This test was removed because it was testing the behavior of tokio::runtime::time::entry::reset, which is removed in this PR.

ADD-SP · 2025-07-19T13:31:54Z

tokio-util/tests/time_delay_queue.rs

 }

-#[tokio::test]
-async fn reset_earlier_after_slot_starts() {


Note

This test was removed because it was testing the behavior of tokio::runtime::time::entry::reset, which is removed in this PR.

ADD-SP · 2025-07-19T13:41:16Z

tokio-util/tests/time_delay_queue.rs

    sleep(ms(20)).await;

-    assert!(queue.is_woken());
+    assert_ready_some!(poll!(queue));


Note

queue.reset_at resets inner Sleep, however, the new implementation will drop the inner timer and create a new one. So the waker will not be called, we have to poll manually.

ADD-SP · 2025-07-19T13:41:37Z

tokio-util/tests/time_delay_queue.rs

    sleep(ms(20)).await;

-    assert!(queue.is_woken());
+    assert_ready_some!(poll!(queue));


Note

queue.reset_at resets inner Sleep, however, the new implementation will drop the inner timer and create a new one. So the waker will not be called, we have to poll manually.

tokio/src/runtime/scheduler/current_thread/mod.rs

tokio/src/runtime/scheduler/multi_thread/worker.rs

ADD-SP · 2025-07-20T09:18:06Z

tokio/src/util/mod.rs

    feature = "signal",
-    feature = "time",
 ))]
 pub(crate) use wake_list::WakeList;


Note

We no long use WakeList in the time subsystem.

tokio/src/runtime/time/mod.rs

tokio/src/runtime/time/wheel/entry.rs

tokio/src/runtime/time_alt/entry.rs

tokio/src/runtime/time/wheel/entry.rs

tokio/src/runtime/time_alt/entry.rs

tokio/src/runtime/scheduler/util.rs

tokio/src/runtime/time/tests/mod.rs

Signed-off-by: ADD-SP <qiqi.zhang@konghq.com>

…e` macros this enables rustfmt to fix the format Signed-off-by: ADD-SP <qiqi.zhang@konghq.com>

Signed-off-by: ADD-SP <qiqi.zhang@konghq.com>

Darksonn · 2025-11-20T23:32:12Z

It would be good to check whether this has the same bug as #6682, since this also introduces sharding of timers.

carllerche · 2025-11-21T17:09:59Z

Thanks to both @ADD-SP and @Darksonn for sticking with this. I know it is a big complex one.

I commented in the issue that I am fine w/ the benchmark perf tradeoff.

This change is a really big one that is fundamentally changing a core part of Tokio. We can't just just drop it in all at once. What I suggest is we take a phased approach, similar to what I did with the alt multi-threaded runtime. The alt multi-threaded runtime ultimately didn't land, but that is actually a success IMO because a) bugs were uncovered after merging it that were not during testing / loom and b) I was able to get real services to test the runtime and was not able to demonstrate any real-world benefits (micro benches can be decoupled from reality).

So, what I would suggest is to start a new PR with the code from this one.

Keep the old timer code as the default timer
Add your timer code as a completely separate timer implementation.
Use an enum for the runtime to pick between which one is enabled (similar to the alt MT runtime)
Only enable the new timer strategy with tokio_unstable.
Use a runtime configuration setting (only available with tokio_unstable) to opt into the new timer implementation.

Then we can merge the PR and start the next phase of testing: getting real service teams to try it out. Having both available in a released version of Tokio makes this testing easier. You can use deployment configuration to only use the new timer code in a fraction of the production servers. This reduces risk and lets us collect data.

Assuming the first phase works, we can then consider swapping the default timer implementation to the new one when tokio_unstable is enabled, followed by making it available to all.

Also, I think being able to pick from timer implementations is a long term win. For example, we may want to have a "current-thread" optimized timer, and/or variants that are only very coarse. For example, maybe @Noah-Kennedy / and other pingora maintainers could bring in what they need to a timer variant.

ADD-SP · 2025-11-23T10:57:02Z

It would be good to check whether this has the same bug as #6682, since this also introduces sharding of timers.

Thanks for this context! @Darksonn

The root cause of #6682 is that next_wake is not up to date when a timer is reregistered. In this PR, we don’t have this issue because:

When the local wheel is available, the existing worker thread is still running, so we don’t need next_wake to decide whether to unpark the resource driver.
When the local wheel is not available, push_remote_timer always calls notify_parked_remote.

Signed-off-by: ADD-SP <qiqi.zhang@konghq.com>

…ting timer impl Signed-off-by: ADD-SP <qiqi.zhang@konghq.com>

Signed-off-by: ADD-SP <qiqi.zhang@konghq.com>

Darksonn

Overall looks reasonable to me.

tokio/src/time/interval.rs

tokio/src/runtime/scheduler/multi_thread/worker.rs

tokio/src/runtime/scheduler/mod.rs

Darksonn · 2025-11-26T09:21:44Z

@carllerche Good call to make it optional. I think it's a lot easier to merge this PR now. There are a few pieces of test code remaining and such, but once that's removed I think we can ship it as an optional configuration.

Signed-off-by: ADD-SP <qiqi.zhang@konghq.com>

Bumps tokio from 1.48.0 to 1.49.0. Release notes Sourced from tokio's releases. Tokio v1.49.0 1.49.0 (January 3rd, 2026) Added net: add support for TCLASS option on IPv6 (#7781) runtime: stabilize runtime::id::Id (#7125) task: implement Extend for JoinSet (#7195) task: stabilize the LocalSet::id() (#7776) Changed net: deprecate {TcpStream,TcpSocket}::set_linger (#7752) Fixed macros: fix the hygiene issue of join! and try_join! (#7766) runtime: revert "replace manual vtable definitions with Wake" (#7699) sync: return TryRecvError::Disconnected from Receiver::try_recv after Receiver::close (#7686) task: remove unnecessary trait bounds on the Debug implementation (#7720) Unstable fs: handle EINTR in fs::write for io-uring (#7786) fs: support io-uring with tokio::fs::read (#7696) runtime: disable io-uring on EPERM (#7724) time: add alternative timer for better multicore scalability (#7467) Documented docs: fix a typos in bounded.rs and park.rs (#7817) io: add SyncIoBridge cross-references to copy and copy_buf (#7798) io: doc that AsyncWrite does not inherit from std::io::Write (#7705) metrics: clarify that num_alive_tasks is not strongly consistent (#7614) net: clarify the cancellation safety of the TcpStream::peek (#7305) net: clarify the drop behavior of unix::OwnedWriteHalf (#7742) net: clarify the platform-dependent backlog in TcpSocket docs (#7738) runtime: mention LocalRuntime in new_current_thread docs (#7820) sync: add missing period to mpsc::Sender::try_send docs (#7721) sync: clarify the cancellation safety of oneshot::Receiver (#7780) sync: improve the docs for the errors of mpsc (#7722) task: add example for spawn_local usage on local runtime (#7689) #7125: tokio-rs/tokio#7125 #7195: tokio-rs/tokio#7195 #7305: tokio-rs/tokio#7305 #7467: tokio-rs/tokio#7467 #7614: tokio-rs/tokio#7614 #7686: tokio-rs/tokio#7686 #7689: tokio-rs/tokio#7689 ... (truncated) Commits e3b89bb chore: prepare Tokio v1.49.0 (#7824) 4f577b8 Merge 'tokio-1.47.3' into 'master' f320197 chore: prepare Tokio v1.47.3 (#7823) ea6b144 ci: freeze rustc on nightly-2025-01-25 in netlify.toml (#7652) 264e703 Merge tokio-1.43.4 into tokio-1.47.x (#7822) dfb0f00 chore: prepare Tokio v1.43.4 (#7821) 4a91f19 ci: fix wasm32-wasip1 tests (#7788) 601c383 ci: upgrade FreeBSD from 14.2 to 14.3 (#7758) 484cb52 sync: return TryRecvError::Disconnected from Receiver::try_recv after `Re... 16f20c3 rt: mention LocalRuntime in new_current_thread docs (#7820) Additional commits viewable in compare view Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase. Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: @dependabot rebase will rebase this PR @dependabot recreate will recreate this PR, overwriting any edits that have been made to it @dependabot merge will merge this PR after your CI passes on it @dependabot squash and merge will squash and merge this PR after your CI passes on it @dependabot cancel merge will cancel a previously requested merge and block automerging @dependabot reopen will reopen this PR if it is closed @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

ADD-SP changed the title ~~[WIP]~~ [WIP] time: delayed cancellation of timers Jul 19, 2025

ADD-SP force-pushed the add_sp/time-local-wheel branch 11 times, most recently from c2d5790 to d04c22f Compare July 19, 2025 13:19

ADD-SP changed the title ~~[WIP] time: delayed cancellation of timers~~ [WIP] time: delay the cancellation of timers Jul 19, 2025

ADD-SP commented Jul 19, 2025

View reviewed changes

tokio/src/runtime/scheduler/current_thread/mod.rs Outdated Show resolved Hide resolved

ADD-SP commented Jul 19, 2025

View reviewed changes

tokio/src/runtime/scheduler/multi_thread/worker.rs Outdated Show resolved Hide resolved

ADD-SP force-pushed the add_sp/time-local-wheel branch 2 times, most recently from 695eea8 to e9255ee Compare July 20, 2025 08:21

ADD-SP mentioned this pull request Jul 20, 2025

time: delay the Arc::clone of the scheduler::Handle until registering timer #7461

Closed

ADD-SP commented Jul 20, 2025

View reviewed changes

tokio/src/runtime/time/mod.rs Outdated Show resolved Hide resolved

Darksonn reviewed Nov 19, 2025

View reviewed changes

tokio/src/runtime/time/wheel/entry.rs Outdated Show resolved Hide resolved

tokio/src/runtime/time_alt/entry.rs Show resolved Hide resolved

tokio/src/runtime/time/wheel/entry.rs Outdated Show resolved Hide resolved

tokio/src/runtime/time_alt/entry.rs Show resolved Hide resolved

Darksonn reviewed Nov 19, 2025

View reviewed changes

tokio/src/runtime/scheduler/util.rs Outdated Show resolved Hide resolved

Darksonn reviewed Nov 19, 2025

View reviewed changes

tokio/src/runtime/time/tests/mod.rs Show resolved Hide resolved

ADD-SP added 6 commits November 19, 2025 20:28

add safety comments in entry.rs

aebf2b3

Signed-off-by: ADD-SP <qiqi.zhang@konghq.com>

tolerate spurious wakeup

6f5735e

Signed-off-by: ADD-SP <qiqi.zhang@konghq.com>

move the crate::runtime::scheduler::util out of the `cfg_rt_and_tim…

0215b54

…e` macros this enables rustfmt to fix the format Signed-off-by: ADD-SP <qiqi.zhang@konghq.com>

improve the test coverage of timer cancellation

ef4c37f

Signed-off-by: ADD-SP <qiqi.zhang@konghq.com>

fixup! improve the test coverage of timer cancellation

b72f924

revert spurious changes

c1a8676

Signed-off-by: ADD-SP <qiqi.zhang@konghq.com>

ADD-SP added 6 commits November 25, 2025 21:59

isolate two kinds of timer implementations

b63057c

Signed-off-by: ADD-SP <qiqi.zhang@konghq.com>

merge: sync changes from the base branch

cdf6994

fix rustfmt reports

e4ed82c

Signed-off-by: ADD-SP <qiqi.zhang@konghq.com>

revert tokio-util/tests/time_delay_queue.rs as it is using the exis…

91eb3e1

…ting timer impl Signed-off-by: ADD-SP <qiqi.zhang@konghq.com>

improve the test coverage

4465bab

Signed-off-by: ADD-SP <qiqi.zhang@konghq.com>

fixup! improve the test coverage

804f799

Darksonn mentioned this pull request Nov 25, 2025

Performance regression in 1.45.0 #7744

Open

Darksonn approved these changes Nov 26, 2025

View reviewed changes

tokio/src/time/interval.rs Outdated Show resolved Hide resolved

tokio/src/runtime/scheduler/multi_thread/worker.rs Outdated Show resolved Hide resolved

tokio/src/runtime/scheduler/mod.rs Outdated Show resolved Hide resolved

ADD-SP mentioned this pull request Nov 26, 2025

Share your experience with the alternative timer #7745

Open

improve style

f611539

Signed-off-by: ADD-SP <qiqi.zhang@konghq.com>

ADD-SP changed the title ~~time: delay the cancellation of timers~~ time: add alternative timer for better multicore scalability Nov 27, 2025

ADD-SP merged commit 73d733a into tokio-rs:master Nov 27, 2025
92 checks passed

ADD-SP deleted the add_sp/time-local-wheel branch November 27, 2025 01:29

ADD-SP mentioned this pull request Nov 27, 2025

Design document of the alternative timer #7747

Closed

Darksonn mentioned this pull request Jan 2, 2026

chore: prepare Tokio v1.49.0 #7824

Merged

howardjohn mentioned this pull request Jan 5, 2026

performance: consider a faster timer agentgateway/agentgateway#779

Open

Uh oh!

Conversation

ADD-SP commented Jul 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review guide

Uh oh!

ADD-SP Jul 19, 2025

Choose a reason for hiding this comment

Note

Uh oh!

ADD-SP Jul 19, 2025

Choose a reason for hiding this comment

Note

Uh oh!

ADD-SP Jul 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Note

Uh oh!

ADD-SP Jul 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Note

Uh oh!

Uh oh!

Uh oh!

ADD-SP Jul 20, 2025

Choose a reason for hiding this comment

Note

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Darksonn commented Nov 20, 2025

Uh oh!

carllerche commented Nov 21, 2025

Uh oh!

ADD-SP commented Nov 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Darksonn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Darksonn commented Nov 26, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ADD-SP commented Jul 19, 2025 •

edited

Loading

ADD-SP Jul 19, 2025 •

edited

Loading

ADD-SP Jul 19, 2025 •

edited

Loading

ADD-SP commented Nov 23, 2025 •

edited

Loading