rt: instrument task poll times with a histogram #5685

carllerche · 2023-05-12T18:49:34Z

Adds support for instrumenting the poll times of all spawned tasks. Data is tracked in a histogram. The user must specify the histogram scale and bucket ranges. Implementation-wise, the same strategy is used in the runtime where we are just using atomic counters. Because instrumenting each poll duration will result in frequent calls to Instant::now(), I think it should be an opt-in metric.

There are still some open questions related to the builder API, but those can be punted as it is an unstable API. I will track the questions in the metrics stabilization tracking issue.

tobz

Some doc comment nits, but a few major questions.

tokio/src/runtime/metrics/worker.rs

tokio/src/runtime/metrics/batch.rs

hds

Couple of comments on the public API. They reflect my personal preferences and experience, so take them as you wish. (-:

tokio/src/runtime/builder.rs

tokio/src/runtime/metrics/runtime.rs

addressed

…togram

tokio/src/runtime/builder.rs

tokio/src/runtime/metrics/runtime.rs

…togram

hds

Looks good to me!

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [tokio](https://tokio.rs) ([source](https://github.com/tokio-rs/tokio)) | dependencies | minor | `1.28.2` -> `1.29.1` | | [tokio](https://tokio.rs) ([source](https://github.com/tokio-rs/tokio)) | dev-dependencies | minor | `1.28.2` -> `1.29.1` | --- ### Release Notes <details> <summary>tokio-rs/tokio (tokio)</summary> ### [`v1.29.1`](https://github.com/tokio-rs/tokio/releases/tag/tokio-1.29.1): Tokio v1.29.1 [Compare Source](tokio-rs/tokio@tokio-1.29.0...tokio-1.29.1) ##### Fixed - rt: fix nesting two `block_in_place` with a `block_on` between (#5837]) #5837]: tokio-rs/tokio#5837 ### [`v1.29.0`](https://github.com/tokio-rs/tokio/releases/tag/tokio-1.29.0): Tokio v1.29.0 [Compare Source](tokio-rs/tokio@tokio-1.28.2...tokio-1.29.0) Technically a breaking change, the `Send` implementation is removed from `runtime::EnterGuard`. This change fixes a bug and should not impact most users. ##### Breaking - rt: `EnterGuard` should not be `Send` (#5766]) ##### Fixed - fs: reduce blocking ops in `fs::read_dir` (#5653]) - rt: fix possible starvation (#5686], #5712]) - rt: fix stacked borrows issue in `JoinSet` (#5693]) - rt: panic if `EnterGuard` dropped incorrect order (#5772]) - time: do not overflow to signal value (#5710]) - fs: wait for in-flight ops before cloning `File` (#5803]) ##### Changed - rt: reduce time to poll tasks scheduled from outside the runtime (#5705], #5720]) ##### Added - net: add uds doc alias for unix sockets (#5659]) - rt: add metric for number of tasks (#5628]) - sync: implement more traits for channel errors (#5666]) - net: add nodelay methods on TcpSocket (#5672]) - sync: add `broadcast::Receiver::blocking_recv` (#5690]) - process: add `raw_arg` method to `Command` (#5704]) - io: support PRIORITY epoll events (#5566]) - task: add `JoinSet::poll_join_next` (#5721]) - net: add support for Redox OS (#5790]) ##### Unstable - rt: add the ability to dump task backtraces (#5608], #5676], #5708], #5717]) - rt: instrument task poll times with a histogram (#5685]) #5766]: tokio-rs/tokio#5766 #5653]: tokio-rs/tokio#5653 #5686]: tokio-rs/tokio#5686 #5712]: tokio-rs/tokio#5712 #5693]: tokio-rs/tokio#5693 #5772]: tokio-rs/tokio#5772 #5710]: tokio-rs/tokio#5710 #5803]: tokio-rs/tokio#5803 #5705]: tokio-rs/tokio#5705 #5720]: tokio-rs/tokio#5720 #5659]: tokio-rs/tokio#5659 #5628]: tokio-rs/tokio#5628 #5666]: tokio-rs/tokio#5666 #5672]: tokio-rs/tokio#5672 #5690]: tokio-rs/tokio#5690 #5704]: tokio-rs/tokio#5704 #5566]: tokio-rs/tokio#5566 #5721]: tokio-rs/tokio#5721 #5790]: tokio-rs/tokio#5790 #5608]: tokio-rs/tokio#5608 #5676]: tokio-rs/tokio#5676 #5708]: tokio-rs/tokio#5708 #5717]: tokio-rs/tokio#5717 #5685]: tokio-rs/tokio#5685 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Co-authored-by: cabr2-bot <cabr2.help@gmail.com> Reviewed-on: https://codeberg.org/Calciumdibromid/CaBr2/pulls/1958 Reviewed-by: crapStone <crapstone01@gmail.com> Co-authored-by: Calciumdibromid Bot <cabr2_bot@noreply.codeberg.org> Co-committed-by: Calciumdibromid Bot <cabr2_bot@noreply.codeberg.org>

rt: instrument task poll times with a histogram

e748dfe

github-actions bot added the R-loom Run loom tests on this PR label May 12, 2023

tobz previously requested changes May 12, 2023

View reviewed changes

tokio/src/runtime/metrics/worker.rs Outdated Show resolved Hide resolved

tokio/src/runtime/metrics/batch.rs Outdated Show resolved Hide resolved

tokio/src/runtime/metrics/batch.rs Outdated Show resolved Hide resolved

tokio/src/runtime/metrics/batch.rs Outdated Show resolved Hide resolved

carllerche added 2 commits May 12, 2023 13:32

rm poll duration histogram

7005375

fix comment

f4cf71e

carllerche mentioned this pull request May 12, 2023

wip: add support for rt poll count histogram tokio-rs/tokio-metrics#48

Draft

carllerche added 4 commits May 12, 2023 14:17

fix some tests

58a8c37

fix loom tests

74ace21

fix warning

5501746

test: ensure histogram not populated when not enabled

c87f26c

hds reviewed May 12, 2023

View reviewed changes

tokio/src/runtime/builder.rs Outdated Show resolved Hide resolved

tokio/src/runtime/metrics/runtime.rs Show resolved Hide resolved

carllerche added 4 commits May 12, 2023 17:26

add some tests

c0be5b9

docs todo, see if ci passes

cdcee6a

try again

ed9b52a

try to get CI to pass, again

ad55a45

tweak builder API

a73b06d

Darksonn added A-tokio Area: The main tokio crate M-metrics Module: tokio/runtime/metrics labels May 15, 2023

carllerche added 4 commits May 15, 2023 09:36

Merge remote-tracking branch 'origin' into rt-metrics-poll-histogram

f365680

docs

8d0f8b4

Merge remote-tracking branch 'origin/master' into rt-metrics-poll-his…

4740ad8

…togram

fix docs

4f82d4f

carllerche marked this pull request as ready for review May 15, 2023 18:26

Noah-Kennedy approved these changes May 15, 2023

View reviewed changes

hds reviewed May 15, 2023

View reviewed changes

tokio/src/runtime/builder.rs Outdated Show resolved Hide resolved

tokio/src/runtime/builder.rs Show resolved Hide resolved

tokio/src/runtime/metrics/runtime.rs Show resolved Hide resolved

tokio/src/runtime/metrics/runtime.rs Show resolved Hide resolved

carllerche added 2 commits May 15, 2023 14:43

apply feedback

e53a9a1

Merge remote-tracking branch 'origin/master' into rt-metrics-poll-his…

9863e6a

…togram

hds approved these changes May 15, 2023

View reviewed changes

carllerche merged commit c84d0a1 into master May 15, 2023
59 checks passed

carllerche deleted the rt-metrics-poll-histogram branch May 15, 2023 22:20

carllerche mentioned this pull request May 15, 2023

meta: Runtime metrics stabilization #4073

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rt: instrument task poll times with a histogram #5685

rt: instrument task poll times with a histogram #5685

carllerche commented May 12, 2023 •

edited

tobz left a comment

hds left a comment

hds left a comment

rt: instrument task poll times with a histogram #5685

rt: instrument task poll times with a histogram #5685

Conversation

carllerche commented May 12, 2023 • edited

tobz left a comment

Choose a reason for hiding this comment

hds left a comment

Choose a reason for hiding this comment

hds left a comment

Choose a reason for hiding this comment

carllerche commented May 12, 2023 •

edited