Skip to content

server: build Tokio runtime using oxide-tokio-rt#913

Merged
hawkw merged 5 commits into
masterfrom
eliza/oxide-tokio-rt
Jun 28, 2025
Merged

server: build Tokio runtime using oxide-tokio-rt#913
hawkw merged 5 commits into
masterfrom
eliza/oxide-tokio-rt

Conversation

@hawkw

@hawkw hawkw commented Jun 26, 2025

Copy link
Copy Markdown
Member

As described in RFD 579, we are now recommending that the oxide-tokio-rt crate be used to construct the Tokio runtime in all production software that uses Tokio. This crate provides common functions for constructing Tokio runtimes with the recommended configurations.

In particular, it currently does the following:

  • On illumos, configures the runtime to emit DTrace probes, using tokio-dtrace.
  • Disables Tokio's LIFO slot optimization. This feature is intended to improve message-passing latency, but because tasks in the LIFO slot do not currently participate in work-stealing, it can result in extreme latency spikes in some cases (see omicron#8334 for a worked example).

Thus, this commit changes propolis-server to construct its Tokio runtimes using oxide-tokio-rt so that these common recommended configurations are included. This requires enabling Tokio's unstable features in the workspace.

As described in [RFD 579], we are now recommending that the
[`oxide-tokio-rt`] crate be used to construct the Tokio runtime in all
production software that uses Tokio. This crate provides common
functions for constructing Tokio runtimes with the recommended
configurations.

In particular, it currently does the following:

- On illumos, configures the runtime to emit DTrace probes, using
  [`tokio-dtrace`].
- Disables Tokio's LIFO slot optimization. This feature is intended to
  improve message-passing latency, but because tasks in the LIFO slot
  do not currently participate in work-stealing, it can result in
  extreme latency spikes in some cases (see omicron#8334 for a worked
  example).

Thus, this commit changes `propolis-server` to construct its Tokio
runtimes using `oxide-tokio-rt` so that these common recommended
configurations are included. This requires enabling Tokio's unstable
features in the workspace.

[RFD 579]: https://rfd.shared.oxide.computer/rfd/0579
[`oxide-tokio-rt`]: https://github.com/oxidecomputer/oxide-tokio-rt
[`tokio-dtrace`]: https://github.com/oxidecomputer/tokio-dtrace
@hawkw hawkw requested review from iximeow and pfmooney June 26, 2025 21:56

@pfmooney pfmooney left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apply it to standalone as well?

Comment thread bin/propolis-server/src/lib/vm/ensure.rs Outdated
Comment thread .cargo/config.toml
@hawkw

hawkw commented Jun 26, 2025

Copy link
Copy Markdown
Member Author

The Clippy thing seems like a new lint that's unrelated to this change: https://github.com/oxidecomputer/propolis/actions/runs/15913312178/job/44885677202?pr=913#step:5:1947

Comment thread .cargo/config.toml Outdated
@hawkw

hawkw commented Jun 26, 2025

Copy link
Copy Markdown
Member Author

I'm gonna go clean up all the Clippy mess in a separate PR.

@hawkw hawkw mentioned this pull request Jun 27, 2025
@hawkw

hawkw commented Jun 27, 2025

Copy link
Copy Markdown
Member Author

We could also consider configuring Clippy to warn on all uses of #[tokio::main] in the repo, as I suggested here. I opted not to do this, since I felt like the remaining uses of #[tokio::main] in xtasks, examples, and propolis-cli, were all basically fine and it didn't feel necessary to add explicit #[expect] attributes for the lint on all of them. If others disagree, I'm happy to add this.

@pfmooney

Copy link
Copy Markdown
Contributor

We could also consider configuring Clippy to warn on all uses of #[tokio::main] in the repo

Yeah, for the reasons you cited, that seems a bit unsporting.

@hawkw

hawkw commented Jun 27, 2025

Copy link
Copy Markdown
Member Author

We could also consider configuring Clippy to warn on all uses of #[tokio::main] in the repo

Yeah, for the reasons you cited, that seems a bit unsporting.

Yup, the advice to warn on that was really more intended to workspaces like Omicron, which has a ton of smaller production binaries using Tokio, and folks are adding new binaries a lot. In Propolis, I don't think it's as useful, since any new binaries we add will generally just be xtasks/dev tools.

@hawkw hawkw merged commit 6b5734f into master Jun 28, 2025
11 checks passed
@hawkw hawkw deleted the eliza/oxide-tokio-rt branch June 28, 2025 00:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants