crate_universe: Make module extension restarts less expensive.
#2691
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Help with CI would be appreciated, I can't spot the problem right now. |
redsun82
reviewed
Jun 12, 2024
This PR reduces the number of expensive module extension implementation restarts in combination with multiple from_cargo configurations. As each configuration processing calls module_ctx.path (which can and will cause restarts), we executed `_generate_hub_and_spokes` (i.e. `cargo-bazel) a lot and then threw that away on the next restart. Triggering `module_ctx.path` early is therefore a significant performance optimization.
criemen
force-pushed
the
criemen/less-restarts
branch
from
a664575
to
ed10465
Compare
|
I significantly reworked the implementation of this, as the previous iteration would've still caused expensive restarts with from_cargo was used in different modules. The same CI job fails, and I still don't understand why that is. |
illicitonion
approved these changes
Jun 13, 2024
There was a problem hiding this comment.
Seems reasonable, thanks!
The CI job should be fixed by #2688 - should get that merged and this unblocked soon :)
|
I checked with the bazel devs, and |
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
rrbutani
referenced
this pull request
in bazel-contrib/toolchains_llvm
Jul 9, 2024
[](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [rules_rust](https://togithub.com/bazelbuild/rules_rust) | http_archive | minor | `0.46.0` -> `0.47.1` | --- ### Release Notes <details> <summary>bazelbuild/rules_rust (rules_rust)</summary> ### [`v0.47.1`](https://togithub.com/bazelbuild/rules_rust/releases/tag/0.47.1) [Compare Source](https://togithub.com/bazelbuild/rules_rust/compare/0.46.0...0.47.1) ### 0.47.1 #### Bzlmod ```python bazel_dep(name = "rules_rust", version = "0.47.1") ``` #### WORKSPACE ```python load("@​bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") http_archive( name = "rules_rust", integrity = "sha256-35cwdTOaqqu4y+aXgIUU2C2PAKMz4+uyJ7/UMIGCmFs=", urls = ["https://github.com/bazelbuild/rules_rust/releases/download/0.47.1/rules_rust-v0.47.1.tar.gz"], ) ``` Additional documentation can be found at: https://bazelbuild.github.io/rules_rust/#setup #### What's Changed - Remote vendor doesn't require srcs present by [@​illicitonion](https://togithub.com/illicitonion) in [https://github.com/bazelbuild/rules_rust/pull/2688](https://togithub.com/bazelbuild/rules_rust/pull/2688) - Add flag to refer to a `sh_toolchain` for process wrapper bootstrap shebangs by [@​UebelAndre](https://togithub.com/UebelAndre) in [https://github.com/bazelbuild/rules_rust/pull/2694](https://togithub.com/bazelbuild/rules_rust/pull/2694) - Added Rust 1.79.0 by [@​UebelAndre](https://togithub.com/UebelAndre) in [https://github.com/bazelbuild/rules_rust/pull/2695](https://togithub.com/bazelbuild/rules_rust/pull/2695) - `crate_universe`: Make module extension restarts less expensive. by [@​criemen](https://togithub.com/criemen) in [https://github.com/bazelbuild/rules_rust/pull/2691](https://togithub.com/bazelbuild/rules_rust/pull/2691) - Add example of using JNI to call into Rust by [@​illicitonion](https://togithub.com/illicitonion) in [https://github.com/bazelbuild/rules_rust/pull/2690](https://togithub.com/bazelbuild/rules_rust/pull/2690) - Apply extra_rustc_flags list to all toolchains by [@​nmattia](https://togithub.com/nmattia) in [https://github.com/bazelbuild/rules_rust/pull/2693](https://togithub.com/bazelbuild/rules_rust/pull/2693) - rust-analyzer: Factor aliases into the project by [@​illicitonion](https://togithub.com/illicitonion) in [https://github.com/bazelbuild/rules_rust/pull/2709](https://togithub.com/bazelbuild/rules_rust/pull/2709) - crate_universe docs: Split bzlmod from workspace by [@​illicitonion](https://togithub.com/illicitonion) in [https://github.com/bazelbuild/rules_rust/pull/2700](https://togithub.com/bazelbuild/rules_rust/pull/2700) - Updated external dependencies docs by [@​UebelAndre](https://togithub.com/UebelAndre) in [https://github.com/bazelbuild/rules_rust/pull/2711](https://togithub.com/bazelbuild/rules_rust/pull/2711) - Update wasm-bindgen to 0.2.92 by [@​alexkirsz](https://togithub.com/alexkirsz) in [https://github.com/bazelbuild/rules_rust/pull/2714](https://togithub.com/bazelbuild/rules_rust/pull/2714) - upstream wrappers for cargo, rustc, cargo-clippy by [@​kolloch](https://togithub.com/kolloch) in [https://github.com/bazelbuild/rules_rust/pull/2703](https://togithub.com/bazelbuild/rules_rust/pull/2703) - Clean up downloads that go through the API instead of the CDN by [@​ograff](https://togithub.com/ograff) in [https://github.com/bazelbuild/rules_rust/pull/2718](https://togithub.com/bazelbuild/rules_rust/pull/2718) - Updated rust-analyzer to use repo vs generated files as crate roots by [@​UebelAndre](https://togithub.com/UebelAndre) in [https://github.com/bazelbuild/rules_rust/pull/2717](https://togithub.com/bazelbuild/rules_rust/pull/2717) - crate-universe allows workspaces with MODULE.bazel only by [@​matt-duch](https://togithub.com/matt-duch) in [https://github.com/bazelbuild/rules_rust/pull/2720](https://togithub.com/bazelbuild/rules_rust/pull/2720) - Release 0.47.0 by [@​UebelAndre](https://togithub.com/UebelAndre) in [https://github.com/bazelbuild/rules_rust/pull/2725](https://togithub.com/bazelbuild/rules_rust/pull/2725) - Update bindgen.bzl to use toolchain system include directories as well by [@​hlopko](https://togithub.com/hlopko) in [https://github.com/bazelbuild/rules_rust/pull/2727](https://togithub.com/bazelbuild/rules_rust/pull/2727) - Use newer macos runners by [@​ograff](https://togithub.com/ograff) in [https://github.com/bazelbuild/rules_rust/pull/2732](https://togithub.com/bazelbuild/rules_rust/pull/2732) #### New Contributors - [@​nmattia](https://togithub.com/nmattia) made their first contribution in [https://github.com/bazelbuild/rules_rust/pull/2693](https://togithub.com/bazelbuild/rules_rust/pull/2693) - [@​kolloch](https://togithub.com/kolloch) made their first contribution in [https://github.com/bazelbuild/rules_rust/pull/2703](https://togithub.com/bazelbuild/rules_rust/pull/2703) - [@​matt-duch](https://togithub.com/matt-duch) made their first contribution in [https://github.com/bazelbuild/rules_rust/pull/2720](https://togithub.com/bazelbuild/rules_rust/pull/2720) **Full Changelog**: bazelbuild/rules_rust@0.46.0...0.47.1 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/bazel-contrib/toolchains_llvm). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40MjUuMSIsInVwZGF0ZWRJblZlciI6IjM3LjQyNS4xIiwidGFyZ2V0QnJhbmNoIjoibWFzdGVyIiwibGFiZWxzIjpbXX0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
](https://renovatebot.com){kind=link}
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR reduces the number of expensive module extension implementation restarts in combination with multiple
from_cargoconfigurations.As each configuration processing calls
module_ctx.path(which can and will cause restarts), we executed _generate_hub_and_spokes (i.e.cargo-bazel) a lot and then threw that away on the next restart.Triggering
module_ctx.pathearly is therefore a significant performance optimization.On our repository, this gives us a 10sec speedup on module extension processing (local M1 mac, nothing happening in parallel), see our MODULE.bazel if you're interested in what we're doing.
This excessive restarting also exposed an upstream bazel bug on Windows 2019, where bazel spuriously fails to clean up the working directory (c.f. bazelbuild/bazel#22710).