Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cargo installing multiple bin crates doesn't build them in parallel #9741

Open
the8472 opened this issue Jul 28, 2021 · 5 comments
Open

Cargo installing multiple bin crates doesn't build them in parallel #9741

the8472 opened this issue Jul 28, 2021 · 5 comments
Labels
C-feature-request Category: proposal for a feature. Before PR, ping rust-lang/cargo if this is not `Feature accepted` Command-install Performance Gotta go fast! S-needs-design Status: Needs someone to work further on the design for the feature or fix. NOT YET accepted.

Comments

@the8472
Copy link
Member

the8472 commented Jul 28, 2021 •
edited by rustbot
Loading

Describe the problem you are trying to solve

When installing multiple bin crates, such as suggested by the rustc-dev-guide build instructions they are built sequentially. This doesn't fully utilize available parallelism on many-core systems and thus leads to longer build times than necessary.

Describe the solution you'd like

It would be nice if the binaries could be built in parallel under a shared jobserver to reduce wall-time for the install command.
@the8472 the8472 added the C-feature-request Category: proposal for a feature. Before PR, ping rust-lang/cargo if this is not `Feature accepted` label Jul 28, 2021
@nipunn1313
Copy link
Contributor

@rustbot claim

@nipunn1313 nipunn1313 mentioned this issue Aug 15, 2021
Merged
@nipunn1313
Copy link
Contributor

I put up #9793 as a step in the right direction, however I was unable to figure out how to actually parallelize the install command.

I tried naively using rayon to parallelize, but the Package struct was not Sized and I was unable to proceed with that strategy. It seemed that potentially cargo was not built to have multiple cargo commands running in parallel - and some more work might be necessary to make things work - to share job pool across multiple cargo compile commands issued by cargo itself.

@nipunn1313
Copy link
Contributor

@rustbot release-assignment

bors added a commit that referenced this issue Aug 19, 2021
@bors
Auto merge of #9793 - nipunn1313:install_parallel2, r=alexcrichton
b64c96b 
Determine packages to install prior to installing

Old logic (pseudocode)
```
for krate in to_install {
    pkg = determine_pkg(krate);
    install_pkg(pkg);
}
```
New logic
```
let pkgs = to_install.into_iter(|krate| determine_pkg(krate));
pkgs.into_iter(|pkg| install_pkg(pkg));
```

This has the short term benefit of dumping most error messages out earlier in the process (eg a typo in the second package name).

Longer term, it might help with #9741 - as only the second loop would be parallelized. First loop shouldn't be parallelized because it would lead to redundant registry/git updates.
This was referenced Aug 3, 2022
Closed
Closed
@skull-squadron
Copy link

Please make this happen. I have 90+ cores sitting idle during tool install builds and it takes forever. 🙏

@weihanglo
Copy link
Member

weihanglo commented Sep 30, 2022 •
edited
Loading

Just got an idea as a workaround.

Since Cargo honors GNU make jobserver, We can do tricks with the combos of parallel and make:

# Makefile
all:
        +parallel cargo install ::: cli1 cli2 cli3 cli4 cli5 cli6 cli7 cli8...

Then run make -j$(getconf _NPROCESSORS_ONLN).

By doing so, the number of rustc invocations can be controlled by jobserver. OTOH, GNU parallel works as a scheduler to coordinate which binary to install next. You might also tweak -j for parallel to control the parallelism.

@weihanglo weihanglo added Performance Gotta go fast! S-needs-design Status: Needs someone to work further on the design for the feature or fix. NOT YET accepted. labels May 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-feature-request Category: proposal for a feature. Before PR, ping rust-lang/cargo if this is not `Feature accepted` Command-install Performance Gotta go fast! S-needs-design Status: Needs someone to work further on the design for the feature or fix. NOT YET accepted.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ehuss @skull-squadron @the8472 @nipunn1313 @weihanglo