Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Very slow checking of installed crate versions on cargo binstall #1295

Closed
0atman opened this issue Aug 16, 2023 · 3 comments · Fixed by #1299
Closed

Very slow checking of installed crate versions on cargo binstall #1295

0atman opened this issue Aug 16, 2023 · 3 comments · Fixed by #1299

Comments

@0atman
Copy link

0atman commented Aug 16, 2023 •
edited

If you, like me, maintain a large list of system tools in Rust and re-run the install line now and then to keep up to date, you'll notice a very long delay checking each crate for being up to date, even if no version bump has happened.

After package indexing, which is very fast, this process is entirely local and so shouldn't be slow. Yet it is, about 1s delayed per item.

Here's my preposterous install line that I will take no notes on:

$ time ~/.cargo/bin/cargo binstall -y --locked cargo-info ripgrep ncspot du-dust exa bat irust nu starship zellij bacon cargo-watch porsmo speedtest-rs gitui wiki-tui evcxr_jupyter rtx-cli mprocs procs joshuto rsdate xh fd-find sd alacritty paru gitoxide bob-nvim chatgpt-cli turbocommit aichat cargo-modules gitoxide sqlx-cli dioxus-cli termscp bottom silicon cargo-nextest iamb tuifeed tealdeer just

 INFO resolve: Resolving package: 'aichat'
 INFO resolve: Resolving package: 'bacon'
 INFO resolve: Resolving package: 'cargo-watch'
 INFO resolve: Resolving package: 'alacritty'
 INFO resolve: Resolving package: 'cargo-modules'
 INFO resolve: Resolving package: 'cargo-nextest'
 INFO resolve: Resolving package: 'just'
 INFO resolve: Resolving package: 'chatgpt-cli'
 INFO resolve: Resolving package: 'mprocs'
 INFO resolve: Resolving package: 'fd-find'
 INFO resolve: Resolving package: 'paru'
 INFO resolve: Resolving package: 'gitoxide'
 INFO resolve: Resolving package: 'porsmo'
 INFO resolve: Resolving package: 'speedtest-rs'
 INFO resolve: Resolving package: 'gitui'
 INFO resolve: Resolving package: 'sqlx-cli'
 INFO resolve: Resolving package: 'termscp'
 INFO resolve: Resolving package: 'tealdeer'
 INFO resolve: Resolving package: 'tuifeed'
 INFO resolve: Resolving package: 'sd'
 INFO resolve: Resolving package: 'bottom'
 INFO resolve: Resolving package: 'ncspot'
 INFO resolve: Resolving package: 'silicon'
 INFO resolve: Resolving package: 'nu'
 INFO resolve: Resolving package: 'starship'
 INFO resolve: Resolving package: 'zellij'
 INFO resolve: Resolving package: 'procs'
 INFO resolve: Resolving package: 'iamb'
 INFO resolve: Resolving package: 'rsdate'
 INFO resolve: Resolving package: 'du-dust'
 INFO resolve: Resolving package: 'bob-nvim'
 INFO resolve: Resolving package: 'turbocommit'
 INFO resolve: Resolving package: 'evcxr_jupyter'
 INFO resolve: Resolving package: 'dioxus-cli'
 INFO resolve: Resolving package: 'bat'
 INFO resolve: Resolving package: 'xh'
 INFO resolve: Resolving package: 'cargo-info'
 INFO resolve: Resolving package: 'irust'
 INFO resolve: Resolving package: 'rtx-cli'
 INFO resolve: Resolving package: 'ripgrep'
 INFO resolve: Resolving package: 'joshuto'
 INFO resolve: Resolving package: 'wiki-tui'
 INFO resolve: Resolving package: 'exa'

Output immediately jumps to here, then spins 1s for each of the following lines:

 INFO resolve: aichat v0.8.0 is already installed, use --force to override
 INFO resolve: bacon v2.12.1 is already installed, use --force to override
 INFO resolve: cargo-watch v8.4.0 is already installed, use --force to override
 INFO resolve: cargo-modules v0.9.2 is already installed, use --force to override
 INFO resolve: alacritty v0.12.2 is already installed, use --force to override
 INFO resolve: cargo-nextest v0.9.57 is already installed, use --force to override
 INFO resolve: just v1.14.0 is already installed, use --force to override
 INFO resolve: chatgpt-cli v0.3.1 is already installed, use --force to override
 INFO resolve: mprocs v0.6.4 is already installed, use --force to override
 INFO resolve: fd-find v8.7.0 is already installed, use --force to override
 INFO resolve: paru v1.11.0 is already installed, use --force to override
 INFO resolve: porsmo v0.2.2 is already installed, use --force to override
 INFO resolve: gitoxide v0.28.0 is already installed, use --force to override
 INFO resolve: speedtest-rs v0.1.4 is already installed, use --force to override
 INFO resolve: gitui v0.23.0 is already installed, use --force to override
 INFO resolve: sqlx-cli v0.7.1 is already installed, use --force to override
 INFO resolve: termscp v0.12.1 is already installed, use --force to override
 INFO resolve: tealdeer v1.6.1 is already installed, use --force to override
 INFO resolve: tuifeed v0.3.2 is already installed, use --force to override
 INFO resolve: sd v0.7.6 is already installed, use --force to override
 INFO resolve: ncspot v0.13.4 is already installed, use --force to override
 INFO resolve: bottom v0.9.4 is already installed, use --force to override
 INFO resolve: silicon v0.5.1 is already installed, use --force to override
 INFO resolve: nu v0.83.1 is already installed, use --force to override
 INFO resolve: zellij v0.37.2 is already installed, use --force to override
 INFO resolve: starship v1.16.0 is already installed, use --force to override
 INFO resolve: procs v0.14.0 is already installed, use --force to override
 INFO resolve: iamb v0.0.8 is already installed, use --force to override
 INFO resolve: rsdate v0.5.0 is already installed, use --force to override
 INFO resolve: bob-nvim v2.4.1 is already installed, use --force to override
 INFO resolve: turbocommit v0.12.1 is already installed, use --force to override
 INFO resolve: du-dust v0.8.6 is already installed, use --force to override
 INFO resolve: evcxr_jupyter v0.15.1 is already installed, use --force to override
 INFO resolve: dioxus-cli v0.4.0 is already installed, use --force to override
 INFO resolve: xh v0.18.0 is already installed, use --force to override
 INFO resolve: bat v0.23.0 is already installed, use --force to override
 INFO resolve: cargo-info v0.7.6 is already installed, use --force to override
 INFO resolve: irust v1.70.0 is already installed, use --force to override
 INFO resolve: rtx-cli v2023.8.0 is already installed, use --force to override
 INFO resolve: ripgrep v13.0.0 is already installed, use --force to override
 INFO resolve: joshuto v0.9.5 is already installed, use --force to override
 INFO resolve: wiki-tui v0.8.2 is already installed, use --force to override
 INFO resolve: exa v0.10.1 is already installed, use --force to override
 INFO Done in 42.193393143s

real	0m42.236s
user	0m0.323s
sys	0m0.062s

Is there a sleep(1) in there somewhere? 😅
@0atman 0atman changed the title Very slow checking of if crate up-to-date Very slow checking of installed crate versions on cargo binstall Aug 16, 2023
@NobodyXu
Copy link
Member

Is there a sleep(1) in there somewhere? 😅

Actually yes, you found out the cause faster than I do.

Our crates.io client has a one request per each second rate limit, I think it was copied from crates_io_api.

@NobodyXu
Copy link
Member

We could remove that and just the global rate limit policy.

NobodyXu added a commit that referenced this issue Aug 16, 2023
@NobodyXu
binstalk-registry: Rm rate-limit for crates.io registry
b8d2fa9 
Fixed #1295

The 1 request per second rate-limit is too strict and it makes
`cargo-binstall` very slow when resolving many crates in parallel.

Relying on the rate-limit in `binstalk_downloader::remote::Client`
should be good enough.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
@NobodyXu NobodyXu mentioned this issue Aug 16, 2023
Merged
github-merge-queue bot pushed a commit that referenced this issue Aug 16, 2023
@NobodyXu
binstalk-registry: Rm rate-limit for crates.io registry (#1299)
f0f0c2b 
Fixed #1295

The 1 request per second rate-limit is too strict and it makes
`cargo-binstall` very slow when resolving many crates in parallel.

Relying on the rate-limit in `binstalk_downloader::remote::Client`
should be good enough.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
@NobodyXu NobodyXu mentioned this issue Aug 24, 2023
Merged
@NobodyXu
Copy link
Member

@0atman We have released cargo-binstall v1.3.0 which should fixed this by switching to sparse index and use the global rate-limit instead.
Please have a try and I'd like your feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

binstalk-registry: Rm rate-limit for crates.io registry cargo-bins/cargo-binstall
2 participants
@0atman @NobodyXu