Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cargo-update: exceedingly slow progress on update when local registry/index is used and git-fetch-with-cli is false #9167

Open
datdenkikniet opened this issue Feb 12, 2021 · 10 comments
Labels
A-git Area: anything dealing with git C-bug Category: bug Performance Gotta go fast! S-blocked-external Status: ❌ blocked on something out of the direct control of the Cargo project, e.g., upstream fix

Comments

@datdenkikniet
Copy link

datdenkikniet commented Feb 12, 2021

Problem

When using a local registry located at /home/user/rust/crates.io-index (which is simply a clone of https://github.com/rust-lang/crates.io-index.git) , specified in ~/.cargo/config using the following config:

[source]

[source.mirror]
registry = "file:///home/user/rust/crates.io-index"

[source.crates-io]
replace-with = "mirror"

is used, cargo update progresses incredibly slowly, consumes 1 CPU and slowly increases its RAM usage, given that the project has at least one dependency. That's to say: when running cargo update the process takes very long.

The total update time was 21 minutes.

However, when ~/.cargo/config is changed to

[net]
git-fetch-with-cli = true

[source]

[source.mirror]
registry = "file:///home/jona/projects/rust/crates.io-index"

[source.crates-io]
replace-with = "mirror"

the update actually succeeds relatively quickly, which is the expected behaviour. With this option, the update took about 21 seconds.
Steps

  1. Install rust
  2. Clone https://github.com/rust-lang/crates.io-index.git
  3. Edit/add the local source and overriding of the crates-io source to ~/.cargo/config using the first of the above examples. Point the mirror source to the directory into which crates.io-index was cloned.
  4. Create an empty/basic project
  5. Add a dependency to this project (I used time = "0.1.0", but as far as I can tell any dependency that triggers an update of the registry will cause this issue)
  6. Attempt to run cargo update

Possible Solution(s)

Use

[net]
git-fetch-with-cli = true

in cargo configuration.

Notes

Output of cargo version:
I tested found the issue on 1.46.0, 1.49.0 (d00d64d 2020-12-05), and 1.50.0.

@datdenkikniet datdenkikniet added the C-bug Category: bug label Feb 12, 2021
@ehuss
Copy link
Contributor

ehuss commented Feb 12, 2021

Thanks for the report. I have noticed this, too, and it can be extremely slow. Unfortunately I think this is an issue with libgit2 (libgit2/libgit2#2836). For now, git-fetch-with-cli is probably the best workaround.

@romecode
Copy link

Super slow on https git pulls on macos here. Spent a while trying different things... this worked for me.

[source.crates-io]
registry = "git://github.com/rust-lang/crates.io-index.git"

@datdenkikniet
Copy link
Author

@romecode good to see that you have fixed your problem, but I don't think your solution is relevant to this issue.

I've edited the issue itself to clearly include the workaround for the problem.

@kozik47
Copy link

kozik47 commented Mar 31, 2022

[source.crates-io]
registry = "git://github.com/rust-lang/crates.io-index.git"

Thanks. This fixed extremely slow 'Updating crates.io index' for me on Arch

@asottile-sentry
Copy link

in my situation, in addition to taking a bunch of CPU the update also allocated ~8.6GB of ram which was not freed after the update (eventually leading to OOM). using the workaround above worked great

@RouHim
Copy link

RouHim commented Aug 18, 2022

It even leads to out of memory exceptions, when cross compiling in an alpine arm container in github actions:
https://github.com/RouHim/this-week-in-past/runs/7892722833

(fixed with git-fetch-with-cli = true)

@weihanglo
Copy link
Member

FWIW, there is an ongoing project on integrating gitoxide1 to replace some uses of libgit2 in Cargo. I wouldn't say it will be landed soon but we can keep a eye on it, and maybe help @Byron if needed.

Footnotes

  1. It is a project sponsored by Rust Foundation. Here is a recent discussion: https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/.60gitoxide.60.20integration.20for.20shallow.20clone.20support

@Byron
Copy link
Member

Byron commented Aug 18, 2022

gitoxide at this time supports git-aware transport exclusively so will do a 'proper' clone with file:///home/… and /home/…, and in doing that it is faster than git itself (when it is also using git-aware transport). Thus I'd also expect this issue to be fixed once the crates-io indices are cloned and fetched with gitoxide.

Edit: It also turns out that part of the slowness comes from libgit2 implementing its own slow upload-pack, which gitoxide doesn't do either - instead it calls git-upload-pack to deal with the remote side. That should help with performance too until gitoxide implements the server side as well.

@dxspro28
Copy link

Set your source to this mirror. It worked for me
[source.crates-io]
replace-with = 'tuna'

[source.tuna]
registry = "https://mirrors.tuna.tsinghua.edu.cn/git/crates.io-index.git"

@datdenkikniet
Copy link
Author

Thanks for thinking along @dxspro28, but that solution is irrelevant to the problem described here. Configuring the registry to be some other online source is not a solution when one wishes to use a local registry.

Additionally, unless the registry located at https://mirrors.tuna.tsinghua.edu.cn/git/crates.io-index.git contains some crates that are not available through crates.io, there is no real reason to use that over the default.

olksdr added a commit to getsentry/relay that referenced this issue Nov 9, 2022
It looks like docker build was killed because it used too much memory
while updating the registry index, when forcing cargo to use cli, I
could pass the step when the index is updated and successfully build the
image.

Fixes #1559 
It seems related to rust-lang/cargo#9167
@weihanglo weihanglo added S-blocked-external Status: ❌ blocked on something out of the direct control of the Cargo project, e.g., upstream fix Performance Gotta go fast! labels Apr 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-git Area: anything dealing with git C-bug Category: bug Performance Gotta go fast! S-blocked-external Status: ❌ blocked on something out of the direct control of the Cargo project, e.g., upstream fix
Projects
None yet
Development

No branches or pull requests

9 participants