Cargo fetching registry extremely slow when cargo does it but not with manual git command

[korken89]

Problem
When utilizing cargo in CI we have noticed that the download of the registry, during Updating crates.io index, is extremely slow on the order of 5-20 min (and sometimes failing), which is stalling our CI pipelines.
To experiment we tried running a build locally after deleting .cargo/registry and saw the same slow speed, however if we run the git command manually, i.e. git fetch --force --update-head-ok 'https://github.com/rust-lang/crates.io-index' 'refs/heads/master:refs/remotes/origin/master' 'HEAD:refs/remotes/origin/HEAD', it downloads at ~10 MB/s.

So the slowdown only seems to happen when cargo is running the command.

Steps

1. Remove .cargo/registry (as with a CI running a clean build).
2. Build any project, the Updating crates.io index will take 5-20 min with download speed in the range 10-300 kB/s.
3. Manually running the fetch will download at ~10 MB/s (on our 200/200 MBit connection).

Notes

Output of cargo version:

cargo 1.50.0 (f04e7fab7 2021-02-04)

OS:

Linux 5.10.16-arch1-1 #1 SMP PREEMPT Sat, 13 Feb 2021 20:50:18 +0000 x86_64 GNU/Linux

[alexcrichton]

This is unfortunately likely a libgit2 issue which we don't have a ton of control over. We try to report bugs upstream when we can but it sometimes takes awhile to reduce and/or fix them.

[korken89]

Thanks for checking!

We worked around it with some hacky cache-like fixes that we could do as we have full control over the server on which the CI runs on, so we don't have this issue anymore.

[Eh2406]

If you are no longer experiencing the problem then I will close for now. If you want to try to isolate something we can report upstream, then we can re-open or start a new one.

[HanabishiRecca]

Yeah, git downloads like 10x faster with the same repo.
Also --depth=1 reduces the download/repo size significantly. But unfortunately libgit2 does not support it.

[simon-frankau]

Looks like some form of --depth=1 support has now landed in libgit2 - libgit2/libgit2#3058 (comment). Maybe it's time to start experimenting with this so cargo can pick it up soon after the libgit2 feature is looking really solid?

[HanabishiRecca]

Recently introduced sparse protocol effectively mitigates the issue. And it is the default since 1.70.
I will not be surprised if git registry will be deprecated completely in the future.

[weihanglo]

We have landed shallow clone support via -Zgitoxide. Please have a look if you're interested in it.

For libgit2, maybe? The steps for that route is first waiting for a new release and then updating git2-rs bindings.

I will not be surprised if git registry will be deprecated completely in the future.

Depends on what do you mean by “deprecated completely”. I believe git registry will still be there for a long while, but yes, now Cargo favors sparse registry.