Skip to content

background-worker: Speed up index cloning by 3x #6043

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Feb 7, 2023

Conversation

Turbo87
Copy link
Member

@Turbo87 Turbo87 commented Feb 5, 2023

Previously we were using libgit2 to clone the index, which regularly took 2-3 minutes. This PR switches us over to using the git CLI, which appears to be significantly faster and clones the index in just over 30 seconds.

@Turbo87 Turbo87 added C-internal 🔧 Category: Nonessential work that would make the codebase more consistent or clear A-backend ⚙️ labels Feb 5, 2023
@Eh2406
Copy link
Contributor

Eh2406 commented Feb 5, 2023

cc @Byron, sounds like a good place for gitoxide some day.

@Byron
Copy link
Member

Byron commented Feb 5, 2023

Thanks for the heads-up! There is nothing stopping you from using gitoxide for this today, and depending on the hardware the worker runs on it should be a little faster, but with this repo no more than a second even with ample cores. The repository doesn't actually spend a lot of time in the delta-resolution phase which is the only phase that can be sped up meaningfully.

Since it's a background worker, one would also not be interested in progress here so using the git CLI seems like a good choice.

The only advantage one would have with gitoxide is to more easily detect network-related errors to retry, but then again, a simple retry-three-times logic might also be sufficient here.

@Turbo87
Copy link
Member Author

Turbo87 commented Feb 5, 2023

@jtgeibel FYI I've tested this out on staging and it appears to work fine :)

@Turbo87 Turbo87 merged commit 760f551 into rust-lang:master Feb 7, 2023
@Turbo87 Turbo87 deleted the worker-clone branch February 7, 2023 09:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-backend ⚙️ C-internal 🔧 Category: Nonessential work that would make the codebase more consistent or clear
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants