New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Leverage local links on git checkouts #4919

Merged
merged 1 commit into from Jan 8, 2018

Conversation

Projects
None yet
4 participants
@alexcrichton
Copy link
Member

alexcrichton commented Jan 8, 2018

This commit updates the handling of git checkouts from the database to use
hardlinks if possible, speeding up this operation for large repositories
significantly.

As a refresher, Cargo caches git repositories in a few locations to speed up
local usage of git repositories. Cargo has a "database" folder which is a bare
checkout of any git repository Cargo has cached historically. This database
folder contains effectively a bunch of databases for remote repos that are
updated periodically.

When actually building a crate Cargo will clone this database into a different
location, the checkouts folder. Each rev we build (ever) is cached in the
checkouts folder. This means that once a checkout directory is created it's
frozen for all of time.

This latter step is what this commit is optimizing. When checking out the
database onto the local filesystem at a particular revision. Previously we were
instructing libgit2 to fall back to a "git aware" transport which was
exceedingly slow on some systems for filesystem-to-filesystem transfers. This
optimization (we just forgot to turn it on in libgit2) is a longstanding one and
should speed this up significantly!

Closes #4604

Leverage local links on git checkouts
This commit updates the handling of git checkouts from the database to use
hardlinks if possible, speeding up this operation for large repositories
significantly.

As a refresher, Cargo caches git repositories in a few locations to speed up
local usage of git repositories. Cargo has a "database" folder which is a bare
checkout of any git repository Cargo has cached historically. This database
folder contains effectively a bunch of databases for remote repos that are
updated periodically.

When actually building a crate Cargo will clone this database into a different
location, the checkouts folder. Each rev we build (ever) is cached in the
checkouts folder. This means that once a checkout directory is created it's
frozen for all of time.

This latter step is what this commit is optimizing. When checking out the
database onto the local filesystem at a particular revision. Previously we were
instructing libgit2 to fall back to a "git aware" transport which was
exceedingly slow on some systems for filesystem-to-filesystem transfers. This
optimization (we just forgot to turn it on in libgit2) is a longstanding one and
should speed this up significantly!

Closes #4604
@rust-highfive

This comment has been minimized.

Copy link

rust-highfive commented Jan 8, 2018

r? @matklad

(rust_highfive has picked a reviewer for you, use r? to override)

@matklad

This comment has been minimized.

Copy link
Member

matklad commented Jan 8, 2018

LGTM!

@bors r+

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Jan 8, 2018

📌 Commit 5cca4e8 has been approved by matklad

bors added a commit that referenced this pull request Jan 8, 2018

Auto merge of #4919 - alexcrichton:faster-git-clone, r=matklad
Leverage local links on git checkouts

This commit updates the handling of git checkouts from the database to use
hardlinks if possible, speeding up this operation for large repositories
significantly.

As a refresher, Cargo caches git repositories in a few locations to speed up
local usage of git repositories. Cargo has a "database" folder which is a bare
checkout of any git repository Cargo has cached historically. This database
folder contains effectively a bunch of databases for remote repos that are
updated periodically.

When actually building a crate Cargo will clone this database into a different
location, the checkouts folder. Each rev we build (ever) is cached in the
checkouts folder. This means that once a checkout directory is created it's
frozen for all of time.

This latter step is what this commit is optimizing. When checking out the
database onto the local filesystem at a particular revision. Previously we were
instructing libgit2 to fall back to a "git aware" transport which was
exceedingly slow on some systems for filesystem-to-filesystem transfers. This
optimization (we just forgot to turn it on in libgit2) is a longstanding one and
should speed this up significantly!

Closes #4604
@bors

This comment has been minimized.

Copy link
Contributor

bors commented Jan 8, 2018

⌛️ Testing commit 5cca4e8 with merge c4003c4...

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Jan 8, 2018

☀️ Test successful - status-appveyor, status-travis
Approved by: matklad
Pushing c4003c4 to master...

@bors bors merged commit 5cca4e8 into rust-lang:master Jan 8, 2018

3 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
homu Test successful
Details

@alexcrichton alexcrichton deleted the alexcrichton:faster-git-clone branch Jan 25, 2018

@alexcrichton alexcrichton referenced this pull request Mar 22, 2018

Merged

1.25 announcement #237

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment