Skip to content

/cargo

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lazily download packages #2406

Merged
merged 11 commits into from Feb 29, 2016

Conversation

Reviewers
No reviews
Assignees

@brson brson

Labels
relnotes
Projects
None yet
Milestone
No milestone
5 participants
@alexcrichton @rust-highfive @brson @bors @huonw
@alexcrichton
Copy link
Member

alexcrichton commented Feb 22, 2016

Currently Cargo will download an entire resolution graph all at once when in fact most packages may not be relevant to a compilation. For example target-specific dependencies and dev-dependencies are unconditionally downloaded regardless of whether they're actually needed or not.

This commit alters the internals of Cargo to avoid downloading everything immediately and just switches to lazily downloading packages. This involved adding a new LazyCell primitive (similar to the one in use on crates.io) and also propagates CargoResult in a few more locations.

Overall this ended up being a pretty large refactoring so the commits are separated in bite-sized chunks as much as possible with the end goal being this PR itself.

Closes #2394

alexcrichton added some commits Feb 19, 2016

@alexcrichton
Fuse SourceMap and PackageSet 
This commit moves the SourceMap structure into the PackageSet structure, and
simultaneously massively cuts down on the API surface area of PackageSet. It's
intended that eventually a PackageSet will be a lazily loaded set of packages so
we don't have to download everything all at once, and this is the commit in
preparation of that.
2d25fca
@alexcrichton
Refactor links validation to not use PackageSet 
Eventually we may not have the entire set of packages resident in memory, so
refactor the implementation to validate the `links` attribute in a demand driven
way rather than all at once up front.
5ea7a97
@alexcrichton
Refactor build script output state initialization 
Like with the previous refactor, remove the need to list all packages in a
package set as we only need to lazily do this for the actual packages being
compiled. Whenever a build script is requested to be executed is when we
actually go and try to see if an override was in play.
919cdde
@alexcrichton
Move the `get_package` step later when calculating deps 
Future calls to `get_package` may end up actually downloading a package, so we
want to defer this as late as possible to ensure that we don't download
anything.
dc5671d
@alexcrichton
Remove the ability to list packages in PackageSet 
If we download lazily, that's excessively wasteful!
04c6c74
@alexcrichton
Remove SourceSet 
This isn't used anywhere
c024743
@alexcrichton
Merge the get/download methods on the Source trait 
Nothing currently implements the ability to more efficiently download a set of
packages at any one point in time, and the download/get distinction isn't really
used at all. We can always refactor later, but currently there's no benefit, nor
can it really be seen what the possible benefit is, so let's just merge these
two methods into one and have them operate on one id at a time.
9cf7051
@alexcrichton
Go through `layout` to get doc dir output directories
0b256d3
@alexcrichton
Implement a fallible PackageSet::get 
This function will lazily download the package specified and fill a cell with
that package. Currently this is always infallible because the `PackageSet` is
initialized with all packages, but eventually this will not be true.
b56b61c
@alexcrichton
Move Context::get_package to returning a Result 
The propagate all the `try!`-s needed upwards
078b670

@rust-highfive rust-highfive assigned huonw Feb 22, 2016

@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member Author

alexcrichton commented Feb 22, 2016

r? @brson

@rust-highfive rust-highfive assigned brson and unassigned huonw Feb 22, 2016

@rust-highfive

This comment has been minimized.

Sign in to view
Copy link

rust-highfive commented Feb 22, 2016

r? @huonw

(rust_highfive has picked a reviewer for you, use r? to override)
@alexcrichton
Stop downloading all packages immediately 
Instead delay all downloads until just before they're needed. This should ensure
that we don't download any more packages than we really need to.
Loading status checks…
f994592

@alexcrichton alexcrichton force-pushed the alexcrichton:download-less branch from b63312b to f994592 Feb 26, 2016

@brson

This comment has been minimized.

Sign in to view
Copy link
Contributor

brson commented Feb 29, 2016

@bors r+
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Feb 29, 2016

📌 Commit f994592 has been approved by brson
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Feb 29, 2016

⌛️ Testing commit f994592 with merge 8fc3fd8...

bors added a commit that referenced this pull request Feb 29, 2016

@bors
Auto merge of #2406 - alexcrichton:download-less, r=brson 
Currently Cargo will download an entire resolution graph all at once when in fact most packages may not be relevant to a compilation. For example target-specific dependencies and dev-dependencies are unconditionally downloaded regardless of whether they're actually needed or not.

This commit alters the internals of Cargo to avoid downloading everything immediately and just switches to lazily downloading packages. This involved adding a new `LazyCell` primitive (similar to the one in use on crates.io) and also propagates `CargoResult` in a few more locations.

Overall this ended up being a pretty large refactoring so the commits are separated in bite-sized chunks as much as possible with the end goal being this PR itself.

Closes #2394
Loading status checks…
8fc3fd8
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Feb 29, 2016

☀️ Test successful - cargo-linux-32, cargo-linux-64, cargo-mac-32, cargo-mac-64, cargo-win-gnu-32, cargo-win-gnu-64, cargo-win-msvc-32, cargo-win-msvc-64

@bors bors merged commit f994592 into rust-lang:master Feb 29, 2016
3 checks passed

3 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
homu Test successful
Details

This was referenced Feb 29, 2016

Merged
Merged
Closed

@alexcrichton alexcrichton deleted the alexcrichton:download-less branch Feb 29, 2016

@alexcrichton alexcrichton added the relnotes label Mar 10, 2016

@alexcrichton alexcrichton referenced this pull request Mar 15, 2016

Open

Uplift to the nursery? #268

@bluss bluss referenced this pull request Apr 14, 2016

Open

Fetches unused dependencies #1

@alexcrichton alexcrichton referenced this pull request Mar 12, 2018

Closed

[wip] Download crates in parallel #5161

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.