Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Caching the dependencies #6853

Merged
merged 10 commits into from Apr 26, 2019

Conversation

Projects
None yet
5 participants
@Eh2406
Copy link
Contributor

commented Apr 15, 2019

There are 2 sources of facts for the resolver:

  1. The Registry tells us for a Dependency what versions are available to fulfil it.
  2. The Summary tells us for a version (and features) what dependencies need to be fulfilled for it to be activated.

The Registry was cached with a RegistryQueryer back in #5112. This adds a DepsCache to cache the calculation of which dependencies are activated by features.

In the happy path flag_activated means that we don't get to reuse build_deps, but the more we backtrack the more time we save. In pathological cases like #6258 (comment), I have measured this as a 10% improvement with release.

This also means that build_deps can be run in a context free way, which may be useful in a follow up PR to solve #6258 (comment).

@rust-highfive

This comment has been minimized.

Copy link

commented Apr 15, 2019

r? @nrc

(rust_highfive has picked a reviewer for you, use r? to override)

@Eh2406 Eh2406 force-pushed the Eh2406:caching-the-dependency branch from be40c3a to 2246b8b Apr 15, 2019

@nrc

This comment has been minimized.

Copy link
Member

commented Apr 15, 2019

@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 18, 2019

☔️ The latest upstream changes (presumably #6840) made this pull request unmergeable. Please resolve the merge conflicts.

@Eh2406 Eh2406 force-pushed the Eh2406:caching-the-dependency branch from 2246b8b to 2bd11f0 Apr 18, 2019

@alexcrichton
Copy link
Member

left a comment

Looks pretty reasonable to me, thanks for separating the commits! I think the organization between DepCache and RegistryQueryer may want to be tweaked slightly, but it's not the end of the world either way.

The main thing I think this needs is to expand the documentation as to why this caching strategy is valid, aka just some documentation on DepsCache as to what it's caching and why it's possible.

Show resolved Hide resolved src/cargo/core/resolver/context.rs Outdated
@@ -89,26 +89,26 @@ impl ResolverProgress {
}
}

#[derive(Clone, Copy)]

This comment has been minimized.

Copy link
@alexcrichton

alexcrichton Apr 19, 2019

Member

This seems like it's somewhat of a loss, but is there a reason we couldn't use &'a BTreeSet here for the Required list?

This comment has been minimized.

Copy link
@Eh2406

Eh2406 Apr 24, 2019

Author Contributor

I don't see how. (love to be proven wrong.) This change was to get rid of the lifetime, so that it could be used as the key in the DepsCache above. Then I switched from a Vec to a BTreeSet to reduce the number of equivalent configurations that can end up in the cache. Then added the Rc as that was the kind of reference I usually had.

This explanation should be a comment. I will add after #6860 lands.

Show resolved Hide resolved src/cargo/core/resolver/dep_cache.rs Outdated
@@ -167,6 +167,36 @@ impl<'a> RegistryQueryer<'a> {
}
}

pub fn build_deps(

This comment has been minimized.

Copy link
@alexcrichton

alexcrichton Apr 19, 2019

Member

Similar to below, could this document the return value and what each value of the tuple is?

This comment has been minimized.

Copy link
@alexcrichton

alexcrichton Apr 19, 2019

Member

(also while you're at it if you can add general documentation for what this method is doing that'd be great!

Show resolved Hide resolved src/cargo/core/resolver/dep_cache.rs Outdated
candidate: &Summary,
method: &Method,
) -> ActivateResult<Rc<(Vec<String>, HashSet<InternedString>, Rc<Vec<DepInfo>>)>> {
if let Some(out) = self

This comment has been minimized.

Copy link
@alexcrichton

alexcrichton Apr 19, 2019

Member

Could a comment be added here as to why we can cache the results? It'd be good to explain why this cache works the way it does and what enables it (e.g. the things that don't change and how this is a "pure" query which can be memoized.

@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 25, 2019

☔️ The latest upstream changes (presumably #6860) made this pull request unmergeable. Please resolve the merge conflicts.

@Eh2406 Eh2406 force-pushed the Eh2406:caching-the-dependency branch from 2bd11f0 to 0c24471 Apr 25, 2019

@Eh2406 Eh2406 force-pushed the Eh2406:caching-the-dependency branch from 0c24471 to 5ae13e3 Apr 25, 2019

@Eh2406

This comment has been minimized.

Copy link
Contributor Author

commented Apr 25, 2019

How does this look now?

@alexcrichton

This comment has been minimized.

Copy link
Member

commented Apr 26, 2019

@bors: r+

Looks great to me, thanks for the updates!

@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 26, 2019

📌 Commit 79714ba has been approved by alexcrichton

@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 26, 2019

⌛️ Testing commit 79714ba with merge 0c891a0...

bors added a commit that referenced this pull request Apr 26, 2019

Auto merge of #6853 - Eh2406:caching-the-dependency, r=alexcrichton
Caching the dependencies

There are 2 sources of facts for the resolver:
1. The `Registry` tells us for a Dependency what versions are available to fulfil it.
2. The `Summary` tells us for a version (and features) what dependencies need to be fulfilled for it to be activated.

The `Registry` was cached with a `RegistryQueryer` back in #5112. This adds a `DepsCache` to cache the calculation of which dependencies are activated by features.

In the happy path `flag_activated` means that we don't get to reuse `build_deps`, but the more we backtrack the more time we save. In pathological cases like #6258 (comment), I have measured this as a 10% improvement with release.

This also means that `build_deps` can be run in a context free way, which may be useful in a follow up PR to solve #6258 (comment).
@bors

This comment has been minimized.

Copy link
Contributor

commented Apr 26, 2019

☀️ Test successful - checks-travis, status-appveyor
Approved by: alexcrichton
Pushing 0c891a0 to master...

@bors bors merged commit 79714ba into rust-lang:master Apr 26, 2019

3 checks passed

Travis CI - Pull Request Build Passed
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
homu Test successful
Details

@Eh2406 Eh2406 deleted the Eh2406:caching-the-dependency branch Apr 30, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.