Cache misses when different projects shares same dependencies #196
Comments
|
I have some experience with Gradle, a Java build tool. I think Cargo is pretty much equivalent at the same role in Rust as Gradle in Java world. Ideally, I wish Cargo will eventually integrate the same caching and incremental building functionality as Gradle does. So learning how Gradle does these would help Cargo and Rust community. In Gradle, a build consists of tasks, and the task dependency graph is created at the beginning of the build. However, whether a task will actually execute depends on the state of the task inputs and outputs. As the doc says, if the Gradle found a task's inputs and outputs are unchanged, such task is considered UP-TO-DATE and will not run. I'm not familiar how Cargo handles task dependency internally, but in long term, instead of relying on command arguments and environment variables, maybe Cargo/sccache could define "inputs" and "outputs" of its "tasks". And the caching will then rely on these more reliable objects. Of course, the "inputs" should not only include the content of source file but also the set of recognized parameters to Basically, I think the current Rust support is like write code in a language without types. Everything is string. Developing a strong-typed system for it would make it more reliable and maintainable. |
|
We have definitely discussed making cargo able to do this caching internally. It ought to be able to do even better than sccache by checking the entire dependency graph and caching the output of the entire build, so that if you have previously built the exact same source it should be a quick cache hit. That bug belongs on cargo, though. :) To your original bug: I have some extra Rust cache debug code on a branch that I've never gotten around to landing: https://github.com/luser/sccache/tree/rust-hash-info I just rebased it to master and tried your experiment locally. You can try it out yourself by building sccache from that branch and then running your cargo compiles with --- /tmp/sccache-rust-debug/itoa_2017-10-26_12:13:46 2017-10-26 12:13:46.785861224 -0400
+++ /tmp/sccache-rust-debug/itoa_2017-10-26_12:13:56 2017-10-26 12:13:56.369916844 -0400
@@ -13,8 +13,8 @@
-C
extra-filename=-bb9fe302248b7433
--out-dir
-/build/snippet/target/debug/deps
+/build/snippet2/target/debug/deps
--cap-lints
allow
-L
-dependency=/build/snippet/target/debug/deps
+dependency=/build/snippet2/target/debug/depsSo the issue is that the path to |
|
@luser yeah removing I think that sccache is already hashing the relevant files (native libs, rust libs, etc), so should be safe to remove both! |
|
I looked into this a bit more to see if I could merge #206, but it looks like currently rlibs wind up with the build directory encoded in them in several places. I have a simple test crate that has only The generated rlib files aren't the same, though: Running them through diffoscope shows that the build directory shows up in various places ( I tried using the path prefix remapping available in nightly Rust to see if I could make them match with that, running the same builds like (nicer HTML view of the diff, only available for 30 days) |
|
I filed rust-lang/rust#48019 and mw submitted a potential fix in rust-lang/rust#48162 . |
|
I tested with the latest nightly (1.27.0-nightly). It seems the problem is solved. Can anyone verify? BTW, thanks for the work! |
|
OK, I got around to re-testing this with latest nightly Rust and it does seem workable! My simple test script shows that I can build the same crate from two different directories passing I wrote a patch for sccache that makes this work, but I had to do some local fiddling. Notably, I had to invoke cargo with I'm not sure that patch is 100% correct yet, because if you don't specify the |
|
@mitchmindtree I forget where I left off on that work exactly. I think we decided that we ought to include the current working directory in the hash for Rust compilation (at least when debug symbols are enabled, I think) and then we could handle the |
|
I rewrote my patch because rebasing it would have been a pain. It seems to work reasonably well in testing: That should fix this issue as filed: if you have two crates that depend on the same crates.io crate they should be able to get cache hits with that patch. (Obviously you need to be compiling the exact same version of the crate with the exact same versions of its dependencies and the compiler options need to match as well.) |
I have Rust 1.22 nightly installed. I installed sccache with
cargo install sccache(sccache 0.2.2). I followed instruction to set up RUSTC_WRAPPER=sccache. I noticed that even multiple projects shares same dependencies, sccache will not have cache hits as expected. My OS is Windows 10.Steps to reproduce:
snippet1with one dependency and no transitive dependency:Cargo.lock may look like
snippet1tosnippet2with no change.cargo buildthesnippet1project. Thencargo cleanandcargo buildagain. Observe 1 cache hit insccache -s.cargo buildthesnippet2project andsccache -s.Expected result:
sccache should now report 2 cache hits.
Actual result:
sccache still reports 1 cache hit, indicate itoa 0.3.4 from
snippet2was not a cache hit.Now, create
.cargo/configfile in the parent directory and point a commontarget-dirfor the two projects. Repeat steps 1-4. sccache will report 2 cache hits.This shows that sccache probably uses the cargo parameters
--out-diror even-L dependencyas part of the hash key. I'm not 100% sure if there is any corner case, but this behavior defeats the purpose of sharing common library across projects with sccache.Related cargo output for
snippet1:snippet2:The text was updated successfully, but these errors were encountered: