Automatically enable caching of Rust packages

[alex]

Right now by default when you have a language: rust repo on travis, it redownloads your dependencies on every run. I believe caching /home/travis/.cargo/ would resolve this, but someone who works on Cargo/Rust should probably chime in.

[frewsxcv]

/cc @alexcrichton and other people on the Rust tooling team

[alexcrichton]

Aha, thanks for the cc! Here's a bit of a rundown on where cargo downloads deps:

• The $HOME/.cargo folder is the default value for Cargo's home directory (where it downloads all metadata/packages). This can be changed by setting the CARGO_HOME environment variable.
• Inside Cargo's home directory there are two primary locations today, git and registry.
• The git folder contains basically a git db for all git dependencies crates (Rust libraries) have. For example if I depend on another git repository it will be cloned by Cargo into this folder. Not much interesting here beyond that.
• The registry folder contains three sub-directories:
  • index - this is the checked out version of Cargo's "index". The index is a listing of all libraries on crates.io (our default remote crate source) and is itself a git repository. Cargo will update this if it needs to, but the idea is it's incrementally updated through the use of git.
  • cache - this location is where all crates are downloaded to by default. Each crate is currently a tarball
  • src - this is where tarballs are extracted to.

In other words, Yes caching .cargo should do the trick! If you have any other questions though, please feel free to just ask!

[BanzaiMan]

Thanks for the info. Which directories are useful in caches? Users can specify, of course, but I suspect @alex wants a shortcut such as:

cache: cargo

[alex]

@BanzaiMan yeah, I think cache: cargo should be the default on language: rust repos.

[alex]

@alexcrichton It looks like caching $HOME/.cargo will not actually cache compiled crates though, is that right?

[alexcrichton]

Yeah, that's correct. All compiled output is in the project's local target folder by default.

[BanzaiMan]

So, the right directory to cache for the proposed cache: cargo is $TRAVIS_BUILD_DIR/target?

[alexcrichton]

It's generally pretty rare that you actually can cache binary artifacts across runs of the compiler, it requires a number of stars to align:

• The exact same compiler must be used. Many Rust projects test on all three of the nightly, beta, and stable channels meaning they may not hit the cache very often. This also means that every nightly build will rarely use the cache (as the compiler's changing) and every 6 weeks (our release cycle width) all the caches will be busted.
• All upstream dependencies must be the exact same version. Dependencies for libraries tend to change over time as they publish new versions, so this is also unlikely to reduce the cache hits.

Overall I'd recommend only caching $HOME/.cargo which is known to be statically never-changing data. That contains data like the crates.io index (a git repo that can be updated in a delta-fashion) and downloaded sources (which will never change).

[alex]

I've thought about this on and off for the last 6 weeks. Here's why I believe caching built libraries would be worth it:

• The returns are huge: for a small library with only 2 direct dependencies, I spend 20+ seconds in build, and 2 seconds in test. This only gets worse as your dependency count goes up.
• I believe travis already has a cache-per-configuration, sthe issue of nightly vs. beta vs. stable busting caches is ameliorated.
• Being cache busted every 6 weeks is no big deal, that's forever! :-)
• Projects with any level of active development will get a win. Sure dependencies drift over time, but when I'm actively working on a project I can ring up dozens of commits per day, dependendencies don't drift that quickly.
• Seeing the impact of caching wheels in Python makes me hungry for faster tests :-)

Given that, I think caching binary artifacts would be good.

@alexcrichton if I cache the target/ directory, will that do the right thing with regards to "use that file if it's the right version, compile a fresh one if a dependency increases"?

[alex]

Hmm, it looks like I'm not correct about there being a per-environment cache. @BanzaiMan how hard would it be to split the cache out based on which rust channel you're using?

[alexcrichton]

@alex yep, that should do the trick!

[alex]

Cool, it sounds like making Travis have a separate cache per Rust channel is the only missing piece then.

[joshk]

We can easily do separate caches per Rust channel! I can talk to @BanzaiMan about this later in the week.

[alex]

@joshk Cool, please let me know if that's somethign I can help out with!

[BanzaiMan]

To recap, caching $HOME/.cargo is recommended?

For the name, is cargo OK? That is,

cache: cargo

[alex]

Yes, I also feel that caching target/ would be valuable if we were able to split out caches out by rust line.

[BanzaiMan]

@alex By "split out caches out by rust line", you mean the rust value in build matrices? Then, yes, that is reasonable.

[alex]

Exactly.

On Wed, Oct 28, 2015 at 8:12 AM, Hiro Asari notifications@github.com
wrote:

@alex https://github.com/alex By "split out caches out by rust line",
you mean the rust value in build matrices? Then, yes, that is reasonable.

—
Reply to this email directly or view it on GitHub
#4470 (comment)
.

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

[BanzaiMan]

@alex @frewsxcv @alexcrichton The PR above defines cache: cargo, which adds ~/.cargo and target to the cache. Is that acceptable? If so, I will merge and deploy it soon.

[alex]

Sounds good to me!

[BanzaiMan]

To be clear: caches will be marked with rust version, as well as the branch.

[alexcrichton]

@BanzaiMan sounds perfect!

[BanzaiMan]

Deployed and documented (https://docs.travis-ci.com/user/caching/#Rust-Cargo-cache)! Thanks for your help, everyone!

[alexcrichton]

Awesome, thanks @BanzaiMan!