[Casher] Distinct caches for each job in a build matrix

[BanzaiMan]

Originally opened by @theuni as https://github.com/travis-ci/casher/issues/6 with the title "c/c++ workers share a cache when environment variables are used to build the matrix"

---

As discussed on IRC with @joshk

I believe this is actually a problem in core and not here, but logging a ticket here for discussion.

For c/c++, it seems that the compiler is the only differentiating factor when creating the slug.

Take this example:

language: c
cache:
  directories:
  - mydir
env:
  - FOO=foo BAR=bar
  - FOO=bar BAR=foo

The default compiler will be gcc.
For both builds, the slug will be: cache--compiler-gcc

This means that the builds will stomp each-other's caches. In order to prevent this, the slug would need to be appended with some unique id for each builder. The most reasonable approaches I can come up with would be:

• A hash of the env
• A user-specified unique identifier

As an incredibly hackish work-around, I've used a phony compiler (one that can run "compiler --version" without failing) here: https://travis-ci.org/coryfields/bitcoin/builds/33134451

The "true X" compilers will each result in different slugs, so they get independent caches:

On branch master:
cache--compiler-true1  last modified: 2014-08-20 23:03:03  size: 27.95 MiB
cache--compiler-true2  last modified: 2014-08-20 23:00:08  size: 23.57 MiB

It would be nice to be able to do something like this:

matrix:
  include:
    - cache-id: foobuild
      env: FOO=foo BAR=bar
    - cache-id: barbuild
      env: FOO=bar BAR=foo

[BanzaiMan]

There was a discussion as to how to implement this; one idea is to use language versions and env vars to create a cache_id that acts as an identifier.

This effectively supersedes #3745, which raises a real possibility of cache corruption when multiple jobs upload the caches simultaneously.

[weitjong]

Why not just use the last digit of TRAVIS_JOB_NUMBER?

In ruby

job_number = "#{ENV['TRAVIS_JOB_NUMBER'].split('.').last}"

In bash

job_number=${TRAVIS_JOB_NUMBER##*.}

I have successfully used this approach to roll my own cache store backed by Github for my CI jobs on standard build infra. For container-based build infra where Travis built-in cache store is available, I use the similar fake compiler approach as described above to get by the cache thrashing problem. See my analysis in #3699. My point is, a same cache should never be used by more than one job, so using job number as the cache identifier is a no brainer to me. All my CI jobs always have good cache hit ratio now after ensuring this.

[BanzaiMan]

You can see the past comments here: https://gist.github.com/BanzaiMan/c6f00138e7354b37fd66

[theuni]

Yes, using the job number makes sense.

[weitjong]

@BanzaiMan I am very happy to learn there is plan or discussion going on to fix the "cache thrashing" issue. I think we all agree we need a unique cache id for each job. The question is just how that cache-id should be best computed. IMHO, using a complex hash function may not be necessary. My proposed approach is dead stupid but effective. Of course, it has its own corner case where the stored cache is being invalidated when user joggling around his/her job matrix, but I reckon normal user would not do this frequently and furthermore the invalidated cache will be self-correcting in the next build. Another pro is when creating new topic branch, all the cache can easily be "inherited" initially from the master branch based on their job number.

[AbdealiLoKo]

This is something I need for my project too - without it, builds take way too long

[joshk]

We will look into this feature more in the coming weeks, but there is not quick fix sorry.

[arthurmensch]

Hi,

It would be great to have this feature !

[localheinz]

👍

[cvrebert]

+1 from Bootstrap. Since #4090 isn't forthcoming (and is understandably complicated), this would be the next best thing.

[discordier]

+1 also from various PHP developers.
This corrupts all composer based builds as it corrupts the caches when having a build matrix of PHP versions and different dependency versions which are not updated very rarely.

Only solution so far seems to:

• disable cache (ugly, as it requires loads of bandwidth)
• symlink the cached directories to some other location on before_install and limiting the concurrent builds to 1 (which keeps the caches at least updated but takes even more time as the builds are now sequential).

Therefore appending the build matrix job number to the cache slug would be highly appreciated.

[BanzaiMan]

Just a heads up: I deployed a fix to production about 10 minutes ago. Please observe how your caches are behaving, and report issues if you see any.

[BanzaiMan]

Allowing custom cache names is an interesting proposition, but it is not implemented in the fix mentioned above.

[BanzaiMan]

This has been deployed and documented.

[weitjong]

Thanks! With this I think I could decommission our custom cache store and switch back to use Travis-CI internal one. The only drawback I can think of with current implementation is that there is no way to exclude certain environment variable to be used/joined in the hash key computation (while we don't have to worry about that when job number is being used as the sole cache key), but I guess we can live with that.

[theuni]

Thanks for this!