Caching directories between runs of a task

[vito]

Common use case: fetching dependencies, syncing BOSH blobs, etc.

Caching the directories that these fetch into and update would dramatically speed up a bunch of builds. Our own ATC JS build spends 99% of its time just downloading npm packages.

Proposal:

Add a cache field to task configs, like so:

---
platform: linux

inputs:
- name: my-release

cache:
- path: my-release/.blobs

run:
  path: my-release/ci/scripts/create-release

Then, given I have a pipeline like so:

jobs:
- name: make-release
  plan:
  - get: my-release
  - task: create-release
    file: my-release/ci/create-release.yml

This would cache the directory my-release/.blobs between runs of that specific task in its job's build plan. So, the cache lookup key would be something like team-id+pipeline-id+job-name+task-name.

Notes:

• This should also have the guarantee that two concurrent builds of the same job do not pollute each others' caches. There should be some sort of copy-on-write semantics, such that each job gets its own copy of the cache (initially empty), and at the end all other caches are marked "stale" and are set to expire.
• Assumes tools will be durable to the directory being initially present + empty on the initial cache run. I think this should be fine. Without this it'll be very annoying to orchestrate.
• The caching is for purely ephemeral data, so it doesn't sacrifice Concourse's "not being a source of truth" principle.
• Has the same cache warming semantics as gets, i.e. it may take a bit for the cache to warm across the workers; it does not influence worker placement.
• Has no effect on one-off builds, as there is not enough information to scope/correlate the caches (compared to a job build).

[concourse-bot]

Hi there!

We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.

The current status is as follows:

• #110017520 Caching directories between runs of a task

This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.

[xoebus]

👍

It's an important feature but I can see it getting abused. Maybe we need a best practices document that sets some guidelines for having a sane pipeline. e.g. don't publish a different artefact to the same version number otherwise nothing will work properly.

Maybe we could have a random chance of running a task without a cache to make sure people aren't depending on it. 👀

[nicorikken]

I'd certainly like to see this come about. Our use-case includes downloading NPM dependencies as well, but also downloading Nix-packages. Copy-on-write would suit that use-case.

Sharing this cache even among pipelines would make sense efficiency-wise, although that would introduce (semantic) dependencies between pipelines.

[vito]

@nicorikken We'll probably not start with cross-pipeline or even cross-job caching as that opens another can of worms: how to configure which caches apply to which tasks/etc. I could see that becoming a nightmare pretty quickly if you e.g. copy a job from another pipeline that has a cache name the same as yours because it was called something too generic.

So we'll probably just stick with as tightly-scoped as possible (individual task within job).

[nicorikken]

@vito Totally agree, makes most sense. There are other 'hacks' for getting around such issues, and worst-case somebody would have to write a specific resource targeting the local disk.

[jchester]

👍 from me, both privately (I have a pipeline pulling umpteen zillion jars) and professionally (pivnet is a hungry gem consumer).

[devcurmudgeon]

not sure i understand this discussion properly, but i think just supporting the possibility to have 'cache' directory/ies untouched between jobs would be very useful on its own. various tools have their own cache lookup processes (eg bitbake)

[tomwhoiscontrary]

👍 we've got microservices built with Gradle, so every one of our 27 tasks starts by downloading Gradle, then all our dependencies, and it's horrendous. Cache plox.

As a stop-gap, could we write some sort of pre-loading resource? It would somehow download the Gradle binary and some set of dependencies, then we would copy that into place before kicking off the build. That would let us exploit Concourse's local caching of resources, right?

[simonvanderveldt]

@nicorikken We'll probably not start with cross-pipeline or even cross-job caching as that opens another can of worms: how to configure which caches apply to which tasks/etc. I could see that becoming a nightmare pretty quickly if you e.g. copy a job from another pipeline that has a cache name the same as yours because it was called something too generic.

So we'll probably just stick with as tightly-scoped as possible (individual task within job).

@vito While I do understand it will be hard to make sure people don't use it for the wrong reasons we see a serious speedup with shared caches between pipelines (and by extension also between jobs).
But the caches we share in these cases are the "normal" per user package manager's caches, so for example for npm this is ~/.npm/ and for sbt this is ~/.ivy2/.

Would it be possible to support this somehow?

@vito Totally agree, makes most sense. There are other 'hacks' for getting around such issues, and worst-case somebody would have to write a specific resource targeting the local disk.

@nicorikken Can you share those other "hacks"?