Please consider using several partial keys in restore_cache

[julienw]

Describe Request:

It is very common, and actually recommended in the CircleCI documentation, that we specify several keys in restore_cache, so that the cache can be reused and primed even if the underlying lock files change.

For example, Here is a real-life configuration from my project:

Saving is:

      - save_cache:
          paths:
            - node_modules
          key: v2-dependencies-{{ checksum "package.json" }}-{{ checksum "yarn.lock" }}

Restoring is:

      - restore_cache:
          keys:
            - v2-dependencies-{{ checksum "package.json" }}-{{ checksum "yarn.lock" }}
            - v2-dependencies-{{ checksum "package.json" }}-
            - v2-dependencies-

But this isn't what the orb is doing:

node-orb/src/commands/install-packages.yml

Lines 62 to 67 in a8e4186

	- when: # Restore cache for NPM
	condition: << parameters.with-cache >>
	steps:
	- restore_cache:
	keys:
	- node-deps-<<parameters.cache-version>>-<<#parameters.include-branch-in-cache-key>>{{ .Branch }}-<</parameters.include-branch-in-cache-key>>{{ checksum "<<parameters.app-dir>>/package-lock.json" }}

node-orb/src/commands/install-packages.yml

Lines 100 to 105 in a8e4186

	- when: # Restore cache for YARN
	condition: << parameters.with-cache >>
	steps:
	- restore_cache:
	keys:
	- node-deps-<<parameters.cache-version>>-<<#parameters.include-branch-in-cache-key>>{{ .Branch }}-<</parameters.include-branch-in-cache-key>>{{ checksum "<<parameters.app-dir>>/yarn.lock" }}

Is there a drawback to always doing this? Or should we make this a configuration option?

I'd be happy looking at this if there is consensus about it.

Thanks!

[patatepartie]

I would just add the {{ arch }} template, inside the key, as it's recommended in some parts of the docs and would help people with builds on different architectures, at no cost for those who don't.

[KyleTryon]

Yes, unfortunately, because caching is achieved via CircleCI config and is not "programable", it would be difficult to support multiple cache keys via an orb. The addition of {{ arch }} however should assist some (I can't think of any downsides).

In semi relation, I have also opened this feature request for CircleCI to allow more programmatic access to cache and artifacts. Please feel free to vote here: https://ideas.circleci.com/api-feature-requests/p/programmatic-cache-and-artifact-access

[julienw]

I think it's quite easy to support the partial keys, what's more difficult is to make it configurable :-) But not sure there's a downside in having it all the time.

[Kurt-von-Laven]

Yeah, sorry, could you clarify that point, @KyleTryon? I don't see why this feature request would imply any special programmatic access to caches. The proposal as I understand it is to apply partial cache restoration as recommended by CircleCI. Having partial cache restoration all the time would break NPM users without a lock file as well as NPM users on version 4-. That being said, I don't personally feel there is a need for this feature being configurable at least in its first iteration since the command can inspect the presence of a lock file and npm --version and branch accordingly.

[KyleTryon]

It looks like we could enable this as non-configurable.

[KyleTryon]

Have a solution prepared: #83