Yarn install does not use the cache-path parameter

[patatepartie]

Orb version:

4.1.0

What happened:

I have the following configuration:

version: 2.1

orbs:
  node: circleci/node@4.1.0

jobs:
  test:
    docker:
      - image: cimg/node:12.18.2
        auth:
          username: $DOCKERHUB_USERNAME
          password: $DOCKERHUB_PASSWORD
    steps:
      - checkout
      - node/install-packages:
          pkg-manager: yarn
          cache-path: ~/.cache/yarn

workflows:
  test:
    jobs:
      - test:
          context:
            - docker-hub-creds

There is, unfortunately, no visual way to describe what I think is happening, because yarn install's output is the same whether the dependencies are cached or not. Only the duration can be an indicator, but it's too variable between jobs to be reliable.

If I understand correctly the code, when cache-path is specified, that's the path which will be cached by save_cache.
That path is, however, not used by yarn install, unless I override the command with override-ci-command: yarn install --frozen-lockfile --cache-folder ~/.cache/yarn.

Expected behavior:

The yarn install command uses the cache directory specified by cache-path.
Alternatively, the documentation could be clarified to explain that the install command needs to be overridden with override-ci-command to specify the cache folder.
Currently the documentation for the cache-path parameter is confusing for yarn users:

By default, this orb will utilize 'npm ci' and cache the '~/.npm' directory. Override which path to cache with this parameter.

Additional Information:

AFAIK, this is also true for npm.
I haven't used recent versions of npm and I don't know how its cache works, though.

[julienw]

I found your issue and had a look this morning, because I was curious. I added a lot of logs to my build to try and troubleshoot this.

I found that yarn uses ~/.cache/yarn by default, as it should.
However... it won't populate it if node_modules is present and uptodate, because it won't try to update it. This means that if your cache with the same cache key already contains node_modules, then no new file will be added to .cache/yarn.

Here is the output of restore_cache:

Found a cache at node-deps-v1-circleci-project-setup-oEx3_vcLhzBBgJRwMsY+dClTUDNYYPpwBWXBvzrAH4g=
Size: 20 MiB
Cached paths:
  * /home/circleci/project/node_modules

Downloading cache archive...
Validating cache...

Unarchiving cache...

Maybe what isn't obvious is that cache-path is used for the save_cache command, not the restore_cache command.

Moreover, trying to save to an existing cache key doesn't work either: this is skipped instead of being replaced.

Initially I got the same problem as you when I tried on my project. I could fix it by adding cache-version: v2 to invalidate the cache.

Therefore I think that everything works like it should :-) and that this bug could be closed.

Hope this helps!

[patatepartie]

Thanks for taking the time to look at the issue @julienw.

I found that yarn uses ~/.cache/yarn by default, as it should.

I didn't realize that was the default. I don't remember how I chose this value, but that wasn't by looking the default up.
In that particular case, indeed, I don't need to override the install command.

If I specify a cache-path different from yarn's default cache dir (~/.other-cache/yarn, for instance), it won't work anymore, however, since the default install command will use yarn's default cache location (~/.cache/yarn).
That was the point of this ticket. The default yarn install command, when cache-path is specified, should be yarn install --frozen-lockfile --cache-folder << parameters.cache-path >>
In that regard, I think the issue is still opened. At least if the maintainers think the use case is worth fixing.

Regarding the node_modules comment: indeed, if it's already present in the job cache (by not having specified cache-path in a previous job, for instance), and is up-to-date with the lock file, the yarn cache will not be used and save_cache will be skipped. Updating the cache version is the solution here, as you mentioned.
That wasn't the problem I had, though, but at least now I understand why save_cache caches node_modules when cache-path is not specified.

One thing I just noticed and which could indicate this Orb's purpose doesn't match my use case, is that restore_cache will only restore on the exact same cache key. The circleci doc (and my previous "manual" implementation) use multiple cache keys for sharing caches, at least partially, between jobs on different branches, in addition to jobs on the same branch.
This is a whole different issue from this one (and deserve its own ticket), though.

[julienw]

If I specify a cache-path different from yarn's default cache dir (~/.other-cache/yarn, for instance), it won't work anymore, however, since the default install command will use yarn's default cache location (~/.cache/yarn).

Yeah that's right. I wonder what is the use case for this though :-)

Regarding the node_modules comment: indeed, if it's already present in the job cache (by not having specified cache-path in a previous job, for instance), and is up-to-date with the lock file, the yarn cache will not be used and save_cache will be skipped.

This is even a bit more subtle: if you try to save with the same key as an existing cache (because same lock / same branch), then the save will be skipped altogether, even if there are files to be saved.

So there were actually 2 problems here, if the cache already exists: 1/ no new file in .cache/yarn because node_modules is uptodate, and 2/ even with some files, we'd need a different key name anyway.

One thing I just noticed and which could indicate this Orb's purpose doesn't match my use case, is that restore_cache will only restore on the exact same cache key. The circleci doc (and my previous "manual" implementation) use multiple cache keys for sharing caches, at least partially, between jobs on different branches, in addition to jobs on the same branch.
This is a whole different issue from this one (and deserve its own ticket), though.

Yeah, actually this is the issue I wanted to file initially and that's why I looked at other issues and found this one first :D I was planning to file this issue indeed.

Update: filed #63

[patatepartie]

Yeah that's right. I wonder what is the use case for this though :-)

I don't have any specific use case in mind 🤔 , but since the Orb allows changing the cache path, it should use it in the yarn install. Otherwise it's inconsistent IMO.
If the intent of the cache-path parameter was just to cache the default cache path instead of the node_modules directory, then a boolean parameter would have been better.

Update: filed #63

Thank you.

[Kurt-von-Laven]

I also found it counter-intuitive that cache-path only works gracefully with npm rather than yarn. I would say it should at least be documented that cache-path should not be used with yarn unless you override the install command as well.

[Jaryt]

Hi! Thank you for your patience. I'll look at getting this into the next major version if possible :)

[Jaryt]

Implemented in #94

Please feel free to review the changes and see if they work for your use-case 😄