Caching on self-hosted runners

[OmgImAlexis]

So two main issues here.

1. Caching on self-hosted runners makes 0 sense in how it currently works. If I'm self hosting the cache should be on the machine that the runner is on not uploaded/downloaded from Github on every run. This is adding a load of extra bandwidth on both Github's side and our side for no good reason. This is also really unintuitive and it's not mentioned anywhere that I could find. If it is mentioned it's not nearly clear enough. (The only place I can find this is in a single comment from a month or so ago Caching on self-hosted runners github/docs#2271 (comment))

2. The cache on Github is slow as can be when using a self-hosted runner. Some cases it completely stalls from the slow speed. (This is in a EC2 instance and we have no issues with speed against any other sites, etc. including github itself for clones, etc.)

This is the code we're using in the action. With the workflow set to use ubuntu-latest the cache downloads in ~1s or 150MB/s - 200MBs.

    - name: Load npm cache
      uses: actions/cache@v2
      with:
        path: ~/.npm
        key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
        restore-keys: |
          ${{ runner.os }}-node-

[paambaati]

This is super-important for teams running self-hosted runners; especially ephemeral workloads. We run a cluster of self-hosted runners on ECS Fargate, and would prefer a cache that can either be mounted as a volume for the containers or on S3, whichever is flexible in terms of implementation and/or cost.

[matchan26]

I strongly agree. The competing gitlab self-hosted runner supports cloud storage caching (e.g. s3) as standard, so there is no reason why github can't do it.

[HieronyM]

Hi @paambaati, Did you find any solution for caching on ECS-fargate workload?

[crohr]

Hi, have a look at runs-on/cache, which provides s3 backend for the cache action, as a drop in replacement.

[deeno35]

How do we raise visibility on this issue? It's a really big miss from a product standpoint IMO. I am definitely seeing what the OP sees in #2 which is the cache is so slow at times that it stalls / times out my jobs. At this point I am disabling caching in order to get consistent (reliable) behavior which is a travesty.

[edit] If dangling a carrot helps, I have a feeling the egress costs are quite high for GH to host these caches. Some talented engineer/team is going to save GH a wad of cache by helping us solve this.

[clarkmcc]

@deeno35 based on the storage pricing model, the storage and egress costs are marked up quite a bit so the lack of local caching is actually part of the product imo. I'm concerned that Github will actually not be motivated to solve this problem for that reason. Is writing a custom action with more flexible caching options possible?

[starikcetin]

@jehaby I am currently on vacation, can do when I get back.

[makarov-roman]

My solution is also custom script, but I use simple cp, which is good enough. copy is fast cause I use tmpfs for working dir. rsync could be better, but it already <10sec for huge monorepo node_modules

[okoreniuk]

can EFS solve this problem?

[starikcetin]

Hello people, unfortunately, I no longer work for that company, so I can't share the solution. However, it is a very simple one. Take a look at rsync docs and you should be able to implement it yourselves.

[crohr]

Have a look at runs-on/cache!

[strophy]

I'm using https://github.com/everpcpc/actions-cache as a solution to store cache from EC2 runners to S3 and it works fine, although a gateway endpoint is needed in the VPC to ensure there are no egress costs. Works fine for me.

[LouisHaftmann]

We encountered an issue with custom cache actions not working when other actions (like setup/node) use the default actions/cache. Our solution was to ensure our custom cache is compatible with the default, eliminating the need to alter our workflow files. If anyone needs it, here you go: https://github.com/falcondev-oss/github-actions-cache-server.

[mmatl]

This seems extremely helpful, thank you for sharing!