Support for Cross OS caching enabled

[Phantsure]

Problems

There have been multiple reported issues related to compression of caches. The issue we are looking to solve:

• Cross-OS compatibility: Currently on windows cache uses a different compression algorithm (gzip) as compared to on linux | mac (zstd). This leads to different versions for caches created on different platforms. Therefore caches created on windows might not be recoverable on linux | mac. For more details on cache version see this.

Proposal

We are looking to solve both the problems as follows::

• Change the default tar used on windows runners to GNUtar. This is already suggested as a workaround for people in these problems. Same tooling will ensure that cache can be reused across all three OSes.
• Fallback to BSDtar with zstd on windows. BSDtar is already present on Windows runners by default but it does not use zstd due to the issue of compression hanging with large caches. In our testing, we found that performing archiving and compression as separate processes (instead of calling tar --use-compress-program) does not have the hang problem for caches of size up to 2GB.

Reasoning to choose GNUtar over BSDtar as default

BSDtar has some implementation problems. That’s the reason it stopped being used in MacOS for our action. For more details see actions/toolkit#552.

Related issues which should get fixed with this proposal

We have consolidated issues related to the above problems here. Feel free to provide feedback regarding these in this issue itself.

• Cache misses due to path-based versioning and its normalization #330
• Support multiple compression algorithms when restoring cache toolkit#543
• zstd compression hanging on Windows with large files #301
• Actions/cache breaks symlink on Windows runners #1010
• Caching problem - unpacking tar #198 (Should be fixed with use of GNUtar)
• Cache action shouldn't use first tar on the path #125 (Won’t fix)
• Allow choosing cache compression algorithm explicitly #389 (Not fixing as of today)

[Phantsure]

New beta release v3.1.0-beta

We have released a beta version of cache under actions/cache@v3.1.0-beta for users to test for functionality. This should cache using GNU tar with zstd as compression and has BSD tar with zstd as fallback on windows. Functionality on linux | mac should remain same.

Expected behaviour

All the mentioned issues in here should be resolved and not happening with this release. Performance might vary for different files on windows. No changes should happen to mac | linux cache performance

Call outs

• Old caches on windows wouldn't be recovered as different compression algorithm is used in beta.
• Don't expect any or major performance improvement

[Phantsure]

Update

Tag v3.1.0-beta has been updated to account for old caches saved by gzip compression on windows and fallback to gzip while recovering and decompressing. So old caches should be recoverable and new caches would be compressed using zstd.

Call out

• While BSD and GNU tars are most compatible there might be some onetime corruption as old caches were saved using BSD tar and recovered using GNU tar.

[Phantsure]

Update

Have released all the changes to v3. Now cache would be able to work cross os. Go ahead to release notes for details on what new is added: https://github.com/actions/cache/blob/main/RELEASES.md#321
Keeping this issue open for sometime for feedback

[Phantsure]

Update

We have reverted the changes due to multiple complains regarding symlinks not working. If a cache containing symlink is saved on linux|mac and restored on windows then GNU tar not able to recover those symlinks. We are actively working to resolve this issue, till then have reverted the changes.

Related issue: #1043

[Phantsure]

Update

Currently we are fixing a problem related to symlinks creation on windows during a tarball extraction which was created on linux|mac. Common example is caching a node_modules folder. Original issue which was reported and lead to reverting original changes.

Solution:

We found that symlink creation is handled differently for different platforms and filesystems. We encountered two solutions to handle this:

• Use --dereference with tar on creation to remove any symlinks present while creation:
  • Pros: Removed symlinks entirely while creation so no problem while handling
  • Cons: Size of tar would increase as this would create copies of same file.
• Add environment variable MSYS=winsymlinks:nativestrict to handle symlinks as per windows symlink: Blog. This works for any tar based on msys2 platform, which is our current tar from git-for-windows.
  • Pros: Creates a symlink for any unix based symlink so should work as usual.
  • Cons: Might break if symlink created is still not handled by code. So user should still understand that symlinks on cross os caches might break.

We have chosen to go with 2nd solution and would love hear feedback from community.

[Phantsure]

Update

We have released new changes with symlink issue fixed and allowing to use cross-os cache as an opt-in feature by adding enableCrossOsArchive input as true to cache action.
Note: All previous workflows should work as usual with cache miss for first time on windows due version changes done. This was done to keep previous workflows on windows cache separate as compared to cross-os cache.

[mrclary]

I'm having problems restoring caches to macOS that were saved on Linux.
My understanding was that save/restore caches between linux and macos should work since cache@v3 uses zstd compression so the cache version should be the same for both systems.

I build a (platform independent) conda package on ubuntu-latest and cache the build directory, but restoring the cache in a job using macos-latest results in a miss due to cache version difference, (see https://github.com/mrclary/spyder/actions/runs/5629412107/job/15254349637)

##[debug]No matching cache found for cache key 'qtconsole_fbd114fbc39b14126e02f1d5eeb35c1f0191377b93473eac16ed1b521cd90602', version '4990d5fb46f0a66ca61e3fc600e6afb08e363e2efa594058cd1e73253237534e and scope refs/heads/build-cache. There exist one or more cache(s) with similar key but they have different version or scope. See more info on cache matching here: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#matching-a-cache-key 
##[debug]Other caches with similar key:
##[debug]Cache Key: qtconsole_fbd114fbc39b14126e02f1d5eeb35c1f0191377b93473eac16ed1b521cd90602, Cache Version: 723cd88e4181a478b4425d3a170e5b990c7fcec8baef2b9e0fe0dedf2a7d4edd, Cache Scope: refs/heads/build-cache, Cache Created: 2023-07-22T06:53:55.95Z

In the run linked above, the job "Build qtconsole Python-3.10" runs on ubuntu-latest and properly restores the cache from a previous run. The job "Build Installer for osx-64 Python-3.10" runs on macos-latest and should restore the same cache (step "Restore qtconsole Cache") but misses due to cache version.

What am I missing here?

[Phantsure]

As per my understanding your Linux cache is saved with a different path than MacOs cache. Path is used to determine version of cache which should be same when trying to restore. Docs for cache version: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#matching-a-cache-key

Try using relative paths if possible?

[mrclary]

Ahh, yes. Thank you, that was the issue. Now I just need to figure out how to use ../ relative parent directory in the path specification...