Cache package release for compression change in windows with symlink fix #1291

Phantsure · 2022-12-29T07:29:24Z

Adding changes related to compression and cross os functionality back on windows.
Adding symlink fix for Linux cache to restore on windows. Check Actions/cache breaks symlink on Windows runners cache#1010 for more details regarding issue.
Making cross-os functionality as opt-in instead of default

Please check commits after first commit to understand the changes done
Original PR: #1281

This reverts commit 86fe4ab.

packages/cache/package.json

packages/cache/src/internal/cacheHttpClient.ts

bishal-pdMSFT · 2022-12-30T07:38:03Z

packages/cache/src/internal/cacheHttpClient.ts

+      !compressionMethod || compressionMethod === CompressionMethod.Gzip
+        ? []
+        : [compressionMethod]


Should we remove this as we are not doing any special casing for gzip?

It will cause cache invalidation, but that will get masked by invalidation due to windows-only

I think this is there so that cache created by gzip is only restorable by gzip. If self-hosted runners don't have zstd then it falls back to gzip.
We can remove this if we callout that having zstd in path is a requirement for cache to work

Then in that case we can remove gzip from our code. And that seems like a big change

Won't the fallback logic take care of this?

@bishal-pdMSFT Yes, but you were asking if we should remove it. Right? We need to keep some component of compression algo in version

So just to understand, if we remove this statement, cache invalidation will happen because the newer versions will include gzip in their version hash and that won't match for older gzip caches. So even in this case, shouldn't gzip caches only be restorable by using gzip. Previously they were identified by not having gzip in version but moving forward will have gzip in version. I think this is what @bishal-pdMSFT is referring to.

Previously they were identified by not having gzip in version but moving forward will have gzip in version.

Yes, this is what I meant. We need not put empty compression method here for gzip and let it be handled just like any other compression method.

Previously they were identified by not having gzip in version but moving forward will have gzip in version.

Then we can do that. Remove empty compression case here.

Empty string still remains if no compression method is passed. Removed gzip case. This was present since initial commit. Let me check if it is really needed

packages/cache/src/cache.ts

lvpx

Added comments

packages/cache/src/internal/cacheHttpClient.ts

lvpx · 2023-01-02T18:56:22Z

packages/cache/src/internal/cacheHttpClient.ts

+      !compressionMethod || compressionMethod === CompressionMethod.Gzip
+        ? []
+        : [compressionMethod]


So just to understand, if we remove this statement, cache invalidation will happen because the newer versions will include gzip in their version hash and that won't match for older gzip caches. So even in this case, shouldn't gzip caches only be restorable by using gzip. Previously they were identified by not having gzip in version but moving forward will have gzip in version. I think this is what @bishal-pdMSFT is referring to.

packages/cache/src/internal/cacheHttpClient.ts

lvpx

Minor comments

packages/cache/src/internal/cacheHttpClient.ts

lvpx

LGTM 🚀

…-1289-revert-1281-phantsure/compression-changes-release

dscho · 2023-02-07T12:04:42Z

packages/cache/src/internal/tar.ts


 const IS_WINDOWS = process.platform === 'win32'
+exportVariable('MSYS', 'winsymlinks:nativestrict')


This is a grave mistake. This code is supposed to be a library and not leave any remnants in the environment after it ran.

This can cause failures in GitHub workflow runs (and indeed, it did over here). You basically changed the behavior of every workflow run that involves MSYS2 (or for that matter, Git for Windows).

Hi @dscho, I hope this was addressed in this PR?

That looks like it would address the issue, if in any release. I was pretty certain that I was at the current @actions/cache version when I raised the concern, though, so I am fairly confident that no released version fixes this? Am I wrong?

You are correct. We were in the process of getting it merged, will be creating a release to get it live.

eregon · 2023-03-22T20:28:54Z

packages/cache/src/internal/cacheHttpClient.ts

+  // Add compression method to cache version to restore
+  // compressed cache as per compression method
+  if (compressionMethod) {
+    components.push(compressionMethod)


This is a bug, it mutates the user-provided argument paths, fix in #1378

Phantsure changed the base branch from main to releases/cache-v3-beta December 29, 2022 11:18

Phantsure changed the base branch from releases/cache-v3-beta to main December 29, 2022 11:19

Phantsure changed the base branch from main to releases/cache-v3-beta December 29, 2022 11:25

Phantsure changed the base branch from releases/cache-v3-beta to main December 29, 2022 11:25

Phantsure marked this pull request as ready for review December 29, 2022 12:42

Phantsure requested review from a team as code owners December 29, 2022 12:42

Phantsure requested a review from lvpx December 29, 2022 12:42

Phantsure added 4 commits December 29, 2022 12:44

Cache package release for compression change in windows

4e74c1c

This reverts commit 86fe4ab.

Add env variable to enable windows symlinks

2867e31

Add cross os opt-in functionality for cache on windows

ec3a872

Fix test

da162e4

Phantsure force-pushed the revert-1289-revert-1281-phantsure/compression-changes-release branch from 208618d to da162e4 Compare December 29, 2022 12:45

bishal-pdMSFT reviewed Dec 30, 2022

View reviewed changes

packages/cache/package.json Show resolved Hide resolved

bishal-pdMSFT reviewed Dec 30, 2022

View reviewed changes

packages/cache/src/internal/cacheHttpClient.ts Outdated Show resolved Hide resolved

bishal-pdMSFT reviewed Dec 30, 2022

View reviewed changes

packages/cache/src/internal/cacheHttpClient.ts Outdated Show resolved Hide resolved

bishal-pdMSFT reviewed Dec 30, 2022

View reviewed changes

packages/cache/src/cache.ts Outdated Show resolved Hide resolved

lvpx reviewed Jan 2, 2023

View reviewed changes

bishal-pdMSFT reviewed Jan 3, 2023

View reviewed changes

packages/cache/src/internal/cacheHttpClient.ts Outdated Show resolved Hide resolved

Phantsure added 5 commits January 3, 2023 08:31

Address review comments

0746965

Fix test

2f89720

Fix tests

6b01e7a

Fix tests

13b027c

Fix tests

4679d0c

lvpx reviewed Jan 3, 2023

View reviewed changes

packages/cache/src/internal/cacheHttpClient.ts Outdated Show resolved Hide resolved

packages/cache/src/internal/cacheHttpClient.ts Outdated Show resolved Hide resolved

Phantsure added 2 commits January 3, 2023 11:23

Address review comments

59285b8

Address review comments

8e23976

lvpx self-requested a review January 3, 2023 11:51

lvpx approved these changes Jan 3, 2023

View reviewed changes

Phantsure added 5 commits January 3, 2023 11:54

Fix tests

115a496

Fix tests

4c28313

Add npm version

5e943b9

Merge branch 'main' of https://github.com/actions/toolkit into revert…

d898c2d

…-1289-revert-1281-phantsure/compression-changes-release

Add release details

ce85f79

Phantsure merged commit b2d865f into main Jan 4, 2023

dscho reviewed Feb 7, 2023

View reviewed changes

eregon reviewed Mar 22, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cache package release for compression change in windows with symlink fix #1291

Cache package release for compression change in windows with symlink fix #1291

Phantsure commented Dec 29, 2022 •

edited

Loading

bishal-pdMSFT Dec 30, 2022

bishal-pdMSFT Dec 30, 2022

Phantsure Dec 30, 2022

Phantsure Dec 30, 2022

bishal-pdMSFT Dec 30, 2022

Phantsure Dec 30, 2022

lvpx Jan 2, 2023

bishal-pdMSFT Jan 3, 2023

Phantsure Jan 3, 2023

Phantsure Jan 3, 2023

lvpx left a comment

lvpx Jan 2, 2023

lvpx left a comment

lvpx left a comment

dscho Feb 7, 2023

lvpx Feb 8, 2023

dscho Feb 8, 2023

lvpx Feb 8, 2023

dscho Feb 9, 2023

eregon Mar 22, 2023


		const IS_WINDOWS = process.platform === 'win32'
		exportVariable('MSYS', 'winsymlinks:nativestrict')

Cache package release for compression change in windows with symlink fix #1291

Cache package release for compression change in windows with symlink fix #1291

Conversation

Phantsure commented Dec 29, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lvpx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lvpx left a comment

Choose a reason for hiding this comment

lvpx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Phantsure commented Dec 29, 2022 •

edited

Loading