Multiple directories and path glob pattern support for pipeline caching #2834

ethanis · 2020-03-03T01:02:59Z

This extends the pipeline caching tasks so that multiple paths can be cached with a single task. The following are supported scenarios:

Single path segment with absolute path outside of Pipeline.Workspace.
Single path segment with path relative to System.DefaultWorkingDirectory.
Multiple path segments, all paths are within Pipeline.Workspace.
Glob patterns with all matches within Pipeline.Workspace

The following are unsupported scenarios:

Glob patterns that match anything outside of Pipeline.Workspace
Multiple path segments with absolute path outside of Pipeline.Workspace

The limitation of having multiple paths being rooted in Pipeline.Workspace is because the tarball needs to be created somewhere, and Pipeline.Workspace seemed like the most build agent agnostic location to anchor on.

There will need to be a PR to the azure-pipelines-tasks repository to update the docs for the task inputs when/if these changes make it in.

Finally, here a some examples:

# cache 'out' and 'dist' directory (iff 'out' and 'dist' exists at root of repository)
- task: Cache@2
  displayName: Cache
  inputs:
    key: 'foo.key'
    path: 'out | dist'
    cacheHitVar: 'OUTPUT_RESTORED'

# cache all node_modules
- task: Cache@2
  displayName: Cache
  inputs:
    key: '$(Agent.OS) | npm | package-lock.json'
    path: '**/node_modules,!**/node_modules/**/node_modules'
    cacheHitVar: 'PACKAGES_RESTORED'

# cache binaries directory and 'out' directory (iff 'out' exists at root of repository)
- task: Cache@2
  displayName: Cache
  inputs:
    key: 'foo.key'
    path: '$(Pipeline.Workspace)/b | out'
    cacheHitVar: 'OUTPUT_RESTORED'

# cache node from tools directory
- task: Cache@2
  displayName: Cache
  inputs:
    key: 'foo.key'
    path: '$(Agent.ToolsDirectory)/node'
    cacheHitVar: 'NODE_RESTORED'

The following will fail because 1 path segment is outside Pipeline.Workspace and 1 is within

# cache node from tools directory
- task: Cache@2
  displayName: Cache
  inputs:
    key: 'foo.key'
    path: '$(Agent.ToolsDirectory)/node | out'

Users/etdenn/minimatch cache path

ethanis · 2020-03-03T01:05:00Z

src/Agent.Plugins/PipelineCache/FingerprintCreator.cs

+                        details = matchedDirectories.Values.ToArray();
+                        resolvedSegments.AddRange(matchedDirectories.Values);
+
+                        // TODO: Is it the right behavior to throw an exception if a path segment isn't resolveable?


I'd love the team's input on what this behavior should be. For any given path segment, if there are no matches, should the tasks fail or simply not include it?

hmm, We should fail imo

ethanis · 2020-03-03T01:06:14Z

src/Agent.Plugins/PipelineCache/PipelineCacheServer.cs

            }
-            return uploadPath;
+            // TODO: what is the right way to handle !ContentFormat.SingleTar


I'm not quite clear on when the ContentFormat will be something other than SingleTar so I could use some guidance on how to handle that case here.

ContentFormat would be 'Files' for older Cache entries.

Are 'Files' input a single item? How do I test backwards compatibility with ContentFormat.Files entries?

So, initially we didn't had tarring, and hence for the cache entries before tarring, we had the input as 'Files'. Your approach LGTM.

ethanis · 2020-03-25T17:29:56Z

@fadnavistanmay should the path input be included in the cache version now? If someone adds a new item to cache, but doesn't change the key the cache entry won't be updated to include the new path. This would also allow for easy salt-ing of cache entries making backwards compat easier.

For reference, GitHub Actions includes the path in a cache's version via this PR

fadnavistanmay · 2020-03-25T22:12:02Z

This PR addresses microsoft/azure-pipelines-tasks#11219

fadnavistanmay · 2020-03-26T00:37:20Z

src/Agent.Plugins/PipelineCache/PipelineCacheServer.cs

            }
            else
            {
                DownloadDedupManifestArtifactOptions options = DownloadDedupManifestArtifactOptions.CreateWithManifestId(
                    manifestId,
-                    targetDirectory,
+                    pathSegments[0], // TODO: is this the right format here


Looks right.

fadnavistanmay · 2020-03-26T17:21:38Z

Thanks for the contribution Ethan. LGTM - @b-barthel - do you mind being the second pair of eyes for this. Thanks

ethanis · 2020-04-06T19:28:51Z

@fadnavistanmay / @b-barthel what are the next steps for this PR? My main concern is still with backwards compatibility with existing caches..

GoMino · 2020-05-02T16:00:07Z

Edit: just realized that this is not released yet 😒
Do you have any idea when this feature will be available?

===============

@ethanis the following doesn't seem to work on Mac agent:

# cache all node_modules
- task: Cache@2
  displayName: Cache
  inputs:
    key: '$(Agent.OS) | npm | package-lock.json'
    path: '**/node_modules,!**/node_modules/**/node_modules'
    cacheHitVar: 'PACKAGES_RESTORED'

Because I am trying the following task in my mono repo (using yarn workspace):

pool:
  vmImage: "macOS 10.14"

variables:
  - name: YARN_CACHED_PATH
    value: '**/node_modules,!**/node_modules/**/node_modules'

 - task: Cache@2
    displayName: Cache Yarn packages
    inputs:
      key: 'yarn | "$(Agent.OS)" | yarn.lock'
      restoreKeys: |
        yarn | "$(Agent.OS)"
        yarn
      path: $(YARN_CACHED_PATH)

result in the following error:

Starting: Cache Yarn packages
==============================================================================
Task         : Cache
Description  : Cache files between runs
Version      : 2.0.0
Author       : Microsoft Corporation
Help         : https://aka.ms/pipeline-caching-docs
==============================================================================
Resolving key:
 - yarn      [string]
 - "Darwin"  [string]
 - yarn.lock [file] --> 8F7DDEB929E5F8FE04BAC62DC69D57AC641BCDCB2E6518905F114FC86D237861
Resolved to: yarn|"Darwin"|qocY/6XZs7snurcY99+jFkYGbc+KutavQiN7/4Owleo=
Information, ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session a0973c5a-71c8-4c16-9ca0-a7bdfeb1a2ac
Information, Getting a pipeline cache artifact with one of the following fingerprints:
Information, Fingerprint: `yarn|"Darwin"|qocY/6XZs7snurcY99+jFkYGbc+KutavQiN7/4Owleo=`
Information, There is a cache miss.
tar: could not chdir to '/Users/runner/runners/2.166.4/work/1/s/**/node_modules,!**/node_modules/**/node_modules'

Information, ApplicationInsightsTelemetrySender correlated 1 events with X-TFS-Session a0973c5a-71c8-4c16-9ca0-a7bdfeb1a2ac
##[error]Process returned non-zero exit code: 1
Finishing: Cache Yarn packages

I have also tried using the env variable AZP_CACHING_CONTENT_FORMAT=Files like:

variables:
  - name: YARN_CACHED_PATH
    value: '**/node_modules,!**/node_modules/**/node_modules'
  - name: AZP_CACHING_CONTENT_FORMAT
    value: Files

but in this case it result in the following error:

Starting: Cache Yarn packages
==============================================================================
Task         : Cache
Description  : Cache files between runs
Version      : 2.0.0
Author       : Microsoft Corporation
Help         : https://aka.ms/pipeline-caching-docs
==============================================================================
Resolving key:
 - yarn      [string]
 - "Darwin"  [string]
 - yarn.lock [file] --> 8F7DDEB929E5F8FE04BAC62DC69D57AC641BCDCB2E6518905F114FC86D237861
Resolved to: yarn|"Darwin"|qocY/6XZs7snurcY99+jFkYGbc+KutavQiN7/4Owleo=
Information, ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session 66febed0-4ed9-47b2-942d-b7a8ffe02d37
Information, Getting a pipeline cache artifact with one of the following fingerprints:
Information, Fingerprint: `yarn|"Darwin"|qocY/6XZs7snurcY99+jFkYGbc+KutavQiN7/4Owleo=`
Information, There is a cache miss.
Information, DedupManifestArtifactClient will correlate http requests with X-TFS-Session 66febed0-4ed9-47b2-942d-b7a8ffe02d37
Information, ApplicationInsightsTelemetrySender correlated 3 events with X-TFS-Session 66febed0-4ed9-47b2-942d-b7a8ffe02d37
##[error]The path provided is invalid.
Finishing: Cache Yarn packages

Do you have any idea how to make it work?

ethanis · 2020-05-02T23:13:25Z

@fadnavistanmay: do you have any rough estimates on timeline for getting this PR completed for @GoMino?

covertbert · 2020-05-18T16:09:58Z

Any further updates on this PR?

aminya · 2020-08-22T06:45:50Z

This seems to be an interesting PR. Does this have any performance benefits compared to using separate tasks?

DeeDeeG · 2020-08-23T13:33:54Z

Hi folks.

I strongly support adding this feature. This is definitely a huge feature missing compared to the unofficial/V1 cache task.

Without a feature like this PR is adding, I prefer the old cache task. Having to cache three different node_modules directories separately, based on all three of their package.json and package-lock.json files, either requires a huge key value repeated N nuber of time (in our case, three times), or can lead to mis-matched node_modules folders in different areas of the project, restored from different states of the repo, if not done carefully.

Our key value could be much more compact with globbing. And restoring all three node_modules dirs based on the same key with no code repetition would be the icing on the cake. I stress that this was all available, and as far as I could tell was the primary use pattern, of the unofficial/V1 cache task.

Would be great to have that again here on the official/V2 task.

Thanks to the author for posting, and to the maintainers for considering this PR.

fadnavistanmay · 2020-09-04T21:46:18Z

@ethanis / @mjroghelia - Let's get this PR in.

DeeDeeG · 2020-10-12T19:34:06Z

Hi again, folks. Any update on this? Thanks.

mjroghelia · 2020-10-12T19:45:43Z

@fadnavistanmay sorry, I missed your earlier approval of this.

@ethanis If you rebase this on master and remove the TODO comments with questions that were addressed in @fadnavistanmay's earlier review I will merge it.

ethanis · 2020-10-16T04:37:13Z

@mjroghelia @fadnavistanmay so excited to get this shipped!

ethanis · 2020-10-16T05:18:52Z

Not sure what's going on with this failing Functional Test - it's coming from a project not touched by this PR..

…ne caching (#2834)" This reverts commit f3466da.

…ne caching (#2834)" (#3163) This reverts commit f3466da.

ethanis added 27 commits February 18, 2020 15:23

WIP allow for multiple cache directories to be specified

551e5b2

wip caching multiple directories in tarball

2b71eb1

Caching multiple directories

e55adbd

Merge pull request #1 from ethanis/users/etdenn/minimatch-cache-path

5b039c1

Users/etdenn/minimatch cache path

Use fingerprint code to constuct list of paths to cache

f216caf

Cleanup path fingerprint logging

8a99bf0

Consolidated fingerprint creation code

022db00

Add unit tests for globbing path fingerprints

d8161bf

Some more path fingerprint testing

8b7943d

Clean up comments

62ff73a

Don't delete tarball for debug purposes

a3f97bd

Rename workspace to workingDirectory

98a4cac

Clean up comments

614376f

Pipe input files to tar from stdin

bd08965

Clean up testing code

c5d7509

Clean up legacy comment

292a1cb

Clean up comments

42a0ec6

Create cache relative to pipeline.workspace

741b67d

Clean up code comments

71ed831

Enforce that cache paths are within Pipeline.Workspace

3981899

Allow for single path segments to be outside of Pipeline.Workspace

2262e71

Allow for single, absolute paths outside of workspace

20f32b0

Add some unit test coverage

9f02f36

Remove legacy comment

4f965de

Merge remote-tracking branch 'upstream/master'

fe57a61

Remove extra whitespace

c289135

Remove debugging code

dc42bbe

ethanis commented Mar 3, 2020

View reviewed changes

Place test paths in isolated directory

b391256

Refactor LogSegment method

3bb578a

fadnavistanmay reviewed Mar 26, 2020

View reviewed changes

fadnavistanmay approved these changes Mar 26, 2020

View reviewed changes

damccorm added the enhancement label Apr 2, 2020

ethanis mentioned this pull request Apr 23, 2020

Issue creating tarball: tar: Failed to clean up compressor microsoft/azure-pipelines-artifact-caching-tasks#35

Closed

DeeDeeG mentioned this pull request Aug 24, 2020

CI: Use Cache@2 task in place of the old SaveCache@1 atom/atom#21057

Merged

Merge microsoft:master into ethanis:master

1f530f6

mjroghelia merged commit f3466da into microsoft:master Oct 19, 2020

mjroghelia mentioned this pull request Oct 20, 2020

Add retries on PA downloads #3142

Merged

pd8 mentioned this pull request Nov 11, 2020

Azure Agent cache not restoring cached files #3159

Closed

mjroghelia added a commit that referenced this pull request Nov 12, 2020

Revert "Multiple directories and path glob pattern support for pipeli…

cffc537

…ne caching (#2834)" This reverts commit f3466da.

mjroghelia mentioned this pull request Nov 12, 2020

Revert "Multiple directories and path glob pattern support for pipeline caching (#2834)" #3163

Merged

mjroghelia added a commit that referenced this pull request Nov 12, 2020

Revert "Multiple directories and path glob pattern support for pipeli…

7b8a536

…ne caching (#2834)" (#3163) This reverts commit f3466da.

mjroghelia added a commit that referenced this pull request Nov 12, 2020

Revert "Multiple directories and path glob pattern support for pipeli…

1c82ef4

…ne caching (#2834)" (#3163) This reverts commit f3466da.

DeeDeeG mentioned this pull request Jan 20, 2021

⬆️ tree-sitter-rust@0.17.0 atom/atom#21790

Merged

johnterickson mentioned this pull request Aug 20, 2023

Support glob for path microsoft/azure-pipelines-tasks#11219

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiple directories and path glob pattern support for pipeline caching #2834

Multiple directories and path glob pattern support for pipeline caching #2834

ethanis commented Mar 3, 2020

ethanis Mar 3, 2020

fadnavistanmay Mar 25, 2020

ethanis Mar 3, 2020

fadnavistanmay Mar 25, 2020

ethanis Mar 25, 2020

fadnavistanmay Mar 26, 2020

ethanis commented Mar 25, 2020

fadnavistanmay commented Mar 25, 2020

fadnavistanmay Mar 26, 2020

fadnavistanmay commented Mar 26, 2020

ethanis commented Apr 6, 2020

GoMino commented May 2, 2020 •

edited

ethanis commented May 2, 2020

covertbert commented May 18, 2020

aminya commented Aug 22, 2020 •

edited

DeeDeeG commented Aug 23, 2020 •

edited

fadnavistanmay commented Sep 4, 2020

DeeDeeG commented Oct 12, 2020

mjroghelia commented Oct 12, 2020

ethanis commented Oct 16, 2020

ethanis commented Oct 16, 2020

Multiple directories and path glob pattern support for pipeline caching #2834

Multiple directories and path glob pattern support for pipeline caching #2834

Conversation

ethanis commented Mar 3, 2020

ethanis Mar 3, 2020

Choose a reason for hiding this comment

fadnavistanmay Mar 25, 2020

Choose a reason for hiding this comment

ethanis Mar 3, 2020

Choose a reason for hiding this comment

fadnavistanmay Mar 25, 2020

Choose a reason for hiding this comment

ethanis Mar 25, 2020

Choose a reason for hiding this comment

fadnavistanmay Mar 26, 2020

Choose a reason for hiding this comment

ethanis commented Mar 25, 2020

fadnavistanmay commented Mar 25, 2020

fadnavistanmay Mar 26, 2020

Choose a reason for hiding this comment

fadnavistanmay commented Mar 26, 2020

ethanis commented Apr 6, 2020

GoMino commented May 2, 2020 • edited

ethanis commented May 2, 2020

covertbert commented May 18, 2020

aminya commented Aug 22, 2020 • edited

DeeDeeG commented Aug 23, 2020 • edited

fadnavistanmay commented Sep 4, 2020

DeeDeeG commented Oct 12, 2020

mjroghelia commented Oct 12, 2020

ethanis commented Oct 16, 2020

ethanis commented Oct 16, 2020

GoMino commented May 2, 2020 •

edited

aminya commented Aug 22, 2020 •

edited

DeeDeeG commented Aug 23, 2020 •

edited