Preserve attributes for files cached with Pipeline Caching

[johnterickson]

Pipeline Caching cache entries currently don't include file attributes (e.g. executable bit, mtime) nor preserve symbolic links.

[willsmythe]

The probable solution here is to tar/zip the contents of the cache prior to upload (and when downloading). In some scenarios (like caches with a large number of small files), this should also help improve cache restore/save performance.

[willsmythe]

Created a separate issue (#10925) to track cache content packing using tar/7z. Although packing is likely a solution for preserving file attributes, we should track these as separate requirements for now.

[wichert]

For what it's worth: this issue is currently resulting in electron builds failing on non-windows hosts if you use a cache. The resulting error is cryptic:

[command]/bin/bash --noprofile --norc /Users/vsts/agent/2.154.3/work/_temp/1e16c5e1-16da-4beb-8cbe-5423ac597cdb.sh
yarn run v1.16.0
$ electron-builder --mac
  • electron-builder version=20.41.0
  • artifacts will be published if draft release exists reason=CI detected
  • loaded configuration file=package.json ("build" field)
  • rebuilding native production dependencies platform=darwin arch=x64
  • rebuilding native dependency name=keytar
  • packaging       platform=darwin arch=x64 electron=5.0.2 appOutDir=dist/mac
Error: spawn /Users/vsts/agent/2.154.3/work/1/s/node_modules/app-builder-bin/mac/app-builder EACCES
    at Process.ChildProcess._handle.onexit (internal/child_process.js:240:19)
    at onErrorNT (internal/child_process.js:415:16)
    at process._tickCallback (internal/process/next_tick.js:63:19)
From previous event:
 …  lot more tracebacks ...

The root cause is node_modules/app-builder-bin/mac/app-builder no longer being executable. A workaround is to manually fix permissions for a cache hit:

  - task: CacheBeta@0
    inputs:
      key: |
        yarn
        $(Agent.OS)
        $(Build.SourcesDirectory)/yarn.lock
      path: $(YARN_CACHE_FOLDER)
      cacheHitVar: CACHE_RESTORED
    displayName: Cache Yarn packages

  - script: |
      yarn --frozen-lockfile
    displayName: Install dependencies

  - script: |
      chmod +x node_modules/app-builder-bin/mac/app-builder
      chmod +x node_modules/7zip-bin/mac/7za
    condition: and(eq(variables.CACHE_RESTORED, 'true'), ne(variables['Agent.OS'], 'Windows_NT'))
    displayName: Correct file permissions

[willsmythe]

@wichert - see my comment regarding a simple, temporary way to tar/untar the contents of the cache.

[johnterickson]

Here's a targeted workaround:

https://github.com/johnterickson/cargo2junit/blob/41d3dcec5009116242ba66e6b531f22a66dd1cc2/job.yml#L51-L56

    - task: CacheBeta@0
      inputs:
        cacheHitVar: CACHE_RESTORED
        key: ...
        path: $(CACHE_ROOT)
    - bash: |
        echo REMOVE ME ONCE THIS IS FIXED https://github.com/microsoft/azure-pipelines-tasks/issues/10841
        cd "$(CACHE_ROOT)"
        if [ -f "executable.txt" ]; then chmod +x $(cat executable.txt); fi
      condition: and(succeeded(), ne(variables['CACHE_RESTORED'], 'false'))
...
    - bash: |
        echo REMOVE ME ONCE THIS IS FIXED https://github.com/microsoft/azure-pipelines-tasks/issues/10841
        cd "$(CACHE_ROOT)"
        rm executable.txt
        find . -perm +111 >> executable.txt
        find . -executable >> executable.txt
        cat executable.txt
      displayName: Save list of files with executable bit set

[johnterickson]

Rolling out via #10925