Inability to unzip assets during build on Unix x64 #32805

jaredpar · 2020-02-25T18:16:04Z

Seeing about ~3% of our builds failing with the following error:

Extraction failed for file: /Users/runner/runners/2.165.0/work/1/s/download/libraries_test_assets_OSX_x64_Debug/libraries_test_assets_OSX_x64_Debug.tar.gz

Spot checking the failures these seem to be limited to OSX builds.

Runfo Tracking Issue: Runtime unable to unzip assets

Definition	Build	Kind	Job Name
runtime	1295938	PR 57098	Libraries Test Run release coreclr windows x86 Debug
runtime	1292772	PR 57366	Libraries Test Run checked coreclr OSX x64 Debug
runtime	1270933	PR 56710	Libraries Test Run release coreclr windows x86 Debug
runtime	1256415	PR 54912	Installer Build and Test coreclr windows_arm64 Debug
runtime	1254286	PR 56122	CoreCLR Pri0 Runtime Tests Run windows x64 checked
runtime	1245989	PR 55924	Libraries Test Run checked coreclr Linux arm Release
runtime	1245989	PR 55924	Libraries Test Run checked coreclr Linux_musl arm Release
runtime	1245989	PR 55924	Libraries Test Run checked coreclr Linux_musl x64 Debug
runtime	1245989	PR 55924	Libraries Test Run checked coreclr Linux_musl arm64 Release
runtime	1245989	PR 55924	Libraries Test Run checked coreclr Linux x64 Debug
runtime	1245989	PR 55924	Libraries Test Run checked coreclr Linux x64 Debug
runtime	1245989	PR 55924	Libraries Test Run checked coreclr Linux_musl x64 Debug
runtime	1245989	PR 55924	Libraries Test Run checked coreclr Linux_musl arm64 Release
runtime	1245989	PR 55924	Libraries Test Run checked coreclr Linux arm Release
runtime	1245989	PR 55924	Libraries Test Run checked coreclr Linux_musl arm Release

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
1	2	6

Other tracking issue: https://mseng.visualstudio.com/AzureDevOps/_workitems/edit/1673333

jaredpar · 2020-03-02T21:03:18Z

Issues still occurring, impacting non-Mac builds now too

Build	Kind	Timeline Record
543112	PR #33048	Unzip Test Assets
542475	PR #33035	Unzip Test Assets
542475	PR #33035	Unzip Test Assets
540894	PR #32979	Unzip Test Assets
540894	PR #32979	Unzip Test Assets
540894	PR #32979	Unzip Test Assets
539759	PR #32948	Unzip Test Assets

ViktorHofer · 2020-03-05T19:10:38Z

Just happened again here: https://dev.azure.com/dnceng/public/_build/results?buildId=547852. I believe we started a mail thread about this. cc @safern?

jaredpar · 2020-03-05T23:54:13Z

Failures since March 3rd (since last update)

Build	Kind	Timeline Record
548115	PR #33251	Unzip Test Assets
548115	PR #33251	Unzip Test Assets
547852	PR #33238	Unzip Test Assets
547775	Rolling	Unzip Test Assets
547775	Rolling	Unzip Test Assets
547514	PR #33223	Unzip Test Assets
545323	PR #341	Unzip Test Assets

Evaluated 215 builds
Impacted 5 bulids
Impacted 7 jobs

MattGal · 2020-03-09T16:38:51Z

Just happened again here: https://dev.azure.com/dnceng/public/_build/results?buildId=547852. I believe we started a mail thread about this. cc @safern?

We're tracking this via https://github.com/dotnet/core-eng/issues/9100.
The best thread to follow for this I believe is the IcM ticket which is currently assigned to the artifact team: https://portal.microsofticm.com/imp/v3/incidents/details/177158735/home

If this is hitting often enough, I am willing to bump its priority and sit on a Sev2 IcM bridge; I'm not sure the % hit rate will convince them of this level of importance yet though.

jaredpar · 2020-03-31T05:10:14Z

Closing as the zip disable has fixed this.

ViktorHofer · 2020-07-23T13:56:34Z

Reopening as I'm hitting this in some Windows legs: https://dev.azure.com/dnceng/public/_build/results?buildId=742988&view=logs&j=694d544e-ff71-5faf-b01a-5137c04e57c6&t=38d97292-8f3b-5b9c-e49b-40e136935136

MattGal · 2020-07-23T15:34:40Z

@ViktorHofer this isn't an inability to unzip assets, it's an inability to download them. From your logs:

Downloading artifact libraries_test_assets_Windows_NT_x64_Debug from: https://dev.azure.com/dnceng//_apis/resources/Containers/4666396?itemPath=libraries_test_assets_Windows_NT_x64_Debug&isShallow=true&api-version=4.1-preview.4
Downloading libraries_test_assets_Windows_NT_x64_Debug/libraries_test_assets_Windows_NT_x64_Debug.zip to F:\workspace\_work\1\s\__download__\libraries_test_assets_Windows_NT_x64_Debug\libraries_test_assets_Windows_NT_x64_Debug.zip
Downloaded libraries_test_assets_Windows_NT_x64_Debug/libraries_test_assets_Windows_NT_x64_Debug.zip to F:\workspace\_work\1\s\__download__\libraries_test_assets_Windows_NT_x64_Debug\libraries_test_assets_Windows_NT_x64_Debug.zip
Total Files: 1, Processed: 1, Skipped: 0, Failed: 0, Download time: 1805.634 secs, Download size: 78.441MB

... but if you download that zip, you'll find it's > 380 MB, not 78

Relevant issue: microsoft/azure-pipelines-tasks#13250

ericstj · 2020-07-28T19:40:00Z

@safern previously added a workaround to our pipelines for this. Are we missing that workaround in some places we call DownloadBuildArtifacts?

safern · 2020-07-28T19:52:49Z

I don't think so. This should be disabled by setting a variable on the yml file and we set that variable in a place where all pipelines use:

runtime/eng/pipelines/common/xplat-setup.yml

Lines 29 to 32 in 7242e14

    
           # Workaround for azure devops flakiness when dowloading artifacts 
        
           # https://github.com/dotnet/runtime/issues/32805 
        
           - name: System.DisableZipDownload 
        
             value: true

safern · 2020-07-28T19:53:39Z

@ViktorHofer this isn't an inability to unzip assets, it's an inability to download them. From your logs:

Yeah, but this was the issue tracking this failure and the workaround I provided was to prevent that from happening.

ViktorHofer · 2020-08-04T21:32:18Z

I struggle finding the action item for this issue besides continue ref-counting when the issue happens again. Hence I'm inclined moving this to Future.

jaredpar · 2020-09-29T22:43:05Z

This is still occuring

https://runfo.azurewebsites.net/search/timelines/?bq=repository%3Aruntime+started%3A%7E7&tq=Extraction+failed+for+file

ViktorHofer · 2020-10-01T15:22:28Z

We're tracking this via dotnet/core-eng#9100.

@MattGal the linked issue is closed. From what I understand we implemented a workaround but this started to happen again. How should we proceed here?

MattGal · 2020-10-01T15:42:13Z

We're tracking this via dotnet/core-eng#9100.

@MattGal the linked issue is closed. From what I understand we implemented a workaround but this started to happen again. How should we proceed here?

The task we're discussing isn't one we own, and the linked IcM (https://portal.microsofticm.com/imp/v3/incidents/details/177158735/home) was archived without being mitigated back in May.

I pinged my contact on that team but aside from recreating the same IcM again and being told again they haven't figured out how to fix it, your workaround is probably going to be doing the downloading of artifacts yourself directly via AzDO API calls. I'll update once I hear back.

ViktorHofer · 2020-12-01T18:05:27Z

Just pinged @MattGal about this offline. Even if we might be able to mitigate this by collapsing our legs and depending less on the affected task, we will probably not be able to completely get rid of this in the near future. We might want to create a new IcM for the issue.

MattGal · 2020-12-01T19:07:28Z

At @markwilkie 's suggestion I'll be tracking my efforts to reduce this problem via the linked core-eng issue.

jaredpar · 2020-12-01T19:11:18Z

This is a case where we can safely retry. Pretty much any infra level issue which is identifiable and occurs before tests run is safe to retry. We don't have to wait for core-eng to provide the necessarily logging infra for tests.

MattGal · 2020-12-01T19:14:07Z

This is a case where we can safely retry. Pretty much any infra level issue which is identifiable and occurs before tests run is safe to retry. We don't have to wait for core-eng to provide the necessarily logging infra for tests.

You can safely retry the leg, but the problem is the task "succeeds" when this problem occurs and runfo can't differentiate this from things like an actual malformed archive. It would definitely be preferable (and unfortunately fell through the cracks since July) for the task to actually fail on failure and use its built-in retry mechanisms.

jaredpar · 2020-12-01T19:16:47Z

runfo can't differentiate this from things like an actual malformed archive.

Correct. At the same time we have zero actual malformed archives. So the rate at which this is a false positive is presently zero 😄

Even in the case where we do have a false positive (actual malformed archive) the risk is low here. We retry the job, it fails again and a developer will have to investigate.

MattGal · 2020-12-01T19:27:14Z

@jaredpar makes sense; I'll keep pushing on the actual task getting fixed too.

safern · 2021-01-05T19:27:36Z

@MattGal we just hit this in a rolling build on OSX but I see the dnceng issue is closed. Do we have a way to track AzDo adding retry for Unix code paths?

https://dev.azure.com/dnceng/public/_build/results?buildId=938377&view=logs&j=d8aa34eb-4280-5100-d989-99a78ab22b6c&t=2fa7448e-f849-5fe0-7bed-68068872d0e8&l=18

MattGal · 2021-01-05T19:31:33Z

I have been tracking this since "the surge" via https://github.com/dotnet/core-eng/issues/11551, and AFAIK we're just waiting for this PR to merge; feel free to ping it some more too: microsoft/azure-pipelines-tasks#14065 . I suspect things are just slower than usual because folks are still getting back from holidays.

safern · 2021-01-05T19:43:31Z

Thanks, @MattGal. I didn't know about https://github.com/dotnet/core-eng/issues/11551.

jaredpar added blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' area-Infrastructure labels Feb 25, 2020

Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Feb 25, 2020

This was referenced Mar 2, 2020

Remove unused locals in System.Runtime.WindowsRuntime #33035

Merged

Delete lock accounting #33048

Merged

ViktorHofer mentioned this issue Mar 5, 2020

Unzip Test Assets step in Libraries Test Run fails #32450

Closed

ViktorHofer changed the title ~~Inability to unzip assets during build on OSX 64~~ Inability to unzip assets during build on Unix x64 Mar 5, 2020

jashook mentioned this issue Mar 5, 2020

Infrastructure - Status/Health #702

Closed

safern mentioned this issue Mar 24, 2020

Disable zip download for azure devops download task #34032

Merged

jaredpar closed this as completed Mar 31, 2020

ViktorHofer reopened this Jul 23, 2020

ViktorHofer mentioned this issue Jul 23, 2020

Remove depprojs in favor of PackageReferences #35606

Merged

3 tasks

ericstj removed the untriaged New issue has not been triaged by the area owner label Jul 28, 2020

ericstj added this to the 5.0.0 milestone Jul 28, 2020

ViktorHofer modified the milestones: 5.0.0, Future Aug 4, 2020

safern mentioned this issue Dec 1, 2020

Remove ZipDownloadDisable for azdo artifacts download #45427

Merged

am11 mentioned this issue Jan 10, 2021

Prefer ucontext.h over sys/ucontext.h #46790

Merged

tarekgh mentioned this issue Jan 11, 2021

issue-46477 Fix issue about checking last char in datetime string. #46614

Merged

NewellClark mentioned this issue Jan 15, 2021

Implement struct enumerator for Tensor<T> #46497

Merged

ViktorHofer mentioned this issue Jan 18, 2021

Make changes to reference assembly rerun compile #46999

Merged

NewellClark mentioned this issue Jan 21, 2021

CryptoStream Memory-based ReadAsync/WriteAsync overrides #47207

Merged

This was referenced Feb 14, 2021

Use 'AsSpan' over 'Substring' in 'System.Private.Xml' #48277

Merged

Use 'AsSpan' over 'Substring' in 'System.Private.DataContractSerialization' #48274

Merged

ViktorHofer mentioned this issue Feb 16, 2021

Remove net5 feeds #41620

Merged

2 tasks

ViktorHofer added this to Future in Infrastructure Backlog Feb 17, 2021

ghost moved this from Future to Untriaged in Infrastructure Backlog Feb 18, 2021

ViktorHofer mentioned this issue Feb 21, 2021

Backport removing dead pwd code #48544

Merged

ViktorHofer modified the milestones: Future, 6.0.0 Feb 21, 2021

ghost moved this from Untriaged to 6.0.0 in Infrastructure Backlog Feb 21, 2021

safern mentioned this issue Mar 8, 2021

Enable CheckDownloadedFiles on DownloadBuildArtifacts task #49321

Merged

ghost added the in-pr There is an active PR which will close this issue when it is merged label Mar 8, 2021

safern closed this as completed in #49321 Mar 9, 2021

Infrastructure Backlog automation moved this from 6.0.0 to Done Mar 9, 2021

ghost removed the in-pr There is an active PR which will close this issue when it is merged label Mar 9, 2021

ghost locked as resolved and limited conversation to collaborators May 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Inability to unzip assets during build on Unix x64 #32805

Inability to unzip assets during build on Unix x64 #32805

jaredpar commented Feb 25, 2020 •

edited

Loading

jaredpar commented Mar 2, 2020

ViktorHofer commented Mar 5, 2020

jaredpar commented Mar 5, 2020 •

edited

Loading

MattGal commented Mar 9, 2020

jaredpar commented Mar 31, 2020

ViktorHofer commented Jul 23, 2020

MattGal commented Jul 23, 2020 •

edited

Loading

ericstj commented Jul 28, 2020

safern commented Jul 28, 2020 •

edited

Loading

safern commented Jul 28, 2020

ViktorHofer commented Aug 4, 2020

jaredpar commented Sep 29, 2020

ViktorHofer commented Oct 1, 2020

MattGal commented Oct 1, 2020 •

edited

Loading

ViktorHofer commented Dec 1, 2020

MattGal commented Dec 1, 2020

jaredpar commented Dec 1, 2020

MattGal commented Dec 1, 2020

jaredpar commented Dec 1, 2020

MattGal commented Dec 1, 2020

safern commented Jan 5, 2021

MattGal commented Jan 5, 2021

safern commented Jan 5, 2021

Inability to unzip assets during build on Unix x64 #32805

Inability to unzip assets during build on Unix x64 #32805

Comments

jaredpar commented Feb 25, 2020 • edited Loading

jaredpar commented Mar 2, 2020

ViktorHofer commented Mar 5, 2020

jaredpar commented Mar 5, 2020 • edited Loading

MattGal commented Mar 9, 2020

jaredpar commented Mar 31, 2020

ViktorHofer commented Jul 23, 2020

MattGal commented Jul 23, 2020 • edited Loading

ericstj commented Jul 28, 2020

safern commented Jul 28, 2020 • edited Loading

safern commented Jul 28, 2020

ViktorHofer commented Aug 4, 2020

jaredpar commented Sep 29, 2020

ViktorHofer commented Oct 1, 2020

MattGal commented Oct 1, 2020 • edited Loading

ViktorHofer commented Dec 1, 2020

MattGal commented Dec 1, 2020

jaredpar commented Dec 1, 2020

MattGal commented Dec 1, 2020

jaredpar commented Dec 1, 2020

MattGal commented Dec 1, 2020

safern commented Jan 5, 2021

MattGal commented Jan 5, 2021

safern commented Jan 5, 2021

jaredpar commented Feb 25, 2020 •

edited

Loading

jaredpar commented Mar 5, 2020 •

edited

Loading

MattGal commented Jul 23, 2020 •

edited

Loading

safern commented Jul 28, 2020 •

edited

Loading

MattGal commented Oct 1, 2020 •

edited

Loading