The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1073455	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#79048
1073386	dotnet/arcade	Component Detection (auto-injected by policy)	Log	dotnet/arcade#15916
1073423	dotnet/roslyn	Test	Log	dotnet/roslyn#78976
1073416	dotnet/arcade	Run Helix Tests	Log	dotnet/arcade#15917
1073369	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#79069
1073036	dotnet/runtime	Send tests to Helix (Unix)	Log
1073337	dotnet/runtime	Send job to Helix (Unix)	Log
2733816	dotnet-sdk	🛡 Guardian: BinSkim 4.3.1	Log
1072916	dotnet/runtime	Send tests to Helix (Unix)	Log
1073308	maestro-auth-test/helix-queue-insights-test	Run Tests (Windows)	Log
1073284	maestro-auth-test/helix-queue-insights-test	Run Tests (Windows)	Log	maestro-auth-test/helix-queue-insights-test#1042
1073268	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#79048
1073266	dotnet/roslyn	Run Integration Tests	Log
2733841	dotnet-aspire	🛡 Guardian: AntiMalware Scanner (Binary)	Log
1073253	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#78990
1073245	dotnet/aspire	Run helix tests	Log
1073221	dotnet/runtime	Send to Helix	Log
2733834	dotnet-sdk	🟣 Run ContainerBased Tests (debian12Amd64)	Log
2733833	dotnet-roslyn	Test	Log
1073201	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#49496
1073067	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#79062
2733779	dotnet-dotnet	Checkout dotnet-dotnet@fix-signing-val to s	Log
2733769	dotnet-roslyn	Test	Log
2733747	dotnet-dotnet	Build (centOSStreamContainer)	Log
2733602	dotnet-dotnet	Build (centOSStreamContainer)	Log
2733604	dotnet-runtime	Send job to Helix (Unix)	Log
1072616	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116795
1072170	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#79048
1072111	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#79026
2733265	dotnet-installer	Build	Log
2733263	dotnet-installer	Secure Supply Chain Analysis (auto-injected by policy)	Log	#51059
2733259	dotnet-installer	Build	Log	#51058
2733258	dotnet-installer	Generate delegation SAS Token for dotnetbuilds/internal	Log	#51057
2733257	dotnet-installer	Generate delegation SAS Token for dotnetbuilds/internal	Log	#51056
1072055	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#78833
1071998	dotnet/runtime	Checkout dotnet/runtime@refs/pull/116082/merge to s	Log	dotnet/runtime#116082
1071917	dotnet/runtime	Send to Helix	Log
2733142	dotnet-sdk	Checkout dotnet-sdk@refs/pull/51054/merge to s	Log	#51054
2733056	dotnet-dotnet	Build (azurelinuxArm64MuslCrossContainer)	Log
2733053	dotnet-sdk	Download secrets: HelixProdKV	Log	#51052
2733051	dotnet-sdk	Checkout dotnet-sdk@refs/pull/51052/merge to s	Log	#51052
2733023	dotnet-runtime	Run performance job script	Log
1071748	dotnet/runtime	Send job to Helix (Unix)	Log
1071632	dotnet/runtime	Checkout dotnet/runtime@refs/pull/116791/merge to s	Log	dotnet/runtime#116791
1071502	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116788
2732935	dotnet-dotnet	Build (centOSStreamContainer)	Log
1071200	dotnet/runtime	AOT compile CoreCLR tests	Log
1071202	dotnet/runtime	Send to Helix	Log
1071175	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116779
1071109	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49445
1070592	dotnet/runtime	Send to Helix	Log
2732512	dotnet-performance	Send job to Helix (Unix)	Log
2732477	dotnet-dotnet	Build (centOSStreamContainer)	Log
2732497	dotnet-runtime	Send job to Helix (Unix)	Log
2732294	dotnet-dotnet	🛡 Guardian: CredScan (Binary)	Log
2732452	dotnet-roslyn	Test	Log
2732401	dotnet-sdk	Run AoT Tests in Helix	Log	#51010
2732391	dotnet-sdk	Run Tests in Helix	Log	#51008
2732389	dotnet-sdk	Run AoT Tests in Helix	Log	#51006
2732390	dotnet-sdk	Run AoT Tests in Helix	Log	#51007
1070518	dotnet/performance	Component Detection	Log	dotnet/performance#4877
1070520	dotnet/runtime	AOT compile CoreCLR tests	Log
2732285	dotnet-sdk	🛡 Guardian: CredScan (Binary)	Log	#50939
2732323	dotnet-runtime	Send job to Helix (Unix)	Log
1065484	dotnet/runtime	Send job to Helix (Unix)	Log
2732314	dotnet-sdk	Run Tests in Helix	Log	#51004
2732313	dotnet-sdk	Run AoT Tests in Helix	Log	#51003
2732312	dotnet-sdk	Run Tests in Helix	Log	#51002
2732284	dotnet-sdk	Run TestBuild Tests	Log	#51000
1070456	dotnet/runtime	Send job to Helix (Unix)	Log	dotnet/runtime#116637
2732260	dotnet-aspnetcore	Secure Supply Chain Analysis (auto-injected by policy)	Log	#50991
1070418	dotnet/runtime	Send job to Helix (Unix)	Log
2732227	dotnet-sdk	Run TestBuild Tests	Log	#50989
2732225	dotnet-sdk	Run TestBuild Tests	Log	#50988
2732226	dotnet-sdk	Build	Log	#50988
1070358	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#78988
2732204	dotnet-aspire	Validate Hosted Pool Information (1ES PT)	Log	#50987
2732184	dotnet-dotnet	🛡 Guardian: Publish SDL Logs to Artifacts	Log
2732152	dotnet-runtime	Send job to Helix (Unix)	Log
1070132	dotnet/aspnetcore	Run E2E tests	Log	dotnet/aspnetcore#62366
2732048	dotnet-performance	Send job to Helix (Unix)	Log
2732022	dotnet-runtime	Send job to Helix (Unix)	Log
2732021	dotnet-runtime	Send job to Helix (Unix)	Log
2731896	dotnet-dotnet	Build (centOSStreamContainer)	Log
2731860	dotnet-aspire	Build	Log
2731756	dotnet-runtime	Send job to Helix (Unix)	Log
1069667	dotnet/runtime	Send job to Helix (Unix)	Log
2731687	dotnet-roslyn	Test	Log
2731672	dotnet-sdk	Run TestBuild Tests	Log	#50938
2731668	dotnet-arcade-validation	Run Tests (Windows)	Log
2731655	dotnet-performance	Send job to Helix (Unix)	Log
2731644	dotnet-roslyn	Test	Log
2731620	dotnet-dotnet	Build (fedoraContainer)	Log
2731387	dotnet-dotnet	Validate installer packages	Log
2731603	dotnet-performance	Send job to Helix (Unix)	Log
1069495	dotnet/runtime	Build product	Log	dotnet/runtime#116717
1069434	dotnet/runtime	Checkout dotnet/runtime@refs/pull/115966/merge to s	Log	dotnet/runtime#115966
1069490	dotnet/runtime	Build product	Log	dotnet/runtime#116488
2731580	dotnet-runtime	Send job to Helix (Unix)	Log
1069428	dotnet/roslyn	Test	Log	dotnet/roslyn#78992
Displaying 100 of 1000 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
27	180	1000

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)