Work item tests crash on OSX x64 Debug #87287

eerhardt · 2023-06-08T19:25:21Z

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=300158
Build error leg or test failing: System.IO.FileSystem.Watcher.Tests.WorkItemExecution
Pull request: #86974

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "END OF WORK ITEM LOG: Command exited with 127",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Build	Definition	Test	Pull Request
475355	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
474849	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
474532	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
473727	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
473597	dotnet/runtime	System.IO.FileSystem.Manual.Tests.WorkItemExecution
473431	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
473397	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
472898	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
472852	dotnet/runtime	System.IO.Compression.Tests.WorkItemExecution
472282	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
471895	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
471740	dotnet/runtime	System.IO.Compression.Tests.WorkItemExecution
471580	dotnet/runtime	System.IO.FileSystem.Manual.Tests.WorkItemExecution
471389	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
470637	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
470321	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
469012	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
468964	dotnet/runtime	System.IO.FileSystem.Primitives.Tests.WorkItemExecution
468841	dotnet/runtime	System.IO.FileSystem.Manual.Tests.WorkItemExecution
468587	dotnet/runtime	System.IO.FileSystem.Primitives.Tests.WorkItemExecution
468089	dotnet/runtime	System.IO.FileSystem.Primitives.Tests.WorkItemExecution
467619	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
467146	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
466691	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
465768	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
465695	dotnet/runtime	System.IO.FileSystem.Primitives.Tests.WorkItemExecution
465642	dotnet/runtime	XUnitLogChecker.WorkItemExecution	#93906
465587	dotnet/runtime	XUnitLogChecker.WorkItemExecution	#93906
465447	dotnet/runtime	XUnitLogChecker.WorkItemExecution	#93906
465319	dotnet/runtime	System.IO.FileSystem.Primitives.Tests.WorkItemExecution
465327	dotnet/runtime	System.IO.FileSystem.Primitives.Tests.WorkItemExecution
464592	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
463142	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
462608	dotnet/runtime	System.IO.FileSystem.Manual.Tests.WorkItemExecution
461771	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
460596	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
460238	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
459941	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
459223	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
458289	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
457386	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
456786	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
456243	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
456245	dotnet/runtime	System.IO.FileSystem.Manual.Tests.WorkItemExecution
455009	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
454543	dotnet/runtime	System.IO.FileSystem.Manual.Tests.WorkItemExecution
454533	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
454134	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
452981	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
451747	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
450630	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
449186	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
447511	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
446965	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	21	54

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323604
Error message validated: Work item System.IO.FileSystem.Watcher.Tests in job
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:26:54 PM UTC

The text was updated successfully, but these errors were encountered:

ghost · 2023-06-08T19:25:32Z

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=300158
Build error leg or test failing: System.IO.FileSystem.Watcher.Tests.WorkItemExecution
Pull request: #86974

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Work item System.IO.FileSystem.Watcher.Tests in job",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Author:	eerhardt
Assignees:	-
Labels:	`area-Infrastructure-libraries`, `blocking-clean-ci`, `untriaged`, `Known Build Error`
Milestone:	-

ghost · 2023-06-08T19:44:07Z

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=300158
Build error leg or test failing: System.IO.FileSystem.Watcher.Tests.WorkItemExecution
Pull request: #86974

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Work item System.IO.FileSystem.Watcher.Tests in job",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Author:	eerhardt
Assignees:	-
Labels:	`area-System.IO`, `blocking-clean-ci`, `untriaged`, `Known Build Error`
Milestone:	-

Jozkee · 2023-07-21T21:56:36Z

The logs not only show FileSystem.Watcher.Tests. I think this is a broader issue.

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/334444/logs/123

e.g:

2023-07-11T00:12:22.4993970Z /Users/runner/work/1/s/.packages/microsoft.dotnet.helix.sdk/7.0.0-beta.23313.4/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(90,5): error : Work item Common.Tests in job 9199670f-aa4c-4e8e-8150-b650ad0bb59f has failed. [/Users/runner/work/1/s/src/libraries/sendtohelixhelp.proj]
2023-07-11T00:12:22.4995970Z /Users/runner/work/1/s/.packages/microsoft.dotnet.helix.sdk/7.0.0-beta.23313.4/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(90,5): error : Failure log: https://helix.dot.net/api/2019-06-17/jobs/9199670f-aa4c-4e8e-8150-b650ad0bb59f/workitems/Common.Tests/console [/Users/runner/work/1/s/src/libraries/sendtohelixhelp.proj]
2023-07-11T00:12:22.5108980Z ##[error].packages/microsoft.dotnet.helix.sdk/7.0.0-beta.23313.4/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(90,5): error : (NETCORE_ENGINEERING_TELEMETRY=Test) Work item Common.Tests in job 9199670f-aa4c-4e8e-8150-b650ad0bb59f has failed.
Failure log: https://helix.dot.net/api/2019-06-17/jobs/9199670f-aa4c-4e8e-8150-b650ad0bb59f/workitems/Common.Tests/console
2023-07-11T00:12:22.5122420Z /Users/runner/work/1/s/.packages/microsoft.dotnet.helix.sdk/7.0.0-beta.23313.4/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(90,5): error : Work item IcuAppLocal.Tests in job 9199670f-aa4c-4e8e-8150-b650ad0bb59f has failed. [/Users/runner/work/1/s/src/libraries/sendtohelixhelp.proj]
2023-07-11T00:12:22.5126220Z /Users/runner/work/1/s/.packages/microsoft.dotnet.helix.sdk/7.0.0-beta.23313.4/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(90,5): error : Failure log: https://helix.dot.net/api/2019-06-17/jobs/9199670f-aa4c-4e8e-8150-b650ad0bb59f/workitems/IcuAppLocal.Tests/console [/Users/runner/work/1/s/src/libraries/sendtohelixhelp.proj]

ghost · 2023-07-21T21:57:09Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=300158
Build error leg or test failing: System.IO.FileSystem.Watcher.Tests.WorkItemExecution
Pull request: #86974

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Work item System.IO.FileSystem.Watcher.Tests in job",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Build	Definition	Step Name	Console log	Pull Request
347502	dotnet/runtime	Send to Helix	Log
347514	dotnet/runtime	Send to Helix	Log
347516	dotnet/runtime	Send to Helix	Log
347506	dotnet/runtime	Send to Helix	Log
347504	dotnet/runtime	Send to Helix	Log
347527	dotnet/runtime	Send to Helix	Log
346179	dotnet/runtime	Send to Helix	Log
346184	dotnet/runtime	Send to Helix	Log
346183	dotnet/runtime	Send to Helix	Log
346189	dotnet/runtime	Send to Helix	Log
346180	dotnet/runtime	Send to Helix	Log
346188	dotnet/runtime	Send to Helix	Log
345385	dotnet/runtime	Send to Helix	Log	#89145
344799	dotnet/runtime	Send to Helix	Log
344802	dotnet/runtime	Send to Helix	Log
344773	dotnet/runtime	Send to Helix	Log
344775	dotnet/runtime	Send to Helix	Log
344801	dotnet/runtime	Send to Helix	Log
344791	dotnet/runtime	Send to Helix	Log
344313	dotnet/runtime	Send to Helix	Log	#89145
344014	dotnet/runtime	Send to Helix	Log	#89061
343534	dotnet/runtime	Send to Helix	Log
343456	dotnet/runtime	Send to Helix	Log	#89074
343307	dotnet/runtime	Send to Helix	Log
343300	dotnet/runtime	Send to Helix	Log
343299	dotnet/runtime	Send to Helix	Log
343301	dotnet/runtime	Send to Helix	Log
343298	dotnet/runtime	Send to Helix	Log
343296	dotnet/runtime	Send to Helix	Log
343319	dotnet/runtime	Send to Helix	Log	#89074
342971	dotnet/runtime	Send to Helix	Log
342902	dotnet/runtime	Send to Helix	Log	#87847
342704	dotnet/runtime	Send to Helix	Log	#88974
342426	dotnet/runtime	Send to Helix	Log
342101	dotnet/runtime	Send to Helix	Log
341764	dotnet/runtime	Send to Helix	Log
341747	dotnet/runtime	Send to Helix	Log
341763	dotnet/runtime	Send to Helix	Log
341766	dotnet/runtime	Send to Helix	Log
341752	dotnet/runtime	Send to Helix	Log
341745	dotnet/runtime	Send to Helix	Log
341422	dotnet/runtime	Send to Helix	Log	#88502
341117	dotnet/runtime	Send to Helix	Log
341130	dotnet/runtime	Send to Helix	Log
341115	dotnet/runtime	Send to Helix	Log
341116	dotnet/runtime	Send to Helix	Log
341137	dotnet/runtime	Send to Helix	Log
341114	dotnet/runtime	Send to Helix	Log
340730	dotnet/runtime	Send to Helix	Log
340706	dotnet/runtime	Send to Helix	Log
340711	dotnet/runtime	Send to Helix	Log
340713	dotnet/runtime	Send to Helix	Log
340722	dotnet/runtime	Send to Helix	Log
340731	dotnet/runtime	Send to Helix	Log
340606	dotnet/runtime	Send to Helix	Log	#74623
339887	dotnet/runtime	Send to Helix	Log
339546	dotnet/runtime	Send to Helix	Log
339567	dotnet/runtime	Send to Helix	Log
339550	dotnet/runtime	Send to Helix	Log
339557	dotnet/runtime	Send to Helix	Log
339565	dotnet/runtime	Send to Helix	Log
339554	dotnet/runtime	Send to Helix	Log
339453	dotnet/runtime	Send to Helix	Log	#88892
339194	dotnet/runtime	Send to Helix	Log	#74623
339190	dotnet/runtime	Send to Helix	Log
338975	dotnet/runtime	Send to Helix	Log
338958	dotnet/runtime	Send to Helix	Log
338949	dotnet/runtime	Send to Helix	Log
338278	dotnet/runtime	Send to Helix	Log
338279	dotnet/runtime	Send to Helix	Log
338284	dotnet/runtime	Send to Helix	Log
338258	dotnet/runtime	Send to Helix	Log
338277	dotnet/runtime	Send to Helix	Log
338291	dotnet/runtime	Send to Helix	Log
338281	dotnet/runtime	Send to Helix	Log
337928	dotnet/runtime	Send to Helix	Log
337971	dotnet/runtime	Send to Helix	Log
336905	dotnet/runtime	Send to Helix	Log
336900	dotnet/runtime	Send to Helix	Log
336626	dotnet/runtime	Send to Helix	Log
336635	dotnet/runtime	Send to Helix	Log
336627	dotnet/runtime	Send to Helix	Log
336647	dotnet/runtime	Send to Helix	Log
336638	dotnet/runtime	Send to Helix	Log
336646	dotnet/runtime	Send to Helix	Log
336353	dotnet/runtime	Send to Helix	Log
336347	dotnet/runtime	Send to Helix	Log
335895	dotnet/runtime	Send to Helix	Log	#88626
335021	dotnet/runtime	Send to Helix	Log
335024	dotnet/runtime	Send to Helix	Log
335003	dotnet/runtime	Send to Helix	Log
335004	dotnet/runtime	Send to Helix	Log
334953	dotnet/runtime	Send to Helix	Log
335012	dotnet/runtime	Send to Helix	Log
335002	dotnet/runtime	Send to Helix	Log
334959	dotnet/runtime	Send to Helix	Log
334772	dotnet/runtime	Send to Helix	Log
334717	dotnet/runtime	Send to Helix	Log
334699	dotnet/runtime	Send to Helix	Log
334444	dotnet/runtime	Send to Helix	Log
Displaying 100 of 262 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
6	55	262

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323604
Error message validated: Work item System.IO.FileSystem.Watcher.Tests in job
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:26:54 PM UTC

Author:	eerhardt
Assignees:	-
Labels:	`blocking-clean-ci`, `area-Infrastructure`, `untriaged`, `Known Build Error`
Milestone:	-

carlossanlop · 2023-07-21T22:19:56Z

Yea, doesn't seem to be a FileSystemWatcher-only issue. The jobs are getting unexpectedly killed, according to the console log outputs. Example:

Log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-release-70-staging-9199670faa4c4e8e81/Microsoft.Bcl.AsyncInterfaces.Tests/1/console.7114ae01.log?helixlogtype=result
Output:

2023-07-11T00:05:26.562Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2023-07-11T00:05:26.563Z	INFO   	run.py	__init__(42)	read_results	Searching '/private/tmp/helix/working/A8D4097B/w/9393087D/e' for test results files
2023-07-11T00:05:26.563Z	INFO   	run.py	__init__(42)	read_results	Searching '/tmp/helix/working/A8D4097B/w/9393087D/uploads' for test results files
2023-07-11T00:05:26.563Z	WARNING	run.py	__init__(55)	read_results	No results file found in any of the following formats: xunit, junit, trx
2023-07-11T00:05:26.563Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to '/tmp/helix/working/A8D4097B/w/9393087D/e/__test_report.json'
2023-07-11T00:05:26.564Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1523 bytes
+ exit 127
['Microsoft.Bcl.AsyncInterfaces.Tests' END OF WORK ITEM LOG: Command exited with 127]

akoeplinger · 2023-11-20T13:15:47Z

A couple observations from looking through a couple results:

Only happens on the 7.0 branches
Only happens on runtime-libraries-coreclr outerloop pipelines, across all OSes
The test doesn't get killed, instead it fails because dotnet couldn't be found:

./RunTests.sh: line 168: /tmp/helix/working/B2D409A4/p/dotnet: No such file or directory

I downloaded the test-runtime-net7.0-OSX-Release-arm64.zip correlation payload and it is indeed missing the dotnet binary and host, it only contains the shared framework.

akoeplinger · 2023-11-20T14:08:36Z

The 7.0 branch is missing this fix: #79652

Backport of #79652 to release/7.0 Fixes #87287

Fixes #87287

akoeplinger · 2023-11-20T18:27:01Z

Fixed by #95011

eerhardt added blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab labels Jun 8, 2023

ghost added the untriaged New issue has not been triaged by the area owner label Jun 8, 2023

dotnet-issue-labeler bot added the area-Infrastructure-libraries label Jun 8, 2023

eerhardt mentioned this issue Jun 8, 2023

Add IHostApplicationBuilder interface #86974

Merged

eerhardt added area-System.IO and removed area-Infrastructure-libraries labels Jun 8, 2023

Jozkee added area-Infrastructure and removed area-System.IO labels Jul 21, 2023

carlossanlop added Known Build Error Use this to report build issues in the .NET Helix tab and removed Known Build Error Use this to report build issues in the .NET Helix tab labels Jul 21, 2023

Jozkee changed the title ~~System.IO.FileSystem.Watcher.Tests crash on osx x64 Debug~~ Work item tests crash on OSX x64 Debug Jul 22, 2023

akoeplinger mentioned this issue Nov 20, 2023

Fix libraries outerloop pipelines #79652

Merged

akoeplinger mentioned this issue Nov 20, 2023

[release/7.0] Fix libraries outerloop pipelines #95011

Merged

ghost added the in-pr There is an active PR which will close this issue when it is merged label Nov 20, 2023

akoeplinger added a commit that referenced this issue Nov 20, 2023

[release/7.0] Fix libraries outerloop pipelines

e412f70

Backport of #79652 to release/7.0 Fixes #87287

akoeplinger added a commit that referenced this issue Nov 20, 2023

[release/7.0] Fix libraries outerloop pipelines (#95011)

081f4ff

Fixes #87287

ghost removed the in-pr There is an active PR which will close this issue when it is merged label Nov 20, 2023

akoeplinger closed this as completed Nov 20, 2023

ghost removed the untriaged New issue has not been triaged by the area owner label Nov 20, 2023

github-actions bot locked and limited conversation to collaborators Dec 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Work item tests crash on OSX x64 Debug #87287

Work item tests crash on OSX x64 Debug #87287

eerhardt commented Jun 8, 2023 •

edited by build-analysis bot

ghost commented Jun 8, 2023

Build Information

Error Message

ghost commented Jun 8, 2023

Build Information

Error Message

Jozkee commented Jul 21, 2023

ghost commented Jul 21, 2023

Build Information

Error Message

Report

Summary

Known issue validation

carlossanlop commented Jul 21, 2023 •

edited

akoeplinger commented Nov 20, 2023 •

edited

akoeplinger commented Nov 20, 2023

akoeplinger commented Nov 20, 2023

Work item tests crash on OSX x64 Debug #87287

Work item tests crash on OSX x64 Debug #87287

Comments

eerhardt commented Jun 8, 2023 • edited by build-analysis bot

Build Information

Error Message

Report

Summary

Known issue validation

ghost commented Jun 8, 2023

Build Information

Error Message

ghost commented Jun 8, 2023

Build Information

Error Message

Jozkee commented Jul 21, 2023

ghost commented Jul 21, 2023

Build Information

Error Message

Report

Summary

Known issue validation

carlossanlop commented Jul 21, 2023 • edited

akoeplinger commented Nov 20, 2023 • edited

akoeplinger commented Nov 20, 2023

akoeplinger commented Nov 20, 2023

eerhardt commented Jun 8, 2023 •

edited by build-analysis bot

carlossanlop commented Jul 21, 2023 •

edited

akoeplinger commented Nov 20, 2023 •

edited