Disable core dumps in host tests that intentionally crash

[elinor-fung]

For tests that intentionally crash, we were already clearing out DOTNET_DbgEnableMiniDump/COMPlus_DbgEnableMiniDump, but OS core dumps were still enabled.

• In tests that are explicitly throwing an unhandled exception, call into OS APIs to disable dumps for the current process
• When launching a command that should have dumps disabled, call ulimit -c 0 before the command (for cases where the test doesn't have control over the process)

In this PR build, we no longer have dump files:
linux-x64
osx-x64

Before - six core dumps created / uploaded:
linux-x64
osx-x64

On Windows, we still have two created/uploaded for Muxer_NonAssemblyWithExeExtension and UnhandledException_BreadcrumbThreadDoesNotFinish. I couldn't find a way to disable dump creation just for the process we were launching.

Fixes #116520

cc @dotnet/appmodel @AaronRobinsonMSFT

[dotnet-policy-service]

Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull Request Overview

Disables OS core dumps for tests that intentionally crash by adding a utility to turn off dumps in-process and updating how test commands are launched.

• Introduce Utilities.CoreDump.Disable() for Linux/macOS interop disabling of core dumps.
• Extend Command with .DisableDumps() to wrap process launches with ulimit -c 0 on *nix.
• Update all crash-intent tests to call .DisableDumps() and remove the old expectedToFail parameter.
• Include CoreDump.cs in all relevant test asset projects.

Reviewed Changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/installer/tests/TestUtils/Command.cs	Added _disableDumps flag, DisableDumps() method, and wrap in bash -c "ulimit -c 0 ..." when launching.
src/installer/tests/HostActivation.Tests/StartupHooks.cs	Replaced .Execute(expectedToFail: true) with .DisableDumps().Execute().
src/installer/tests/HostActivation.Tests/SelfContainedAppLaunch.cs	Removed expectedToFail, now just .Execute().
src/installer/tests/HostActivation.Tests/NativeHosting/LoadAssemblyAndGetFunctionPointer.cs	Added .DisableDumps() before .Execute().
src/installer/tests/HostActivation.Tests/NativeHosting/GetFunctionPointer.cs	Added .DisableDumps() before .Execute().
src/installer/tests/HostActivation.Tests/NativeHosting/ApplicationExecution.cs	Added .DisableDumps() before .Execute().
src/installer/tests/HostActivation.Tests/InvalidHost.cs	Removed expectedToFail param and updated .Execute().
src/installer/tests/HostActivation.Tests/FrameworkDependentAppLaunch.cs	Removed expectedToFail, updated .Execute().
src/installer/tests/HostActivation.Tests/DotnetArgValidation.cs	Removed expectedToFail and updated .Execute().
src/installer/tests/HostActivation.Tests/DependencyResolution/AdditionalDeps.cs	Removed expectedToFail usage and updated .Execute().
src/installer/tests/HostActivation.Tests/Breadcrumbs.cs	Added .DisableDumps() before .Execute().
src/installer/tests/Assets/Projects/HelloWorld/SelfContained.csproj	Include CoreDump.cs compile item.
src/installer/tests/Assets/Projects/HelloWorld/Program.cs	Call Utilities.CoreDump.Disable() in the crash branch.
src/installer/tests/Assets/Projects/HelloWorld/HelloWorld.csproj	Include CoreDump.cs compile item.
src/installer/tests/Assets/Projects/CoreDump.cs	New interop utility to disable core dumps on Linux/macOS.
src/installer/tests/Assets/Projects/Component/Component.csproj	Include CoreDump.cs compile item.
src/installer/tests/Assets/Projects/Component/Component.cs	Call Utilities.CoreDump.Disable() before throwing.
src/installer/tests/Assets/Projects/AppWithCustomEntryPoints/Program.cs	Call Utilities.CoreDump.Disable() before throwing.
src/installer/tests/Assets/Projects/AppWithCustomEntryPoints/AppWithCustomEntryPoints.csproj	Include CoreDump.cs compile item.

Comments suppressed due to low confidence (1)

src/installer/tests/TestUtils/Command.cs:137

• Add an XML doc comment for DisableDumps() to explain that it clears dump‐creation env vars and wraps the process with ulimit -c 0 on Linux/macOS.

        public Command DisableDumps()

[AaronRobinsonMSFT]

I couldn't find a way to disable dump creation just for the process we were launching.

Setting WER dump count to 0?

[AaronRobinsonMSFT]

I couldn't find a way to disable dump creation just for the process we were launching.

Setting WER dump count to 0?

Oh, you meant as in the specific instance rather than all instances of the exe, right?

[elinor-fung]

I couldn't find a way to disable dump creation just for the process we were launching.

Setting WER dump count to 0?

Oh, you meant as in the specific instance rather than all instances of the exe, right?

Yeah, specific instance. I want to leave dumps enabled for any other crashes (the intentionally crashing process is dotnet in these cases).

[MichalStrehovsky]

Yeah, specific instance. I want to leave dumps enabled for any other crashes (the intentionally crashing process is dotnet in these cases).

I think this will achieve that:

runtime/src/tests/nativeaot/SmokeTests/ControlFlowGuard/ControlFlowGuard.cs

Lines 67 to 74 in 96e283d

	[DllImport("kernel32", ExactSpelling = true)]
	static extern uint GetErrorMode();
	[DllImport("kernel32", ExactSpelling = true)]
	static extern uint SetErrorMode(uint uMode);
	// Don't pop the WER dialog box that blocks the process until someone clicks Close.
	SetErrorMode(GetErrorMode() | 0x0002 /* NOGPFAULTERRORBOX */);

The NOGPFAULTERRORBOX disables WER for the process.

[elinor-fung]

Thanks - I'll try out NOGPFAULTERRORBOX separately. Going to get this in first, since the dumps are killing CI on Linux (but not Windows)

[elinor-fung]

/ba-g failure is #116647