Fix EPERM when killing sudo-launched createdump process in CollectCrashDumpWithCreateDump

[Copilot]

Tests on macOS were failing with AggregateException: Not all processes in process tree could be terminated (Operation not permitted) when a timed-out createdump process could not be killed.

Description

Root cause: createdump is launched via sudo (createdump.StartInfo.FileName = "sudo"), so the sudo process and its child createdump binary run as root. On macOS/Linux, a non-root process cannot send SIGKILL to root-owned processes. This is exactly what Process.Kill(entireProcessTree: true) was hitting in the timeout path, causing an AggregateException to propagate and fail the test.

A previous workaround silently retried the failing Kill call up to 5 times, masking the error. This PR removes that workaround and fixes the underlying cause.

Fix: Added a KillWithSudo helper that kills the timed-out createdump process using sudo -n kill -9 <pid>. Since passwordless sudo access is already required to launch createdump in the first place, this approach works correctly.

The helper includes the following design decisions:

• Non-interactive sudo: sudo -n is used so that sudo fails fast (with a non-zero exit code) instead of hanging waiting for a password prompt if passwordless sudo is not configured.
• No output redirection: stdout and stderr from the sudo kill process are not redirected — output flows directly to the console. Since this method is only called on a failure path, verbose output is acceptable and desirable.
• No timeout on sudoKill: WaitForExit() is called without a timeout. If sudo kill itself hangs, we are already in a cascade of failures with no meaningful recovery path, so no attempt is made to kill sudoKill.
• Exception propagation: exceptions from KillWithSudo are allowed to propagate so that failures are visible rather than silently swallowed.

Before (workaround):

else
{
    // Workaround for https://github.com/dotnet/runtime/issues/93321
    const int MaxRetries = 5;
    for (int i = 0; i < MaxRetries; i++)
    {
        try
        {
            createdump.Kill(entireProcessTree: true);
            break;
        }
        catch (Exception e) when (i < MaxRetries - 1)
        {
            Console.WriteLine($"Process.Kill(entireProcessTree: true) failed:");
            Console.WriteLine(e);
            Console.WriteLine("Retrying...");
        }
    }
}

After (fix):

else
{
    // createdump was launched via 'sudo', so the process runs as root.
    // We cannot send SIGKILL to a root-owned process from a non-root process.
    // Use 'sudo kill' to terminate the timed-out process.
    KillWithSudo(createdump);
}

Where KillWithSudo kills the given process using Process.Start("sudo", $"-n kill -9 {pid}")?.WaitForExit(), with output flowing directly to the console and exception propagation.

Changes

• [src/tests/Common/CoreCLRTestLibrary/CoreclrTestWrapperLib.cs]: Remove the retry workaround; add KillWithSudo helper method using the Process.Start(fileName, arguments) convenience overload, sudo -n, no stdout/stderr redirection (output goes directly to the console — already on a failure path where verbose output is fine), unconditional WaitForExit() (no timeout — if sudo kill hangs we are already in a cascade of failures with no recovery path), exception propagation (no try/catch blocks); call it in the timeout path instead of Kill(entireProcessTree: true); remove all unnecessary UseShellExecute = false assignments throughout the file (the default in .NET Core)

Note

This PR was generated with AI assistance (GitHub Copilot).

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Test failure: AggregateException: Not all processes in process tree could be terminated.</issue_title>
<issue_description>Failed in: runtime-coreclr jitstress 20231009.2

Failed tests:

coreclr osx x64 Checked jitstress2 @ OSX.1200.Amd64.Open
    - baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh
    - baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh
    - baseservices/exceptions/unhandled/unhandled/unhandled.sh

Error message:

Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree)
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at baseservices_exceptions._simple_ParallelCraMainThread_ParallelCraMainThread_._simple_ParallelCrashMainThread_ParallelCrashMainThread_sh()
Expected: True
Actual:   False

Stack trace:

   at baseservices_exceptions._simple_ParallelCraMainThread_ParallelCraMainThread_._simple_ParallelCrashMainThread_ParallelCrashMainThread_sh()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

Queued	Pipeline	Pipeline_Configuration	OS	Arch	Test	Outcome	Properties	BuildReason
2023-10-10T05:50:17.995Z	runtime-coreclr jitstress	Checked-jitstress2	osx.1200.amd64.open	x64	baseservices/exceptions/unhandled/unhandled/unhandled.sh	Failed	{"BuildNumber":"20231009.2","DefinitionId":"109","Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","operatingSystem":"OSX.1200.Amd64.Open","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","System.StageName":"Build","BuildId":"432736","architecture":"x64","configuration":"Checked-jitstress2","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"9585790","System.JobAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.StageAttempt":"1","Reason":"Schedule"}	Schedule
2023-10-10T05:50:17.995Z	runtime-coreclr jitstress	Checked-jitstress2	osx.1200.amd64.open	x64	baseservices/exceptions/unhandled/unhandled/unhandled.sh	Failed	{"BuildNumber":"20231009.2","DefinitionId":"109","Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","operatingSystem":"OSX.1200.Amd64.Open","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","System.StageName":"Build","BuildId":"432736","architecture":"x64","configuration":"Checked-jitstress2","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"9585790","System.JobAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.StageAttempt":"1","Reason":"Schedule"}	Schedule
2023-10-10T05:50:17.995Z	runtime-coreclr jitstress	Checked-jitstress2	osx.1200.amd64.open	x64	baseservices/exceptions/unhandled/unhandled/unhandled.sh	Failed	{"BuildNumber":"20231009.2","DefinitionId":"109","Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","operatingSystem":"OSX.1200.Amd64.Open","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","System.StageName":"Build","BuildId":"432736","architecture":"x64","configuration":"Checked-jitstress2","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"9585790","System.JobAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.StageAttempt":"1","Reason":"Schedule"}	Schedule
2023-10-10T05:50:17.995Z	runtime-coreclr jitstress	Checked-jitstress2	osx.1200.amd64.open	x64	baseservices/exceptions/unhandled/unhandled/unhandled.sh	Failed	{"BuildNumber":"20231009.2","DefinitionId":"109","Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","operatingSystem":"OSX.1200.Amd64.Open","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","System.StageN...

• Fixes Test failure: AggregateException: Not all processes in process tree could be terminated. #93321

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[danmoseley]

@copilot I don't want a PR to just remove the workaorund - it will just expose the problem again. I want you to remove the workaround locally, loop to try to get a repro. If you get a repro, investigate. Any fix, put in this PR.

[Copilot]

@copilot I don't want a PR to just remove the workaorund - it will just expose the problem again. I want you to remove the workaround locally, loop to try to get a repro. If you get ...

The root cause is that createdump is launched via sudo, so both sudo and its child createdump run as root. On macOS (and Linux), a non-root process cannot send SIGKILL to root-owned processes, which is why Process.Kill(entireProcessTree: true) was failing with EPERM.

The fix (commit a1b48fe):

• Adds a KillProcessTreeWithSudo helper that recursively enumerates children and kills each one using sudo kill -9 <pid> — since we already have passwordless sudo access (otherwise createdump wouldn't have launched)
• Replaces the now-removed retry workaround with a call to KillProcessTreeWithSudo(createdump) in the timeout (else) branch

[danmoseley]

@copilot look at feedback

[Copilot]

@copilot look at feedback

Addressed all feedback in 85f204a: added exception logging, reduced WaitForExit timeout from 30s → 5s, and added a comment about the race condition.

[am11]

In the if block we have:

                Task.WaitAll(copyError, copyOutput);
                Console.WriteLine("chown stdout:");
                Console.WriteLine(copyOutput.Result);
                Console.WriteLine("chown stderr:");
                Console.WriteLine(copyError.Result);

The else block can also use the same pattern for kill command and inline it like the if block does.

[jkotas]

@copilot #125916 (comment)

[Copilot]

@copilot #125916 (comment)

Fixed in 84153ab — changed sudoKill.Kill(entireProcessTree: true) to sudoKill.Kill() to avoid hitting the same EPERM when trying to kill the root-owned kill child process.