Unable to debug .NET 10/AOT apps: "the debugger timed out trying to pause process"

[littlehoneybunnytuttifruttipumpkinpie]

Description

Console application often hangs when being debugged.
I'm also getting error "the debugger timed out trying to pause process" when debugging a console app after upgrading from .NET 8 to .NET 10 and when pressing F9 to set a breakpoint while my application is running in debug mode.
When asked to terminate the program and selecting "Yes", I get a dialog "The breakpoint failed to bind".
Visual Studio becomes unresponsive afterwards, unless I terminate the program. Sometimes I have to terminate and restart Visual Studio.
Note that this happens every time, thus it's not possible to debug the application. Visual Studio became completely unusable.

The issue may be related to these:
https://developercommunity.visualstudio.com/t/VS2022-1712-Program-freezes-when-debu/10834051
#112747
https://www.reddit.com/r/dotnet/comments/1kn8g6d/diagnosing_large_net_framework_48_application/

The same app works fine when published. Also worked fine with .NET 8 until upgraded to .NET 10 preview 4 just a few days ago. I don't recall whether I've updated Visual Studio itself around that time.

In my case, the issue seem to happen mainly when running multi-threaded code using tasks and a few tasks have been started. As this issues is repeatable, I counted that it happens every time after starting a number of tasks, then the application gets stuck and becomes unresponsive with the F9/debug only triggering the error, but not sure that it's related only to debugging.
The main program that gets stuck is included as a project in parent project that is being ran.

Disabling “Break all processes when one process breaks” in Debug->Options did not help.

I've identified the issue to be likely related to using Task.Run, which I'm enclosing in the reproduction steps. Though it also happens with regular async/await Task, but not as often.

Reproduction Steps

Start a console app in debug mode. When app seems stuck, press F9 on any line of code.
The code below seems to be the main culprit. Task.Run is used due to CPU-intensive process.

            short max_tasks = 10; //out of 24 available
            List<Task> task_list = [];
            foreach (string keyName in dictionaryName.Keys)
            {
                task_list.Add(
                    Task.Run(
                    () =>
                    {
                        methodName(keyName);
                    })
                );

                //Wait if running too many tasks
                while (task_list != null && task_list.Count >= max_tasks)
                {
                    //Wait for any task
                    await Task.WhenAny(task_list.ToArray());

                    lock (task_list)
                    {
                        for (int i = task_list.Count - 1; i >= 0; i--)
                        {
                            if (task_list[i] == null || task_list[i].Status == TaskStatus.RanToCompletion || task_list[i].Status == TaskStatus.Faulted)
                            {
                                task_list.RemoveAt(i);
                            }
                        }
                    }
                }

            //Wait for remaining tasks started above
            await Task.WhenAll(task_list.ToArray());

The program seems to work fine with max_tasks=1.
When converting to more basic task_list.Add(methodName());, the program seems to work more often, but frequently still manages to hang and cause errors.

Expected behavior

Expected behavior is for the program to run to completion, while being able to debug it.

Actual behavior

Actual behavior is that the program hangs, while hanging and causing errors when trying to debug.

Regression?

It worked fine with .NET 8.

Known Workarounds

No response

Configuration

.NET 10.0.0-preview.4.25258.110

VS Info:
Microsoft Visual Studio Community 2022
Version 17.13.7
VisualStudio.17.Release/17.13.7+36105.23
Microsoft .NET Framework
Version 4.8.09032

Main app settings:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <PublishAot>true</PublishAot>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Apparatus.AOT.Reflection" Version="1.1.3" />
  </ItemGroup>
</Project>

Other information

No response

[tommcdon]

Hi @littlehoneybunnytuttifruttipumpkinpie would you mind sharing a reproducible test case? The code in the repro steps seems to be missing some variable declarations.
We have a private build of .NET 10 Preview 4 with a speculative fix for the debugger hang - https://github.com/tommcdon/runtime/releases/tag/10_p4_dbg_fix. Would you mind giving it a try?

[littlehoneybunnytuttifruttipumpkinpie]

I downloaded and placed the fix in "...\10.0.0-preview.4.25258.110". Then needed to run VS with VSDebugger_ValidateDotnetDebugLibSignatures=0 and start it from the same command prompt.
Unfortunately the debugger still hangs when attempting to debug the program, with symptoms identical to those prior (same error messages).

It is possible that it's not the debugger that hangs, but the program hangs due to potential race conditions. However, I wasn't yet able to identify the problem in the program itself, especially that it always worked fine up until .NET 8. It still works fine when published, but hangs and never completes only with F5 in Visual Studio, even when not setting any breakpoints. It's difficult to identify the problem in the program (if any) due to layers of code built up over the years, though it was never problematic previously.
I also just tried building and running the program from the debug folder and it works fine there too, taking 30 seconds. While at the same time it's hang in Visual Studio debug mode and never completes. An initial breakpoint is reached prior to Task.Run loop, then it hangs when continued.

Unfortunately I cannot provide the full program as it is prioprietary and nests multiple custom projects and libraries that require gigabytes of data to be loaded from db and perform complex mathematical calculations (using MathNet.Numerics, if that matters). Thus I only provided an abstraction of the main part of the program that starts multiple CPU tasks via Task.Run (it's structured in specific way to also allow for keyboard press and CPU temp monitoring that I removed in recent tests).

[littlehoneybunnytuttifruttipumpkinpie]

I was able to produce a complete test program that exhibits the same problems using the exact code below:

internal static async Task DebuggerIssueTest()
{
    int max_tasks = 10;
    List<Task> task_list = [];

    for (int i = 0; i <= 10000; i++)
    {
        task_list.Add(DebuggerIssueTest2());

        if (!await Sys.WaitAndReleaseTask(max_tasks, task_list))
        {
            await Task.WhenAll(task_list.ToArray()); //Finish all tasks
            return;
        }
    }
    await Task.WhenAll(task_list.ToArray());
}

internal static async Task DebuggerIssueTest2()
{
    try
    {
        int max_tasks = 10;
        List<Task> task_list = [];

        for (int i = 0; i <= 10000; i++)
        {
            task_list.Add(Task.Run(LongRunningTask));

            if (!await Sys.WaitAndReleaseTask(max_tasks, task_list))
            {
                await Task.WhenAll(task_list.ToArray()); //Finish all tasks
                return;
            }
        }
        await Task.WhenAll(task_list.ToArray());
    }
    catch (Exception ex)
    {
        int debug = 1;
    }
}

internal static void LongRunningTask()
{
    try
    {
        double result = 0;
        for (int j = 0; j < 1_000_000_000; j++)
        {
            result += Math.Sqrt(j) * Math.Sin(j);
        }
    }
    catch (Exception ex)
    {
        int debug = 1;
    }
}


        public static async Task<bool> WaitAndReleaseTask(short max_tasks, List<Task> task_list, bool check_for_esc_key = true)
        {
            try
            {
                //Exit with false when 'Esc' or 'Q' pressed
                if (check_for_esc_key && CheckForEscapeOrQKey())
                {
                    Console.WriteLine("'Esc' key detected. Aborting.");
                    if (!Console.Title.Contains("ABORTING"))
                    {
                        Console.Title += " - ABORTING";
                    }
                    return false;
                }

                //Wait if running too many tasks
                while (task_list != null && task_list.Count >= max_tasks)
                {
                    //Console.WriteLine($"Running tasks: {task_list.Count}.");

                    //Try waiting for any task, catching and ignoring potential errors when tasks_list contains a null task (very rare)
                    try
                    {
                        await Task.WhenAny(task_list.ToArray());
                    }
                    catch
                    {
                        //Do nothing
                    }

                    lock (task_list)
                    {
                        for (int i = task_list.Count - 1; i >= 0; i--)
                        {
                            if (task_list[i] == null || task_list[i].Status == TaskStatus.RanToCompletion || task_list[i].Status == TaskStatus.Faulted)
                            {
                                task_list.RemoveAt(i);
                            }
                        }
                    }
                }

                return true;
            }
            catch (Exception ex)
            {
                Log.log.Error(ex);
                return Input.GetInputYesNo("Error in await Sys.WaitAndReleaseTask. (y) to Continue; (n) to abort further tasks.", true);
            }
        }

[tommcdon]

Thank you @littlehoneybunnytuttifruttipumpkinpie - we will use this test app to investigate!