.NET 10 runtime live-lock with Task.Delay?

[ladeak]

I have a .NET 10 console application that enters a live-lock in the runtime since .NET 10 update? The application has a loop something like this:

...
try
{
    while(!token.IsCancellationRequested)
    {
       LogInformation("Iteration Start");
       DoWork();
       Gc.Collect(2);
       LogInformation("1 minute sleep");
       await Task.Delay(TimeSpan.FromMinutes(1), token);
    }
}
catch(OperationCancelledException ex)
{
     LogInformation("Shutdown");
}
finally
{
    LogInformation("Process Close");
}

This is the code since .NET8, and started to observe the following behavior ~2 weeks after .NET9 -> .NET10 update:

The application is running for multiple days, and after multiple days the last message it displays in the logs is "1 minute sleep", and then is stops 'working'. No logs/no telemetry pushed to

The application runs in a chiseled Linux container since. The container is not restarted, the process is still running within the container and it using 100% CPU. The memory consumption is 'normal' of the process.

Connecting any diagnostics to the application fails: dotnet-trace/dotnet-stack/dotnet-dump/dotnet-gcdump/procdump.
All tools managed to connect to the live-locked process, but all of them hang when capturing any diagnostic information.

What would be the suggested next steps to understand the root cause? Is it possible that Task.Delay causes a live-lock?

[Clockwork-Muse]

1. We'd need to know what DoWork() is actually doing, as it may be the culprit. If your example is accurate, it's strange you're not passing the cancellation token there as well.
2. Manually calling Gc.Collect(...) isn't usually something a program should be doing, and may be a sign that your code has other problems. Note that this will be a blocking call, so in extreme cases this can severely degrade the performance of your application.
3. await has no active thread, and consumes no/negligible CPU for Task.Delay(...); the application would actually show 0% CPU usage (barring certain extreme uses or a custom threadpool scheduler or other unusual things).

[ladeak]

1. DoWork() is more or less the same for many years. It loads data from DB, processes it, and saves it to the DB. The token is not passed in. This method does a lot of logging and telemetry on its own if the cycle was running - but this is not visible to be happening at the time live-lock happens.
2. GC.Collect is there to suggest a cleanup before the next iteration. It is not necessary - I can try removing it.
3. This is my understanding of Task.Delay too.

[Clockwork-Muse]

It loads data from DB, processes it, and saves it to the DB.

This call should also be async, and take in the cancellation token (even if the only async part is the database call).

Note that logging/telemetry is usually buffered for performance reasons, so it's possible that the problem is happening in your DoWork(), it's just the buffer hasn't been flushed to output yet.

As a side note, if your application really has a one-minute sleep cycle like that, would it just be better to have the container start up every minute and run whatever processing?

[ladeak]

I completely understand your points, but do work is thousands of lines of code that has not really changed substantially for a longer period. My main concern is if there any live-lock at any part of the 'user-code', I should be able to go there and create a memory dump or take a memory snapshot of the threads -> but these operations don't work at all. For now I added some Console.Write's to avoid the logging's buffer and removed the GC call. I suppose this is another 2 weeks at least until the errors shows again.
It is not scheduled every minute as a cron job to make sure that it iteration completes before the next one starts.

[KeterSCP]

@ladeak what .NET 10 version are you running on? Can it be related to #123950, which was fixed in 10.0.4?

[ladeak]

It can be very well related! I will double check the version soon. Thank you for the suggestion 😊