"async" fails as the number of threads increases

UPDATE 03/05/2021:
A few months ago we discovered that this was somehow tied to the use of the async keyword, and have since been able to prove it in the AcmeWebApi test application. As @davidfowl has stated in our other thread, this is most likely a race condition. Since it is virtually impossible to write any application that doesn't use async in some way now (due to the core library changes). it is not possible to run tests that only use synchronous code. I can say that when we did have synchronous code, we were seeing a drastically higher performance metric than we currently are seeing.

Leaving the following, as that is what we started the thread with:

We have run into several issues with ASP.NET Core that appear to be threading related. I initially created #26955, as that was the first issue that we ran into, but creating an application that can be tested is till ongoing. In the process of creating an application for that purpose, we were able to replicate another issue, which is the topic of this thread. The application linked below replicates this issue under the following conditions:

• 3,500+ concurrent clients (we are required to exceed 8,000).
• Average throughput in total across all connections is 6,000 req/s (this is our minimum for maximum throughput).
• VM has 2 dedicated CPUs and 4GB RAM.

Under these conditions we observe numerous HeartbeatSlow issues across random connections (threads), which in our full application leads to complete system failure over time. We are working on providing ways to replicate the other issues that we have observed, but these are currently the only ones that we can replicate for you in a test application.

This issue ONLY exists on Linux (we used Ubuntu 20.04.1 LTS for verification) and results in both a significant reduction of throughput and a significantly higher latency. When running in our full application, this issue, along with others, causes a complete system failure (APPCRASH) as the process runs out of memory. No matter how much we try, this issue, and the others, cannot be replicated in a Windows environment (Windows 10 and Windows Server 2019 were tested).

SDK: 3.1.301
VS: 16.8.2

The test application is available in the private repo AcmeWebApi.

FULL DISCOLSURE
I work for Webroot / Carbonite / OpenText and the application discussed above is the property of said entities. Microsoft is a direct / indirect customer of ours, so I am limited on the information that I am allowed to provide.

[sebastienros]

Your repository is currently private. Can you make it public?

@sebastienros done

Thanks for contacting us.
We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We will evaluate the request when we are planning the work for the next milestone. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[sebastienros]

Based on
https://github.com/Kaelum/AcmeWebApi/blob/main/src/AcmeWebApi/Services/ApiService.cs#L329
and
https://github.com/Kaelum/AcmeWebApi/blob/main/src/AcmeWebApi/Handlers/TcpHandler.cs#L415-L420

You are probably filling up the thread pool since you use 3,500 simultaneous connections that are all queued and block each thread.

I don't think it shows a bug in aspnet. Do you want to explain what you are trying to achieve to get some advice on how to approach it differently?

@sebastienros if you looked at the application, you'd see that 18,000 threads are available.

[halter73]

I assume you're referring to how 18,0000 "min" worker/completion threads are specified here which is used here to call ThreadPool.SetMinThreads.

Unfortunately, calling SetMinThreads to set a high number of "min" threads is not a server bullet that allows you scale blocking code. I wish it was. It would make porting legacy pre-async apps far easier. The SetMinThreads MSDN doc specifically warns agains this:

Caution: By default, the minimum number of threads is set to the number of processors on a system. You can use the SetMinThreads method to increase the minimum number of threads. However, unnecessarily increasing these values can cause performance problems. If too many tasks start at the same time, all of them might appear to be slow. In most cases, the thread pool will perform better with its own algorithm for allocating threads. Reducing the minimum to less than the number of processors can also hurt performance.

And while it might not be obvious given the "min" in the name, the CLR doesn't just spin up 18,000 worker + 18,000 completion threads during the call to SetMinThreads. Instead it merely tells the ThreadPool that if there is currently less than the specified "min" ThreadPool threads and there isn't an idle thread ready to handle a new work item when it's dispatched to the ThreadPool, the ThreadPool should immediately spawn a new thread for the work item instead of using the default (slow) algorithm for managing thread creation and destruction.

If the CLR were to actually try to spin up 18,000 worker + 18,000 completion threads during the call to SetMinThreads, it would most likely be killed by the Linux OOM killer (or "(APPCRASH) as the process runs out of memory") before the call to SetMinThreads could complete. Threads use a lot of memory. Since the CLR spawns the threads lazily, the process doesn't get killed until later when too many concurrent items are dispatched leading to too many spawned threads.

And even if all those threads were somehow preallocated and available, a system with far less cores than threads can still easily become overwhelmed because of all of the preemption and context switching that all those threads incur. That's what the warning on the MSDN doc is getting at.

@halter73 I never said otherwise. There are enough threads, and there is no thread starvation. The simple fact that this works perfectly on Windows, and not at all on Linux, should raise a red flag. Anywho, I'll wait for Microsoft to fix the issues. This is only the beginning of all the issues that we've uncovered.

[davidfowl]

cc @kouvel

[sebastienros]

@Kaelum Sorry if you mentioned it already, but were you running in IIS or directly through kestrel on Windows?

@sebastienros you should look at the application. IIS doesn't run on Linux.