[ARM][.NET 7] - System.ArgumentOutOfRangeException on ThreadPool ctor

[tiaraju]

Description

Hi there!

We're running our system in a .NET 7 (more specifically, 7.0.12,) Linux env, and, when changing to use ARM instances, we noticed an unusual behavior where the instances started crashing.

We were unable to pinpoint a cause for that. It happens quite constantly. For a service running with 120 instances, we would have at least 5 instances unhealthy at a single point in time. For the time being, we didn't find anything to trigger this crash.
The exception we see in the logs is the following one:

Unhandled exception. System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'state')
   at System.Threading.ThreadPool.<>c.<.cctor>b__78_0(Object state)
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

We've also noticed that this problem was reported in #84407 and solved in version 7.0.7, PR #84641. Nevertheless, we still noticed it happening when using ARM.
When running with AMD, we didn't notice the same behavior.

Do you have any idea about what may be causing that?
Thanks!

Reproduction Steps

Running a .NET 7.0.12 Web API, using docker on top of AWS EC2 Linux image.

Expected behavior

We expect to see no crashes on the SocketAsyncEngine, as mentioned in the exception above

Actual behavior

Currently, running a service with ~ 120 instances, at a certain point in time, some instances start to crash with the following exception message

Unhandled exception. System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'state')
   at System.Threading.ThreadPool.<>c.<.cctor>b__78_0(Object state)
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

Regression?

No response

Known Workarounds

No response

Configuration

We're running a .NET 7.0.12 web API
We are using the docker image available at mcr.microsoft.com/dotnet/aspnet:7.0
Running on a Debian GNU/Linux 11 (bullseye)

Other information

No response

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Hi there!

We're running our system in a .NET 7 (more specifically, 7.0.12,) Linux env, and, when changing to use ARM instances, we noticed an unusual behavior where the instances started crashing.

We were unable to pinpoint a cause for that. It happens quite constantly. For a service running with 120 instances, we would have at least 5 instances unhealthy at a single point in time. For the time being, we didn't find anything to trigger this crash.
The exception we see in the logs is the following one:

Unhandled exception. System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'state')
   at System.Threading.ThreadPool.<>c.<.cctor>b__78_0(Object state)
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

We've also noticed that this problem was reported in #84407 and solved in version 7.0.7, PR #84641. Nevertheless, we still noticed it happening when using ARM.
When running with AMD, we didn't notice the same behavior.

Do you have any idea about what may be causing that?
Thanks!

Reproduction Steps

Running a .NET 7.0.12 Web API, using docker on top of AWS EC2 Linux image.

Expected behavior

We expect to see no crashes on the SocketAsyncEngine, as mentioned in the exception above

Actual behavior

Currently, running a service with ~ 120 instances, at a certain point in time, some instances start to crash with the following exception message

Unhandled exception. System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'state')
   at System.Threading.ThreadPool.<>c.<.cctor>b__78_0(Object state)
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

Regression?

No response

Known Workarounds

No response

Configuration

We're running a .NET 7.0.12 web API
We are using the docker image available at mcr.microsoft.com/dotnet/aspnet:7.0
Running on a Debian GNU/Linux 11 (bullseye)

Other information

No response

Author:	tiaraju
Assignees:	-
Labels:	area-System.Threading
Milestone:	-

[EgorBo]

Looks like another instance of this problem was fixed recently in aspnet #50690, but it was only backported to 8.0 cc @antonfirsov

[antonfirsov]

@tiaraju any chance you can upgrade to .NET 8.0 RC2 (including the ASP.NET Core Runtime)? Ideally, that should fix this problem.

[tiaraju]

Hey!
thanks for ur response @EgorBo @antonfirsov.

We won't be able to upgrade it to .NET 8 atm, not in the production env at least.

Any plans on releasing the patch to .NET 7 as well?

[antonfirsov]

Any plans on releasing the patch to .NET 7 as well?

The chances are higher if we have a definite proof this is casued by the ASP.NET bug. Would you be able to try a custom build from a patch? (This would mean replacing one dll on the production instances, or at least on a reasonable portion of them)

[tiaraju]

The chances are higher if we have a definite proof this is casued by the ASP.NET bug. Would you be able to try a custom build from a patch? (This would mean replacing one dll on the production instances, or at least on a reasonable portion of them)

Hey, just to make sure I got ur suggestion right, you mean a test running .NET 7 with a custom build applying the changes proposed on your PR? Is that it?

[antonfirsov]

@tiaraju yes.

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Hi there!

We're running our system in a .NET 7 (more specifically, 7.0.12,) Linux env, and, when changing to use ARM instances, we noticed an unusual behavior where the instances started crashing.

We were unable to pinpoint a cause for that. It happens quite constantly. For a service running with 120 instances, we would have at least 5 instances unhealthy at a single point in time. For the time being, we didn't find anything to trigger this crash.
The exception we see in the logs is the following one:

Unhandled exception. System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'state')
   at System.Threading.ThreadPool.<>c.<.cctor>b__78_0(Object state)
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

We've also noticed that this problem was reported in #84407 and solved in version 7.0.7, PR #84641. Nevertheless, we still noticed it happening when using ARM.
When running with AMD, we didn't notice the same behavior.

Do you have any idea about what may be causing that?
Thanks!

Reproduction Steps

Running a .NET 7.0.12 Web API, using docker on top of AWS EC2 Linux image.

Expected behavior

We expect to see no crashes on the SocketAsyncEngine, as mentioned in the exception above

Actual behavior

Currently, running a service with ~ 120 instances, at a certain point in time, some instances start to crash with the following exception message

Unhandled exception. System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'state')
   at System.Threading.ThreadPool.<>c.<.cctor>b__78_0(Object state)
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

Regression?

No response

Known Workarounds

No response

Configuration

We're running a .NET 7.0.12 web API
We are using the docker image available at mcr.microsoft.com/dotnet/aspnet:7.0
Running on a Debian GNU/Linux 11 (bullseye)

Other information

No response

Author:	tiaraju
Assignees:	-
Labels:	area-System.Net.Sockets, untriaged
Milestone:	-

[tiaraju]

@tiaraju yes.

Cool.
We'll run the tests and I'll come back with the results as soon as we got them

[antonfirsov]

@tiaraju am I getting it right that you will be able to build the custom Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.dll by altering the aspnetcore sources? (Much appreciated if yes, or let me know if I should do it for you.)

[chris-peterson]

We are observing this as well (several application crashes/day)

Our environment is various containerized workloads; the issue manifests on multiple patch versions of dotnet, ranging from 7.0.5 through latest (7.0.12)

Not able to run an RC, but could incorporate a patched assembly into our runtime container if that helps get the fix backported

[tiaraju]

@tiaraju am I getting it right that you will be able to build the custom Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.dll by altering the aspnetcore sources? (Much appreciated if yes, or let me know if I should do it for you.)

I'll start to try to run some tests, but If you could help me with that, I'd be grateful @antonfirsov 😅

would be possible to release a -rc, maybe?

[antonfirsov]

I have created a patched build on top of the 7.0.12 state:
Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.zip

You need to replace Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.dll in /usr/lib/dotnet/shared/Microsoft.AspNetCore.App/7.0.12 (or whatever folder your runtime lives in) with the provided file.

Let us know if this makes the issue go away!

[codeimpossible]

FWIW @antonfirsov my company has been having this exact same issue with net7 on arm and the patched build you posted worked for us. Is there a chance of this getting included in a upcoming .net 7 release?