InvalidProgramException with message "JIT Compiler encountered an internal limitation." thrown during HTTP calls

[martincostello]

This morning when a new EC2 instance was launched for an ASP.NET Core 2.2.5 application of ours in a production environment, all outbound HTTP requests to dependent services failed with the following exception:

System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to write data to the transport connection: JIT Compiler encountered an internal limitation.. ---> System.InvalidProgramException: JIT Compiler encountered an internal limitation.
   at T System.Threading.LazyInitializer.EnsureInitialized<T>(ref T target)
   at ValueTask System.Net.Sockets.Socket.SendAsyncForNetworkStream(ReadOnlyMemory<byte> buffer, SocketFlags socketFlags, CancellationToken cancellationToken)
   at ValueTask System.Net.Sockets.NetworkStream.WriteAsync(ReadOnlyMemory<byte> buffer, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at ValueTask System.Net.Sockets.NetworkStream.WriteAsync(ReadOnlyMemory<byte> buffer, CancellationToken cancellationToken)
   at ValueTask System.Net.Http.HttpConnection.WriteToStreamAsync(ReadOnlyMemory<byte> source)
   at ValueTask System.Net.Http.HttpConnection.FlushAsync()
   at async Task<HttpResponseMessage> System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at async Task<HttpResponseMessage> System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at async Task<HttpResponseMessage> System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, bool doRequestAuth, CancellationToken cancellationToken)
   at async Task<HttpResponseMessage> System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, bool doRequestAuth, CancellationToken cancellationToken)
   at async Task<HttpResponseMessage> System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at async Task<HttpResponseMessage> System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at async Task<HttpResponseMessage> Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
<snip>

The inner-most exception appears to indicate a JIT issue.

This is the first time we've ever seen this issue, but it seemed to be unrecoverable. Exceptions were thrown for all HTTP requests made by the application until it was removed from our application load balancer and stopped receiving user traffic.

We've captured two full dumps of the dotnet.exe from the instance before it was terminated which we can share with your privately if it helps diagnose the issue.

Let me know if there's any further information I can share with you other than the dump files.

[AndyAyersMS]

A few questions:

• Which OS is the app running on?
• Is this something that runs fine on older 2.2 instances?
• Does this always fail on 2.2.5 or is the failure intermittent?

cc @leecow for recommendations on getting us access to the dump files.

[martincostello]

Sorry, I meant to write the OS in the first place but forgot to add it... 🤦‍♂

• The application runs on Windows Server 2016.
• The application has been running 2.2.5 in production since 16th May.
• The application has been run on all previous publicly released 2.2.x releases, as well as further in the past various releases of 1.1, 2.0 and 2.1.
• The application has never experienced this specific issue before.

Based on the above, if the issue is new to 2.2.5, it only manifested itself for the first time this morning on a single EC2 instance.

[AndyAyersMS]

Are you using any kind of profiler with the app?

[martincostello]

We have the AppDynamics.Agent.Windows NuGet package installed in the app.

It was either version 4.5.10 or 4.5.11 yesterday morning, I’ll need to check later this morning exactly which it was at that time.

[martincostello]

The version was 4.5.10.

[AndyAyersMS]

@RussKeldorph do we have recommended tech for securely getting large files like dumps?

[martincostello]

I’ve provided them to the AspNetCore team before via a link shared by them to a SharePoint folder.

[RussKeldorph]

@janvorli How have you done this?

[AndyAyersMS]

@martincostello apologies for the slow response. Can you open an issue for this on the developer community site?

Once it's open, you can add a "private" comment and attach your dump file there.

[martincostello]

@AndyAyersMS Dump submitted: https://developercommunity.visualstudio.com/content/problem/613477/invalidprogramexception-with-message-jit-compiler.html

[AndyAyersMS]

Got it, thanks.

First impression is that the dump doesn't have enough information to pinpoint what went wrong, as the process is past the point of exception. Looks like the issue happened while jitting LazyInitializer.EnsureInitialized<T> or perhaps the ...Core helper; from what I can tell neither of these was modified by the profiler.

I'll see if I can get somebody else to take a look; perhaps they can spot something I overlooked.

[AndyAyersMS]

With some timely help from @ChrisAhna, @noahfalk and @davmason, we found that the IL for LazyInitializer.EnsureInitialized<T> for ref-type T was indeed modified by the ReJIT mechanism, and that the new IL supplied for the method is not well-formed. So it appears this is an issue with the way the AppDyanmics Agent is handling this particular method.

[martincostello]

Thanks @AndyAyersMS, I’ll feed this back to AppDynamics on Monday.

[AndyAyersMS]

@martincostello can you update this issue when you've heard from AppDynamics?

[martincostello]

@AndyAyersMS Sure - I emailed them the details from here and the community site, and they're investigating.