Kestrel stops listening/responding to requests after some time (only after updating to .NET 7.0)

[shaynevanasperen]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

I have a Web Application which runs some background services and hosts some UI pages using Razor Pages. It was working perfectly fine on .NET 6.0, but after simply updating the target framework to net7.0 and redeploying to a folder on my machine, I started to encounter this problem.

Immediately after deploying the application, it works fine and continues to respond to requests from my browsers (Firefox, Chrome, Edge). But anywhere from 10 minutes to 6 hours later, the site no longer responds to HTTP requests (hitting refresh in my browser results in the connection timing out), and then it stays this way until I stop and restart it from the command line.

I have tried several things to attempt to narrow down the problem, but everything I have tried has failed. It's extremely difficult to do this because I have to wait so long between each try. As soon as I revert the target framework back to .NET 6.0, the problem goes away again and the site remains responsive permanently. So unfortunately I am unable to produce a minimalistic project which reproduces the issue.

I have a Prometheus /metrics endpoint exposed through the application and I have Grafana Agent configured on my machine to scrape the metrics every 15 seconds (from http://localhost/metrics). I have a branch in my request pipeline to exclude that endpoint from Authorization. The rest of the application is secured with MicrosoftIdentity authentication pointing to my Azure Active Directory tenant using the OpenIdConnect authentication scheme.

The really strange thing is that when the browser requests start to show connection timeouts, my Grafana Agent scraper is still able to scrape the /metrics endpoint every 15 seconds, which I can confirm from the logs. Here is a snippet from my Program.cs:

builder.WebHost.UseUrls($"https://{DomainName}", $"http://{DomainName}", "http://localhost");

if (!builder.Environment.IsDevelopment())
    builder.Services
        .AddLettuceEncrypt(LetsEncrypt.Options.Configure)
        .PersistCertificatesToAzureKeyVault(LetsEncrypt.AzureKeyVault.Configure);

builder.Services
    .AddRazorPages()
    .AddMicrosoftIdentityUI();

builder.Services
    .AddAuthorization(options => options.FallbackPolicy = options.DefaultPolicy)
    .AddAuthentication(OpenIdConnectDefaults.AuthenticationScheme)
    .AddMicrosoftIdentityWebApp(Authentication.MicrosoftIdentity.Configure);

I've tried disabling all my background sevices, and I even tried disabling authentication/authorization completely, and using self-signed certificates instead of LetsEncrypt, but none of these have had any effect.

Expected Behavior

The web application should continue to respond to HTTP requests indefinitely.

Steps To Reproduce

Host a .NET 7.0 Razor Pages web application on Windows 11. Wait for the site to stop responding to HTTP requests.

Exceptions (if any)

Nothing. Logs show nothing either.

.NET Version

7.0.100

Anything else?

No response

[davidfowl]

Host a .NET 7.0 Razor Pages web application on Windows 11. Wait for the site to stop responding to HTTP requests.

I'm not sure this is enough to reproduce the problem.

A couple more questions:

• Are you running on IIS?
• Can you capture a process dump when the application gets into a bad state?

[shaynevanasperen]

I'm not sure this is enough to reproduce the problem.

I will continue my attempts to narrow the problem down. However it is very frustrating because I have to wait so long each time before it get's into a bad state. I am currently running a test on another machine, as perhaps it has something to do with my environment. But even if that IS the case, why should it behave any differently than it does with .NET 6.0? (that's the only variable).

Are you running on IIS?

No. It is hosted with only with Kestrel and I launch the application from the command line using dotnet .\ApplicationName.dll

Can you capture a process dump when the application gets into a bad state?

I shall do this the next time that happens. I assume I should use this tool? https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump. Shall I upload the process dump somewhere after I have captured it?

[Tratcher]

Once you have the dump, open it in VS and look at the threads window for blocked threads, and the parallel tasks window for blocked tasks. That should give you an idea of where things are stalling.

[shaynevanasperen]

I'm not quite sure how to diagnose the issue using the dump, but when I view the Parallel Stacks, I can see that there are some blocked Async Logical Stacks:

Here is another view:

The source code for the Serilog.Sinks.Async.BackgroundWorkerSink looks like this:

The source code for Crypto.Websocket.Extensions.Core.OrderBooks.Sources.OrderBookSourceBase looks like this:

I noticed that both of them use:

Task.Factory.StartNew(...)

Could this be the culprit? Otherwise, could somebody point me in the right direction of what I should look for next?

[Tratcher]

StartNew is fine. _bufferPauseEvent.WaitOne() isn't great. What unblocks that event? There are async structures you can use to avoid blocking the thread, like SemaphoreSlim, Channels, ConcurrentQueue, etc.

[Tratcher]

Does BackgroundWorkerSink.Pump do something similar?

[shaynevanasperen]

_bufferPauseEvent.WaitOne()

I already tried excluding this code path entirely, and it didn't help

Does BackgroundWorkerSink.Pump do something similar?

If you follow the link to the source code, you can see what it does. But that's also irrelevant because I tried excluding this code as well and it didn't help either.

[shaynevanasperen]

In other news, ever since a few weeks ago my Visual Studio credentials keep expiring on me (like every few minutes) and then my app fails to launch until I refresh the credentials. If this happens overnight after my machine reboots to install Windows updates, then my app won't relaunch until I wake up and realize the problem. Prior to a few weeks ago, my credentials remained cached for months at a time.

And now, whenever I have to refresh the credentials, it hardly even works at all: I just get a loading spinner:

I think it all started happening after installing a recent update for Visual Studio. Could this have anything to do with it?

I have now tried running my app on a different machine, and the problem still occurs there too.

I am at my wits' end. I've probably spent over 40 hours trying to resolve this problem and I am no further than when I started. 😭

[BrennanConroy]

Can you share the parallel stacks (not tasks) window after disabling the serilog logging? Also, is it possible to share the .dmp file? (this can contain PII, so if the app contains customer info don't share it)

[shaynevanasperen]

Can you share the parallel stacks (not tasks) window after disabling the serilog logging?

This is the Parallel Stacks view after my most recent incident (which ran for about 7.5 hours before the problem occurred):

In this version, I have removed all of my own background services and the only services added to the container are done via the following lines of code:

builder.Logging.ClearProviders();
builder.Host.UseSerilog(configure);

builder.Services
    .AddLettuceEncrypt(LetsEncrypt.Options.Configure)
    .PersistCertificatesToAzureKeyVault(LetsEncrypt.AzureKeyVault.Configure);

services.AddSingleton<IPolicyEvaluator, AuthorizationPolicyEvaluator>();
services.AddPrometheusCounters();
services.AddPrometheusAspNetCoreMetrics();
services.AddPrometheusHttpClientMetrics();
services.AddPrometheusSqlClientMetrics();

builder.Services
    .AddRazorPages()
    .AddMicrosoftIdentityUI();

builder.Services
    .AddAuthorization(options => options.FallbackPolicy = options.DefaultPolicy)
    .AddAuthentication(OpenIdConnectDefaults.AuthenticationScheme)
    .AddMicrosoftIdentityWebApp(Authentication.MicrosoftIdentity.Configure);

My AuthorizationPolicyEvaluator looks like this:

class AuthorizationPolicyEvaluator : PolicyEvaluator
{
    public AuthorizationPolicyEvaluator(IAuthorizationService authorization) : base(authorization) { }

    public override async Task<PolicyAuthorizationResult> AuthorizeAsync(AuthorizationPolicy policy, AuthenticateResult authenticationResult, HttpContext context, object? resource)
    {
        if (context.Request.Host.HasValue && context.Request.Host.Value.StartsWith("localhost", StringComparison.OrdinalIgnoreCase) &&
            context.Request.Path.HasValue && context.Request.Path.Value == "/metrics")
            return PolicyAuthorizationResult.Success();

        return await base.AuthorizeAsync(policy, authenticationResult, context, resource);
    }
}

This version also includes Serilog logging, but I have now replaced the Serilog.Sinks.Async with Serilog.Sinks.Channels which doesn't contain the Task.Factory.StartNew(...) call I showed previously.

Also, is it possible to share the .dmp file?

This version contains code which loads App Configuration from Azure, which includes sensitive information (API keys) so unfortunately I can't share it this time. I will now remove that as well, and also remove Serilog logging and see if I can reproduce it that way. If I can reproduce it that way, how can I share the .dmp file (as it is about 1GB in size)? Perhaps if you give me an email address I can share the file with you via Google Drive?

[danmoseley]

The VS creds issue should be unrelated - please report it though the VS feedback button so they can help with that.

[shaynevanasperen]

I have now managed to create a minimalistic project which reproduces this issue: https://github.com/shaynevanasperen/WebApplication

I suspect that the problem has something to do with the LettuceEncrypt.Azure library not liking .NET 7.0. I wonder if it could actually be related to the VS credentials issue, because from my understanding, the LettuceEncrypt.Azure library will check my certificate for renewal once every 24 hours (maybe less) and when it decides that it is time to check again, it tries to load the certificate from Azure KeyVault, but when it does that, my credentials are expired and need to be refreshed manually? But then this should also be happening on .NET 6.0, so maybe not then. Maybe it is just some async code in LettuceEncrypt.Azure that causes a deadlock.

[shaynevanasperen]

Actually, I just re-read my initial comment where I had mentioned that I tried it with self-signed certificates instead of using LettuceEncrypt.Azure and that it showed the problem even then. I am now running a test with self-signed certificates in my minimalistic project. Will report back in several hours, if and when I see the problem again.

[shaynevanasperen]

I've just managed to capture it failing immediately upon startup. I'm not sure if this is the same issue but it smells like it could be related. This was with a version of the code where all the background services are disabled, but still using Serilog and LettuceEncrypt.

The log output is as follows:

PS C:\Alphabot\ProductionG20> dotnet .\MissionControl.dll
[17:03:10.332 INF] Loading Azure App Configuration ...
[17:03:12.131 INF] Loading Azure App Configuration from https://missioncontrol.azconfig.io/kv/?key=%2A&label=%00&api-version=1.0 ...
[17:03:14.001 INF] Finished loading Azure App Configuration.
[17:03:14.082 INF] WebApplication built.
[17:03:14.186 INF] Starting WebApplication ...
[17:03:14.204 DBG] Searching for certificate in KeyVault for alphabot.shaynevanasperen.net {"SourceContext": "LettuceEncrypt.Azure.Internal.AzureKeyVaultCertificateRepository"}
[17:03:21.049 INF] Found certificate for alphabot.shaynevanasperen.net from Azure Key Vault with thumbprint C287F457A433F3B92AEE7B1859F34EAE506A63D9 {"SourceContext": "LettuceEncrypt.Azure.Internal.AzureKeyVaultCertificateRepository"}
[17:03:21.054 DBG] Certificate for alphabot.shaynevanasperen.net already found. {"SourceContext": "LettuceEncrypt.Internal.AcmeStates.ServerStartupState"}
[17:03:21.055 DBG] Checking certificates' renewals for alphabot.shaynevanasperen.net {"SourceContext": "LettuceEncrypt.Internal.AcmeStates.CheckForRenewalState"}

Here is a screenshot of my Parallel Stacks when opening the .dmp file:

And this is the Tasks window:

And this is the Threads window: