Possible DefaultHttpContext corruption with the default factory

[Tragetaschen]

I have been on quite the chase to figure out frequent request errors in production with .NET 9.0. There is a "solution" that I found today, but it's on the weird side of things. I'm looking for some input where I might find the actual underlying cause.

I have a "normal" request pipeline with auth, routing, endpoints and a couple custom middlewares. There is a ControllerBase [ApiController] executing a request. What I've been chasing is the ObjectDisposedException that can be thrown when trying to access the DefaultHttpContext.Features property, but from within the active request. I catch that exception in the controller action and log it there before returning an explicit error to the client. On the face of it, there are two ways for this to happen that I could identify.

One is when Uninitialize overwrites the _features member with the struct default. I think I have successfully concluded that this is not what's happening. There is a log enricher that takes the HttpContext from IHttpContextAccessor and, if not null, logs several of its (nested) properties, including many that are sourced from the feature collection. This includes context.RequestAborted.IsCancellationRequested, which is always false. In addition, there is an [UnsafeAccessor] for the _active field of the DefaultHttpContext and this is always true in the logs. The log enricher runs synchronously in the catch within the controller action.

The other is the possibility of a corruption from trying to access HttpContext properties concurrently as outlined here. This is where I spent most of my energy trying to identify if we have that anywhere, but I couldn't identify a pattern from the code. My plan today was to write a custom IHttpContextFactory that would return a DefaultHttpContext wrapped by a custom HttpContext implementation type that checks for concurrent access.

However, after deploying just a custom IHttpContextFactory that just mimics the implementation of DefaultHttpContextFactory, including returning a normal DefaultHttpContext, all of the ODEs that happened ~10 times per minute went away entirely. I realize that HostingApplication has some special sauce for the internal members of DefaultHttpContextFactory that doesn't run with a custom factory, but I fail to see how that can cause the effects that I had been observing.

Custom factory code:

using Microsoft.AspNetCore.Http.Features;
using Microsoft.Extensions.Options;

namespace Services;

public class ConcurrencyCheckingHttpContextFactory : IHttpContextFactory {
  private readonly IHttpContextAccessor? _httpContextAccessor;
  private readonly FormOptions _formOptions;
  private readonly IServiceScopeFactory _serviceScopeFactory;

  public ConcurrencyCheckingHttpContextFactory(IServiceProvider serviceProvider) {
    _httpContextAccessor = serviceProvider.GetService<IHttpContextAccessor>();
    _formOptions = serviceProvider.GetRequiredService<IOptions<FormOptions>>().Value;
    _serviceScopeFactory = serviceProvider.GetRequiredService<IServiceScopeFactory>();
  }

  public HttpContext Create(IFeatureCollection featureCollection) {
    var httpContext = new DefaultHttpContext(featureCollection);

    if (_httpContextAccessor != null) {
      _httpContextAccessor.HttpContext = httpContext;
    }

    httpContext.FormOptions = _formOptions;
    httpContext.ServiceScopeFactory = _serviceScopeFactory;

    return httpContext;
  }

  public void Dispose(HttpContext httpContext) {
    if (_httpContextAccessor != null) {
      _httpContextAccessor.HttpContext = null;
    }
  }
}

With this at the end of the Startup.ConfigureServices method

    services.Replace(new(typeof(IHttpContextFactory), typeof(ConcurrencyCheckingHttpContextFactory), ServiceLifetime.Singleton));