NewRelic Agent 10.19.1 breaks Swagger

[egg-r]

Description

This week we restarted receiving reports of Swagger not being able to load by developers this week. We found that this issue is only occurring on environments which have NewRelic profiling enabled. Profiler enablement is controlled by Kubernetes environment variables in our setup. Disabling profiling via K8s environment variables allowed Swagger to work properly. Turn it back on and Swagger fails with the below console output. We then traced what changed down to the NewRelic latest agent updates, 10.19.1 . Previous docker images with agent version installations prior to version 10.19.x are working fine. With Swagger being broken, we're pretty worried now about other aspects of our application also having issues so we disabled profiling globally while this bug gets reviewed.

Expected Behavior

The NewRelic agent should not be breaking Swagger or other bits of code that rely on Kestrel.

Troubleshooting or NR Diag results

fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      System.InvalidOperationException: Headers are read-only, response has already started.
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpHeaders.ThrowHeadersReadOnlyException()
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpHeaders.set_ContentLength(Nullable`1 value)
         at Microsoft.AspNetCore.Http.DefaultHttpResponse.set_ContentLength(Nullable`1 value)
         at NewRelic.Providers.Wrapper.AspNetCore6Plus.BrowserInjectingStreamWrapper.FlushAsync(CancellationToken cancellationToken)
         at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionBody.FlushAsync(CancellationToken cancellationToken)
         at System.IO.Pipelines.StreamPipeWriter.FlushAsyncInternal(Boolean writeToStream, ReadOnlyMemory`1 data, CancellationToken cancellationToken)
         at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.RespondWithIndexHtml(HttpResponse response)
         at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
         at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
warn: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[2]
      The response has already started, the error handler will not be executed.
fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HMV0T1A0SJP9", Request id "0HMV0T1A0SJP9:00000002": An unhandled exception was thrown by the application.
      System.InvalidOperationException: Headers are read-only, response has already started.
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpHeaders.ThrowHeadersReadOnlyException()
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpHeaders.set_ContentLength(Nullable`1 value)
         at Microsoft.AspNetCore.Http.DefaultHttpResponse.set_ContentLength(Nullable`1 value)
         at NewRelic.Providers.Wrapper.AspNetCore6Plus.BrowserInjectingStreamWrapper.FlushAsync(CancellationToken cancellationToken)
         at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionBody.FlushAsync(CancellationToken cancellationToken)
         at System.IO.Pipelines.StreamPipeWriter.FlushAsyncInternal(Boolean writeToStream, ReadOnlyMemory`1 data, CancellationToken cancellationToken)
         at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.RespondWithIndexHtml(HttpResponse response)
         at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
         at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.HandleException(HttpContext context, ExceptionDispatchInfo edi)
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
         at Scorpion.Stretch.Web.EnableBuffering.InvokeAsync(HttpContext context)
         at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
         at NewRelic.Providers.Wrapper.AspNetCore6Plus.WrapPipelineMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

Steps to Reproduce
Use the latest NR agent 10.19.1

Your Environment
Docker image mcr.microsoft.com/dotnet/aspnet:6.0-bullseye-slim on AKS

Additional context

For Maintainers Only or Hero Triaging this bug
Suggested Priority (P1,P2,P3,P4,P5):
Suggested T-Shirt size (S, M, L, XL, Unknown):

[workato-integration]

https://new-relic.atlassian.net/browse/NR-179063

[tippmar-nr]

@egg-r Thanks for your report. We're currently working on an updated release that will disable ASP.NET Core 6+ browser agent injection by default, which will alleviate the issues you're seeing. That release should be available within the next day or so.

We will be investigating the issue you reported and working to resolve them so that we can re-enable browser agent injection by default in a future release.