Null Reference Exception when upgrading from 10.18 to 10.19

[sjauld]

Description
My apps have started throwing exceptions after updating the NewRelic agent from 10.18 to 10.19. I'm not sure exactly why, but rolling back to 10.18 fixed the issue.

Expected Behavior
Application should not throw an exception

Troubleshooting or NR Diag results

Error message:
Object reference not set to an instance of an object.
System.NullReferenceException: Object reference not set to an instance of an object.
at NewRelic.Providers.Wrapper.AspNetCore6Plus.BrowserInjectingStreamWrapper.IsHtmlResponse(Boolean forceReCheck)
at NewRelic.Providers.Wrapper.AspNetCore6Plus.BrowserInjectingStreamWrapper.FlushAsync(CancellationToken cancellationToken)
at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionBody.FlushAsync(CancellationToken cancellationToken)
at Microsoft.AspNetCore.ResponseCaching.ResponseCachingStream.FlushAsync(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Http.HttpResponseWritingExtensions.StartAndWriteAsyncAwaited(HttpResponse response, String text, Encoding encoding, CancellationToken cancellationToken, Task startAsyncTask)
at <My application> in <my file>:line 49
at Microsoft.AspNetCore.Mvc.ActionResult.ExecuteResultAsync(ActionContext context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeResultAsync>g__Logged|22_0(ResourceInvoker invoker, IActionResult result)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|30_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.Invoke(HttpContext context)
at NWebsec.AspNetCore.Middleware.Middleware.CspMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)

Steps to Reproduce
Update the NR agent to 10.19

Your Environment
Docker mcr.microsoft.com/dotnet/aspnet:6.0

Additional context

For Maintainers Only or Hero Triaging this bug
Suggested Priority (P1,P2,P3,P4,P5):
Suggested T-Shirt size (S, M, L, XL, Unknown):

[workato-integration]

https://new-relic.atlassian.net/browse/NR-178419

[tippmar-nr]

Hi @sjauld -

Thanks for your bug report. We're investigating the issue now and hope to have a fix soon.

At this point, we've been unable to reproduce the issue on our side. Would it be possible for you to share some details about your application configuration - specifically, the sequence of your middleware pipeline configuration might shed some light as to what's going on.

Also, if you can let us know whether installing 10.19.0 caused every request to fail or if it was limited to only certain requests, that would be helpful. If only some requests are failing, any information you can share about the type of request and the response that's being generated would be appreciated.

Please be sure not to share any proprietary or otherwise confidential information in your responses.

Thanks!

[FlorentATo]

We just had an incident in production because of this bug. All docker/kubernetes applications configured to pull the latest version (as recommended) could be affected.

In our case, the issue was triggered by a faulty HttpContext fetched from the dapr sidecar (see the endpoint and the call from the sidecar).

dapr error:

time="2023-11-08T17:39:11.965124743Z" level=info msg="initializing Dapr workflow component" app_id=***REDACTED*** component="dapr (workflow.dapr/v1)" instance=***REDACTED***-dc749cc7b-rtkkz scope=dapr.contrib type=log ver=1.10.5
time="2023-11-08T17:39:12.236059031Z" level=error msg="app returned http status code 500 from subscription endpoint" app_id=***REDACTED*** instance=***REDACTED***-dc749cc7b-rtkkz scope=dapr.runtime type=log ver=1.10.5
time="2023-11-08T17:39:12.236930373Z" level=info msg="dapr initialized. Status: Running. Init Elapsed 9177ms" app_id=***REDACTED*** instance=***REDACTED***-dc749cc7b-rtkkz scope=dapr.runtime type=log ver=1.10.5
time="2023-11-08T17:39:13.024922211Z" level=info msg="placement tables updated, version: 0" app_id=***REDACTED*** instance=***REDACTED***-dc749cc7b-rtkkz scope=dapr.runtime.actor.internal.placement type=log ver=1.10.5
time="2023-11-08T18:37:17.512854778Z" level=info msg="Dapr shutting down" app_id=***REDACTED*** instance=***REDACTED***-dc749cc7b-rtkkz scope=dapr.runtime type=log ver=1.10.5
time="2023-11-08T18:37:17.512923796Z" level=info msg="Stopping PubSub subscribers and input bindings" app_id=***REDACTED*** instance=***REDACTED***-dc749cc7b-rtkkz scope=dapr.runtime type=log ver=1.10.5
time="2023-11-08T18:37:17.512942401Z" level=info msg="Initiating actor shutdown" app_id=***REDACTED*** instance=***REDACTED***-dc749cc7b-rtkkz scope=dapr.runtime type=log ver=1.10.5
time="2023-11-08T18:37:17.512953241Z" level=info msg="Shutting down actor" app_id=***REDACTED*** instance=***REDACTED***-dc749cc7b-rtkkz scope=dapr.runtime type=log ver=1.10.5

Fix (dockerfile):

RUN apt-get update && apt-get install -y wget ca-certificates gnupg \
&& echo 'deb http://apt.newrelic.com/debian/ newrelic non-free' | tee /etc/apt/sources.list.d/newrelic.list \
&& wget https://download.newrelic.com/548C16BF.gpg \
&& apt-key add 548C16BF.gpg \
&& apt-get update \
&& apt-get install -y newrelic-dotnet-agent=10.18.0 \   <- HERE
&& rm -rf /var/lib/apt/lists/*

[tippmar-nr]

@FlorentATo Thanks for sharing that information. With a little poking around in the Dapr implementation, we've figured out how to reproduce the issue locally and are working on a fix.

[sjauld]

Hi @tippmar-nr -

I guess you've identified the issue, but just to confirm, not every request failed under 10.19, just some. I haven't really had time to dig in to differences between failing and non-failing requests yet but hopefully you don't need that any more now you have found the issue.

Thanks,

[tippmar-nr]

@sjauld @FlorentATo can you please try release 10.19.1 and confirm whether the issues are resolved? Thanks!

[sjauld]

Hi @tippmar-nr, that works real good for me.

Thanks for the quick turnaround!

[FlorentATo]

@tippmar-nr I'll try to get back to you on this before the end of the week. Your restrictiveness on this issue is much appreciated!

[FlorentATo]

@tippmar-nr No more errors on our end too 👏