Skip to content

ProxyResponse is null in AddResponseTransform #2562

@arnonax-tr

Description

@arnonax-tr

Describe the bug

I'm not 100% sure it's a bug, but either it is, or something is unclear to me.

Occasionally* I'm seeing that ResponseTransformContext.ProxyResponse is null inside a delegate I pass to TransformBuilderContext.AddResponseTransform, and I don't understand how that's possible. I see that the summary of ResponseTransformContext.ProxyResponse says it "can be null if the destination did not respond", but I see from the logs of the proxied service that it did send a response (with Status code 401- Unauthorized, though I'm not sure it's related). Even if for some reason that response message was lost, then I would expect to have a way to get the error of the root cause (e.g. timeout, DNS issue, network disconnected, etc.). but I don't find any property that return that information. Note that I don't think that the problem is timeout because from the logs I see that I got the callback immediately after the proxied service responded.

*I saw it only twice out of 632 responses from the same endpoint (over a week period), and out of 9635 responses from all (3) endpoints. All of the responses from that endpoint return status 401. In most cases ResponseTransformContext.ProxyResponse is not null, but in those two cases out of 632 it is.

To Reproduce

I cannot reproduce it deterministically (and the chances to hit are very low as I mentioned above), but I can outline the structure of my code in regard to YARP. (It may be possible to reproduce it deterministically using load testing, but I don't have the time and resources for it now)

// In Bootstrapper:
private void ConfigureReverseProxy(IServiceCollection services)
{
        services.AddReverseProxy().LoadFromMemory(new[]
        {
            new RouteConfig
            {
                RouteId = "route1",
                ClusterId = "cluster1",
                Match = new RouteMatch
                {
                    Path = "{**catch-all}"
                }
            }
        },
        new[]
        {
            new ClusterConfig
            {
                ClusterId = "cluster1",
                Destinations = new Dictionary<string, DestinationConfig>
                {
                    ["destination1"] = new() { Address = ConfigurationProvider.StsUrl }
                }
            }
        })
    .AddTransforms<StsProxyTransforms>();
}

internal class MyProxyTransforms : ITransformProvider
{
    void ITransformProvider.ValidateRoute(TransformRouteValidationContext context)
    {
    }

    void ITransformProvider.ValidateCluster(TransformClusterValidationContext context)
    {
    }
    
    public void Apply(TransformBuilderContext context)
    {
        context.AddRequestTransform(async requestContext =>
        {
            // Here we examine the request and do some stuff, but we don't modify the request
            // ...
        });

        context.AddResponseTransform(async responseContext =>
        {
            // THE FOLLOWING LINE OCCASIONALLY THROWS NullReferenceException BECAUSE responseContext.ProxyResponse is null...
            if (!responseContext.ProxyResponse.IsSuccessStatusCode)
            // ...
        }
    }
}

Further technical details

  • Package: Yarp.ReverseProxy (2.1.0)
  • The platform where the error occurs: .Net 6 on Linux

Activity

MihaZupan

MihaZupan commented on Aug 2, 2024

@MihaZupan
Member

You can access the error via httpContext.Features.Get<IForwarderErrorFeature>().
It'll be set by the time you see a null response in a response transform.

arnonax-tr

arnonax-tr commented on Aug 4, 2024

@arnonax-tr
Author

Thanks @MihaZupan , I'll try that. I think it's worth mentioning in the docs of ResponseTransformContext.ProxyResponse

arnonax-tr

arnonax-tr commented on Aug 8, 2024

@arnonax-tr
Author

I tried it and now I see (in another endpoint BTW) the following information:

Error: RequestCanceled
Exception: System.Threading.Tasks.TaskCanceledException
Message: A task was canceled.
Stack trace:
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at Yarp.ReverseProxy.Forwarder.HttpForwarder.SendAsync(HttpContext context, String destinationPrefix, HttpMessageInvoker httpClient, ForwarderRequestConfig requestConfig, HttpTransformer transformer, CancellationToken cancellationToken)

Does it give you any hint about the root cause? To me it still doesn't mean anything useful...

MihaZupan

MihaZupan commented on Aug 13, 2024

@MihaZupan
Member

https://github.com/microsoft/reverse-proxy/blob/b1a1d8b20a2ff599517705771ce87614799e8ceb/src/ReverseProxy/Forwarder/ForwarderError.cs#L26-L29

RequestCanceled generally indicates that the client (e.g. browser) disconnected/canceled the request before YARP was able to get a response from the backend server.
This may be as innocent as the user navigating away to a different web page.

Updating the docs on ProxyResponse to hint at using IForwarderErrorFeature makes sense.

added
Type: DocumentationImprovements or additions to documentation
and removed
Type: BugSomething isn't working
on Aug 13, 2024
added this to the Backlog milestone on Aug 13, 2024
arnonax-tr

arnonax-tr commented on Aug 14, 2024

@arnonax-tr
Author

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type: DocumentationImprovements or additions to documentationhelp wantedWe will welcome a contribution

Type

No type

Projects

No projects

Relationships

None yet

    Participants

    @benjaminpetit@MihaZupan@arnonax-tr

    Issue actions

      ProxyResponse is null in AddResponseTransform · Issue #2562 · dotnet/yarp