Proxy incorrectly relays compressed body

[jgilbert2017]

I'm testing via wget --no-check-certificate -e use_proxy=yes -e http_proxy=127.0.0.1:8000 http://www.google.com.

The request header includes Accept-Encoding: identity which is to say the client can't (or won't) process any encoding transformation.

Code in RequestHandler adds gzip and deflate encodings on the request, but the proxy does not seem to decode the body before passing it back to the client. As a result, wget saves the compressed contents rather than the html as we'd want.

        private void PrepareRequestHeaders(HeaderCollection requestHeaders)
        {
            if (requestHeaders.HeaderExists(KnownHeaders.AcceptEncoding))
            {
                requestHeaders.SetOrAddHeaderValue(KnownHeaders.AcceptEncoding, "gzip,deflate");
            }

            requestHeaders.FixProxyHeaders();
        }

I don't quite understand why this code exists.

Interested in what you guys think the proper fix is.

[justcoding121]

The code here rewrites compression headers of request before sending to server, informing that we only accept these two specific types of compression. This is because we don't have any other decompression supported in our proxy other than these two.

The client will receive the compressed body from server as it is, with one of the response headers mentioning the compression selected by server based on the options we gave through request.

I see the problem here. The client mentioned that it cannot accept a compressed response. So we should not rewrite AcceptEncoding in this particular case. So that server would sent the body uncompressed and we would relay the same to client.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding

[jgilbert2017]

Yeah. The problem is that the client gets to choose what set of encodings it supports.

If the client doesn't include any encodings that the proxy understands then the proxy can't really alter the body, can it!

[justcoding121]

The proxy can, by instructing the server that we only accept identity (no compression at all).

i.e, If the client sends accept-encoding that are all not understood by proxy, then we should certainly overwrite it accept-encoding:identity before we sent it to server. In fact we should add "gzip,deflate, indentity" for all requests in above snippet you added instead of just "gzip,deflate".

Server response with its selection using below header.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding

[jgilbert2017]

I'm not quite following you- for example
Client: Accept-Encoding: SomethingProxyDoesntKnow

As I see it, the proxy has 2 options:

1. do opaque pass through to the server.
2. reject the request

If the proxy alters the Accept-Encoding it will not be able to re-encode the response to the client.

[justcoding121]

If the proxy alters the Accept-Encoding it will not be able to re-encode the response to the client.

Proxy will be able to sent to client the body uncompressed with Content-Encoding:identity header. In fact when we overwrite Accept-Encoding: SomethingProxyDoesntKnow to Accept-Encoding: identity in request, server have no option to pick. It will return the result uncompressed with response header Content-Encoding:identity.

That's the simple solution. But a more efficient solution would be to sent Accept-Encoding:gzip,deflate,identity to server, and then we send the body uncompressed to client with Content-Encoding: identity after decompressing it in proxy using one of the three server chose.

[jgilbert2017]

ah, ok- got it. so really the proxy needs to strip unknown encodings from Accept-Encoding to make sure the proxy doesn't receive a response it can't handle.

perhaps that should only happen if the api client is interested in handling that particular request.

[justcoding121]

Yes. We don't read the request/response body to memory unless API user requested it (we relay it directly to server/client). But we do overwrite the header in all cases currently.

[justcoding121]

This should be fixed now.

[justcoding121]

Reopen if any issue arise.