Url encoding of original query string

[gt5700]

Hi.
The proxy seems to be URL encoding the original query string the client sent before it passes it to the server, rather than sending the query string identical to how it was received.

I have a what I think is a "badly behaving" web site running on an embedded webserver on a network device that uses a web interface for configuration. The query string includes the characters { and } which would normally be Url encoded. When the device receives the Url encoded version of the query string, it's response is invalid, and the web interface is broken.

I have tested with some other proxy servers, and they work ok. I used Wireshark to see that the query string was being Url encoded by the Titanium-Web-Proxy.

Looking through the code, I found if I changed line 101 in Http/HttpWebClient.cs from this...

        await writer.WriteLineAsync(Request.CreateRequestLine(Request.Method,
            useUpstreamProxy || isTransparent ? Request.RequestUriString : Request.RequestUri.PathAndQuery,
            Request.HttpVersion), cancellationToken);

to this...

        await writer.WriteLineAsync(Request.CreateRequestLine(Request.Method,
            useUpstreamProxy || isTransparent ? Request.RequestUriString : Request.RequestUri.GetComponents(UriComponents.PathAndQuery, UriFormat.Unescaped),
            Request.HttpVersion), cancellationToken);

Then the problem goes away. It looks like Request.RequestUri.PathAndQuery is performing Url encoding.

I don't know if this is the only place this needs to be considered in the code. I have no control over the "badly behaving" devices issues, assuming it is wrong what it is doing....

Regards
Garry

[honfika]

I think that the PathAndQuery should be unescaped only when it was received as non escaped.

Because the change above causes that the requests which are escaped (as sent by a modern browser) will be forwarded as unescaped. I think it is not OK.

The problem is that .NET's uri class does not preserve the original format:

> var u = new Uri("http://example.com/a?p=txt%20txt2");
. u.PathAndQuery + "        "+ u.GetComponents(UriComponents.PathAndQuery, UriFormat.Unescaped)
"/a?p=txt%20txt2        /a?p=txt txt2"
> var u = new Uri("http://example.com/a?p=txt txt2");
. u.PathAndQuery + "        "+ u.GetComponents(UriComponents.PathAndQuery, UriFormat.Unescaped)
"/a?p=txt%20txt2        /a?p=txt txt2"

I suggest the following change:

            string url;
            if (useUpstreamProxy || isTransparent)
            {
                url = Request.RequestUriString;
            }
            else
            {
                var uri = Request.RequestUri;
                url = uri.IsWellFormedOriginalString() ? uri.PathAndQuery : uri.GetComponents(UriComponents.PathAndQuery, UriFormat.Unescaped);
            }

            // prepare the request & headers
            await writer.WriteLineAsync(Request.CreateRequestLine(Request.Method, url, Request.HttpVersion), cancellationToken);

Hopefully it will work with your embedded webserver.

[honfika]

Please try the latest beta package.

[gt5700]

I manually updated my code with your suggestion, and I think it works well. I had some other small issues with the devices webserver, and they seem to be fixed with your suggested code as well as the original issue.

[ljh1991125]

I think this correction cause another problem.
After this correction, I can not connect Microsoft one drive using titanium proxy.
When i try to connect "https://login.microsoftonline.com/51aee0f2-088d-4d62-aa6b-e8bc57757c49/oauth2/authorize?client_id=...&response_type=code%20id_token&...&redirect_uri=..." , it happens.
I think the reason is following url is not "IsWellFormedOriginalString" so url unescape is occur.
url = uri.IsWellFormedOriginalString() ? uri.PathAndQuery : uri.GetComponents(UriComponents.PathAndQuery, UriFormat.Unescaped);

I catched url using wireshark, url is changed like this
"https://login.microsoftonline.com/51aee0f2-088d-4d62-aa6b-e8bc57757c49/oauth2/authorize?client_id=...&response_type=code id_token&...&redirect_uri=..."

"code%20id_token" is unescaped like this "code id_token"

And http request failed with http 400 error.
How can i solve this problem?

[honfika]

For me it shows that it is well formed.

> var uri = new Uri("https://login.microsoftonline.com/51aee0f2-088d-4d62-aa6b-e8bc57757c49/oauth2/authorize?client_id=1&response_type=code%20id_token&aa&redirect_uri=2");
> uri.IsWellFormedOriginalString()
true
> uri.PathAndQuery
"/51aee0f2-088d-4d62-aa6b-e8bc57757c49/oauth2/authorize?client_id=1&response_type=code%20id_token&aa&redirect_uri=2"
> 

Are you sure that it is not wellformed in your machine? Which framework? .NET or Core, Mono, etc?

[ljh1991125]

I am using .NET and this is the full url.
https://login.windows.net/51aee0f2-088d-4d62-aa6b-e8bc57757c49/oauth2/authorize?client%5Fid=00000003%2D0000%2D0ff1%2Dce00%2D000000000000&response%5Fmode=form%5Fpost&protectedtoken=true&response%5Ftype=code%20id%5Ftoken&resource=00000003%2D0000%2D0ff1%2Dce00%2D000000000000&scope=openid&nonce=EDCA4ADD3F15B5B1CA8176E97E8F874EA1265CF951DF666E%2DE032D216FCBA8A073E41174EADD070F57BA82D44766CF4D9C94BD609783A8DE3&redirect%5Furi=https%3A%2F%2Fxxx%2Dmy%2Esharepoint%2Ecom%2F%5Fforms%2Fdefault%2Easpx&wsucxt=1&cobrandid=11bd8083%2D87e0%2D41b5%2Dbb78%2D0bc43c8a8e8a&client%2Drequest%2Did=435cf99e%2D8084%2D9000%2D4960%2Dfdc677e4ca21

uri.IsWellFormedOriginalString() is false

[ljh1991125]

I solved this problem without modifying Titanium Proxy source code.
At OnRequest function, I checked Request URI and if Request URI is not well formed, the URI is reconstructed.
But I don't sure if my code is correct.
This is my source code.

public async Task OnRequest(object sender, SessionEventArgs e)
{
...
	if (e.msg.Request.RequestUri.IsWellFormedOriginalString() == false)
	{
		e.msg.Request.RequestUri = new Uri(Uri.EscapeUriString(e.msg.Request.RequestUri.GetComponents(UriComponents.AbsoluteUri, UriFormat.Unescaped)));
	}
...
}

[honfika]

@ljh1991125 This is the new method, please try the latest (3.1.1037) package. I hope it is ok now.

        internal static string GetOriginalPathAndQuery(this Uri uri)
        {
            string leftPart = uri.GetLeftPart(UriPartial.Authority);
            if (uri.OriginalString.StartsWith(leftPart))
                return uri.OriginalString.Substring(leftPart.Length);

            return uri.IsWellFormedOriginalString() ? uri.PathAndQuery : uri.GetComponents(UriComponents.PathAndQuery, UriFormat.Unescaped);
        }

[honfika]

I close this issue, feel free to repoen if needed.