Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HttpClient POST does not send body if request contains basic auth #28012

Closed
butlermatt opened this Issue Dec 6, 2016 · 10 comments

Comments

Projects
None yet
4 participants
@butlermatt
Copy link

butlermatt commented Dec 6, 2016

Hi folks,

I'm using Dart:io to connect to a remote server to send SOAP messages. In previous versions of their firmware, I was able to do this without any issues. That firmware used basic authentication through the URL. They have since updated their firmware.

The new Firmware uses Digest authentication and I am able to send a GET request without any issues. However when I issue a post request it times out after about 8 and 1/2 minutes with a Bad request.

Normally I would say this is due to the change in their firmware. However I am able to successfully complete the same request using cUrl from the command line.

Regretfully, I'm unable to provide access to the remote server, as it is on a security camera. However are there any tips or suggestions on how I can help to further diagnose this issue in the Dart SDK, since cUrl does work without any issues.

@butlermatt

This comment has been minimized.

Copy link
Author

butlermatt commented Dec 6, 2016

To follow up, I do know that the remote server seems to be running gSOAP/2.7

@butlermatt

This comment has been minimized.

Copy link
Author

butlermatt commented Dec 6, 2016

From what I can see from captured pcap files (Between curl and the dart posts) it seems like the Dart request never sends more than the headers. The post body doesn't show up in the requests, whereas it does in the curl post requests.

@a-siva

This comment has been minimized.

Copy link
Contributor

a-siva commented Dec 7, 2016

Is it possible to also compare the packets received back from the server between curl and the dart client.

We would require some kind of reproduction in order to diagnose the issue.

@butlermatt

This comment has been minimized.

Copy link
Author

butlermatt commented Dec 7, 2016

I could provide the pcap files privately (due to potentially confidential information contained there in). But from my recollection, there was no response from the remote server after the Dart headers were received, as it seemed like the server was still waiting for the content body, whereas the curl example contained the body (plus a follow-up with the remainder of the body) and the server provided ack then the expected http response (eg, headers and the body contents)

@butlermatt

This comment has been minimized.

Copy link
Author

butlermatt commented Dec 8, 2016

HI @a-siva Just to follow up, looking at the pcap again.

Process for curl:
After the initial handshake, curl sends the headers (without the authentication digest), and specifically with content-length 0.
The remote server sends and ack packet for that.
The remote server sends 401 unauthorized response (with body).
Mac sends back an ack for that.
Curl then sends the headers (including digest authentication) and body of the request to the remote server (two packets)
Remote server sends an ack packet.
Remote server sends the Http response (headers and soap body).
They close the connection.

On the dart side:
After initial handshake, Dart sends headers (without authentication digest) to server.
Server sends ack.
Server sends 401 unauthorized response (with body).
Dart sends headers again (without digest authentication and without transfer-encoding header).
Server sends ack and 401 response again (with headers and body)
Mac sends ack for that packet.
Dart sends headers again, with digest authentication headers, content-length 0 still.
Server sends ack packet.
No further communication from Dart.

I've confirmed that the body of the request is being written to the request, but it seems it's not being sent.
Regretfully, I do not have an instance of the older firmware anymore to be able to provide the communication flow from that one. However I do know the primary difference with that connection is that the authentication used basic authentication with credentials specified directly in the URL rather than using the HttpClient.authenticate callback.

@butlermatt

This comment has been minimized.

Copy link
Author

butlermatt commented Dec 9, 2016

I managed to create a small reproduction here with Shelf for a server and a very basic client.

https://github.com/butlermatt/basic-error

Start the server running.
Load localhost:8080 in the browser. See that it does stuff (eg basic auth) Credentials are "test" and "test"
Run the client. See that it does authenticate, however it does not post the body (that in writeln) same if you use the add. That data is never posted as the body of the request.

Comment out lines 33 - 50 in the server. Restart, and run client again. The request body is posted as expected.

@butlermatt butlermatt changed the title HttpClient POST request times out after 8+ minutes HttpClient POST does not send body if request contains basic auth Dec 9, 2016

@jonasfj

This comment has been minimized.

Copy link
Contributor

jonasfj commented Aug 9, 2018

In your example you are using HttpClient.authenticate, this means that the HttpClient will attempt the request without an 'authorization' header resulting in a failure. After which the HttpClient will call callback assigned to HttpClient.authenticate, which is where you set the 'authorization' header.

Hence, you are actually doing two requests, but only sending the body once. Since the body is streaming the HttpClient can't just blindly assume it's of a size that can be buffered. Thus, the body isn't attached to the second request which does include the 'authorization' header.

The solution for you is to do as follows:

var client = HttpClient();
var creds = HttpClientBasicCredentials("username", "password");
var u = Uri("https://example.com/mypath");
client.addCredentials(u, "", creds):

var req = await client.postUrl(u);
req.write("hello-world");

var res = await req.close();
var result = await utf8.decodeStream(res);
print(result):

client.close();

In this case you are setting the 'authorization' header upfront, hence, the HttpClient will only making one request. It's also possible that it's better to use the http package from pub, it might be more high-level.


I think this behavior appears slightly surprising. Considering that the documentation for HttpClient.authenticate says:

If the Future completes with true the request will be retried using the updated credentials. Otherwise response processing will continue normally.

When in practice the request can't be retried, because the body is streaming. It might an idea to remove the HttpClient.authenticate property, or forbid it's use for anything other than HEAD and GET -- similar to how we forbid automatically following redirects for anything other than HEAD and GET.
But given that this has been stable for a long time, it might be preferable to update the documentation with appropriate warnings and examples.

I'll admit that useful use cases for HttpClient.authenticate alludes me at this moment, but I'm also pretty new to dart, so maybe this is an opportunity to learn something :)

@jonasfj

This comment has been minimized.

Copy link
Contributor

jonasfj commented Aug 9, 2018

@dart-bot dart-bot closed this in a9178a2 Aug 13, 2018

@aaulia

This comment has been minimized.

Copy link

aaulia commented Feb 2, 2019

Hello,

Is there any plan to actually fix this? Coming from kotlin with OkHttp, HttpClient.authenticate provide similar feature as Authenticator. It really streamline and simplify (re)authentication process to a single spot.

@jonasfj

I'll admit that useful use cases for HttpClient.authenticate alludes me at this moment, but I'm also pretty new to dart, so maybe this is an opportunity to learn something :)

From the above OkHttp Authenticator documentation

Performs either preemptive authentication before connecting to a proxy server, or reactive authentication after receiving a challenge from either an origin web server or proxy server.

The later is really useful in handling refresh token grant type authentication.

@jonasfj

This comment has been minimized.

Copy link
Contributor

jonasfj commented Feb 4, 2019

The later is really useful in handling refresh token grant type authentication.

If you send a POST request with a body and get an authentication challenge from the server after you started streaming the body (because why wouldn't you start sending body immediately), then you can't send the body again, because the body is a stream, and streams can't be restarted.

You could buffer the body in-memory, but then you run out of memory when someone uploads a large file...

So we would need to use some form of restartable stream... I'm not saying it's a bad idea, just that when the body is a stream, retrying the request is not necessarily an option.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.