NTLM with Expect: makes libcurl hang

[weltling]

I did this

There is a bug reported to PHP https://bugs.php.net/bug.php?id=71861 being currently investigated. From what I currently could trace, it looks like a bug or behavior change in cURL. I also wrote a simple C program to mimic the behavior https://gist.github.com/weltling/15400de1018c7e6a4b476a415f08768d .

When the NTLM authentication is not used, or the buffer size is a power of two, there's no hang. Otherwise the program hangs.

I expected the following

The program doesn't hang.

curl/libcurl version

With 7.42.1 the program encounters HTTP error and exits, with 7.47.1 or 7.48.0 the program hangs.

curl -V

curl 7.47.1 (x86_64-pc-win32) libcurl/7.47.1 OpenSSL/1.0.1s zlib/1.2.7.3 WinIDN libssh2/1.7.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL libz

operating system

Windows

[iboukris]

Oh, what a coincidence :)
We've been troubleshooting a very similar issue a few days ago (still not figured out but we have some vague idea).

Can you try to add empty 'Expect' header (to disable Expect) and see if it resolves the issue as it does for us?

[weltling]

Thanks for the quick response :) Yeah, removing the Expect header fixes influences the behavior. This is what i did https://gist.github.com/weltling/39860fccf381eba6329b5c742d30ae08 . The verbose output with 7.47.1 is some different from 7.42.1, but there is no hang anymore.

Thanks.

[iboukris]

With debug build I get a crash which I don't see how it is related :(
a.out: transfer.c:1231: Curl_single_getsock: Assertion `conn->writesockfd != -1' failed.

The fun part, is what you found about how the buffer size affects the behaviour.
Trying to look more in to it...

[weltling]

Yeah, I made a mistake in the latest gist. The buffer size in general - yes, it is one of the concerns in the origninal PHP bug. Namely, if the buffer is not a power of two.

Thanks.

[bagder]

The "buffer size" makes the request larger and it is only when the request is larger than a certain limit that libcurl will use the Expect: logic. Other than that, I can't think why power of 2 would have a significance. I personally don't have any live NTLM server to debug against, so this is a little hard for me to track down...

[weltling]

@bagder I din't state it were a valid concern, just mentioned as it were making a diff to the original reporter of the PHP bug. The actual URL reproducing the issue is https://mail.microsoft.com/ews/exchange.asmx , OFC there's only the client side available. But that's the only base one can refer to :(

Thanks.

[weltling]

Anyways clearing the Expect header did the trick in the current repro snippet, hopefully it were helpful.

Thanks.

[iboukris]

Ok, here is what I see so far.

If we don't send Expect then we send the data right away and once we receive 401 we rewind at http_perhapsrewind (since all the byte were sent).
But if we send Expect, we first receive the 401 message and we set the rewindaftersend flag and only then we send the bytes (to be discarded), however we don't seem to rewind and I looks like we try to resend the bytes again and it hangs (or crash in debug build).

See output from my lab at: http://pastebin.com/raw/XP3RYbMi
Notice it prints twice "Keep sending data to get tossed away!".

I managed to fix it to rewind with the patch below but I'm still not clear about the flow.
Also I think we didn't see this problem in older version of curl (~7.27).

POC fix:

diff --git a/lib/transfer.c b/lib/transfer.c
index 7481e0c..3881aef 100644
--- a/lib/transfer.c
+++ b/lib/transfer.c
@@ -986,6 +986,12 @@ static CURLcode readwrite_upload(struct SessionHandle *data,
       /* we have sent all data we were supposed to */
       k->upload_done = TRUE;
       infof(data, "We are completely uploaded and fine\n");
+
+      if(conn->bits.rewindaftersend) {
+        result = Curl_readrewind(conn);
+        if(result)
+          return result;
+      }
     }

     if(data->req.upload_present != bytes_written) {

[bagder]

Curios, can you figure out why it isn't rewound at the "proper" place https://github.com/curl/curl/blob/master/lib/transfer.c#L893 ?

[iboukris]

I think because we did not send the body yet, 'nread' prints 1025 before these lines.

Expect: 100-continue
...
< HTTP/1.1 401 Unauthorized
...
< WWW-Authenticate: NTLM
...
* Rewind stream after send
* Keep sending data to get tossed away!
<
* Ignoring the response-body
* nread: 1025
* We are completely uploaded and fine
* Curl_done

[iboukris]

The fix passes 'make test' btw.
I think it also make sense to check for rewindaftersend and rewind right after sending.

[bagder]

I think because we did not send the body yet, 'nread' prints 1025 before these lines.

Well OK, if it is 1025 there it just means it still has data to send. It still needs to decide it is done sending somewhere within that function and switch off the KEEP_SEND bit, right?

Does it do that on line 1008?

[iboukris]

Yes, it does that at line 1008 but it doesn't rewind.
Moving the rewind fix to line 1008 also works.