Workaround for recv() buffer flushed when send() returns error on closed connection

[Karlson2k]

Should fix issue #657.

Tested locally - works good.
Not sure that I followed correctly coding style and didn't break anything. Anyway, I'll improve it if needed. _WIN32 in ifdefs can be replaced with some winsock marker.
If code itself is OK, I'd like to add comments before merging.
May be code for early recv() must be moved to one function instead of duplicating in two places.

The code is simple and don't receive more data than fits into the single buffer, but that shouldn't be a problem as those early responses are usually a short error messages with sizes < 200 bytes.

[bagder]

First, there needs to be proper precautions against overflowing that fixed-size buffer, as you don't ever clear the variable that adds data to the buffer. Do we really need to read unconditionally before every send() ? I suspect that may hamper transfer speeds, especially when doing many parallel transfers. Also, it is unfortunate to add ~16K of data to every easy handle for this.

Then, I would like the '#ifdef' to rather be for a name that describes what its for so that we can build and run the code on non-windows platforms too to make sure it works and doesn't ruin anything.

When talking non-plaintext protocols, like TLS, this work-around will probably not work anyway since the TLS library can very well both read and write data on its own.

I would still like this Window TCP stack bug acknowledged by an official page/forum/person so that we know when to switch it off and also if it could just be something we do that triggers it.

[gvanem]

Then, I would like the '#ifdef' to rather be for a name that describes ...

That would then be #ifdef USE_WINSOCK. (and not simply _WIN32 since Windows can use another tcp/ip stack. Although highly unlikely).

I would still like this Window TCP stack bug acknowledged by an official page/forum/person...

Absolutely. I suspect Winsock is working-as-designed. Seems related to how libcurl is shoehorning telnet.c to work using the (*enum_netevents_func)() function.

[bagder]

That would then be #ifdef USE_WINSOCK

I was even thinking USE_WINSOCK_WORKAROUND or similar so that we can build with another stack plus the workaround, to make sure that the work-around version works even when everything else is fine.

Seems related to how libcurl is shoehorning telnet.c to work

I don't think so, as @Karlson2k speaks of problems when doing HTTP POST and the telnet "shoehorning" is only for telnet.

[Karlson2k]

First, there needs to be proper precautions against overflowing that fixed-size buffer, as you don't ever clear the variable that adds data to the buffer.

I did not assign the value because I suspect that structure is nullified somewhere. Anyway I used this mostly like a POC, real code polishing is not done yet.

Do we really need to read unconditionally before every send() ?

Not recv() - select() with single socket before every send(). This is transformed internally to NtDeviceIoControlFile() and NtWaitForSingleObject() (or may be only NtDeviceIoControlFile() - didn't try to go deep with zero timeout). With nonblocking socket we can simply try to recv(), but I'm not sure that sockets are always non-blocking in libcurl.
And yes, it's required before any send() as any send() can trigger detection of remote closure of connection. The only exception - first send, when nothing can be received before.

I suspect that may hamper transfer speeds, especially when doing many parallel transfers.

Don't think so, but only real testing can prove any difference. If difference will not be neglectable, than we can add some Win32 runtime option to disable this workaround in favor for speed.

Also, it is unfortunate to add ~16K of data to every easy handle for this.

We can replace it with curl buffers, but need to carefully checking that this are freed.

Then, I would like the '#ifdef' to rather be for a name that describes what its for so that we can build and run the code on non-windows platforms too to make sure it works and doesn't ruin anything.

USE_WINSOCK_WORKAROUNDS or EARLY_RECV_WHILE_SENDING?

Then, I would like the '#ifdef' to rather be for a name that describes what its for so that we can build and run the code on non-windows platforms too to make sure it works and doesn't ruin anything.

Code call libcurl wrapper for recv() so this should work with TLS as well - this just gives TLS lib chance to read early.

I would still like this Window TCP stack bug acknowledged by an official page/forum/person so that we know when to switch it off and also if it could just be something we do that triggers it.

So do I, but there is not official way to manually report WinSock bugs to Microsoft. Will try to find suitable official forum. If you know any - let me know, please.

That would then be #ifdef USE_WINSOCK. (and not simply _WIN32 since Windows can use another tcp/ip stack. Although highly unlikely).

Cygwin use wrapper over WinSock so libcurl on Cygwin doesn't use WinSock (directly), but workaround is still required.

[Karlson2k]

Seems related to how libcurl is shoehorning telnet.c to work

I don't think so, as @Karlson2k speaks of problems when doing HTTP POST and the telnet
"shoehorning" is only for telnet.

If libcurl sends HTTP POST in one send() than libcurl will not notice this bug. May be some other protocols can trigger this too - I don't know. Currently libcurl passed only 816-818 tests out of 884 tests of internal test suite on Win64.

[bagder]

Currently libcurl passed only 816-818 tests out of 884 tests of internal test suite on Win64.

Right, but that's because of other reasons...

[Karlson2k]

Currently libcurl passed only 816-818 tests out of 884 tests of internal test suite on Win64.

Right, but that's because of other reasons...

May be, or may be not. We didn't know. 😄
For the patch itself: just replace static array with curl_buffer and track all initializations / deinitializations?

[bagder]

For the patch itself: just replace static array with curl_buffer and track all initializations / deinitializations?

That won't suffice. Recall that we could be uploading a 4GB file and the response could be a 4GB response body that starts streaming down while the upload is going on. The protocol allows this.

[bagder]

In addition to that, both easy handles and connections can be re-used for subsequent transfers.

[gvanem]

@bagder ... and the telnet "shoehorning" is only for telnet.

Ok, but do you agree (with no bias) that liburl/Win32 has some "shoehorning" or protocol tweaks? If yes, why should Telnet be handled specially?

Hence looking at the big-picture of how libcurl vs other programs do socket I/O, something can be learned as to why libcurl needs such tweaks in the first place. For example, it would be interesting to know exactly how a C++ program using Qt5 or Boost.Asio would behave in such a case. This Winsock "bug" would have caused hysteria. Here is the design overview of Boost.Asio:
http://www.boost.org/doc/libs/1_37_0/doc/html/boost_asio/overview/core/basics.html

What the inner detail of io_service::run() does on Windows, I don't know yet.

[bagder]

Ok, but do you agree (with no bias) that liburl/Win32 has some "shoehorning" or protocol tweaks? If yes, why should Telnet be handled specially?

It really shouldn't, it just still is and nobody has fixed it. In the telnet case I think mostly because virtually nobody ever use telnet so it doesn't hurt much. I know for me that's why I don't bother much with cleaning that up.

This Winsock "bug" would have caused hysteria

I can't but to think the same, which is also partly why I'd like it acknowledged/dismissed by some sort of official party before we proceed and sprinkle the code with weirdo code.

[gvanem]

weirdo code.

Not the wording I'd chose, but LOL 😄 .

[Karlson2k]

For the patch itself: just replace static array with curl_buffer and track all initializations / deinitializations?

That won't suffice. Recall that we could be uploading a 4GB file and the response could be a 4GB response body that starts streaming down while the upload is going on. The protocol allows this.

In addition to that, both easy handles and connections can be re-used for subsequent transfers.

Code already handles all above.
Currently code fill small 16Kb (or smaller) buffer when sending and read it when started receiving.
Code will not use more than 16Kb, if buffer is full - recv() is not called. Code can be optimized to avoid select() in this case.
If connection if re-used, than current behavior is not changed. (Before connection is re-used, all incoming data must be received.)

In other words:
This patch is practically only add a intermediate tiny buffer between libcurl recv() and WinSock. If data comes early, it's moved from WinSock buffer to internal intermediate buffer so data is not lost when WinSock detect TCP closure.
Small buffer is enough to hold HTTP error responses, for everything bigger - data will wait in WinSock buffers.

[bagder]

Ah yes sorry I forgot that when I then wrote the comment, Is reading only the first 16K really what you want? If the bug is that the last received data (before a send) is lost, then shouldn't the code make sure that the last data is read out from the kernel's socket buffer?

But I would like to take a step back first. Can we produce a stand-alone test case to reproduce this bug with so that we can try it out on different windows versions and see where we have the problems and possibly get to understand it even further? I would like our work-around code to be as narrow and precise as possible.

[Karlson2k]

Is reading only the first 16K really what you want? If the bug is that the last received data (before a send) is lost, then shouldn't the code make sure that the last data is read out from the kernel's socket buffer?

Yes, only last small amount of data, usually a not more than 500 bytes.
Unfortunately, I didn't found any way to force WinSock to push data from system buffers, but seems that WinSock did it automatically when received 'FIN'.

But I would like to take a step back first. Can we produce a stand-alone test case to reproduce this bug with so that we can try it out on different windows versions and see where we have the problems and possibly get to understand it even further? I would like our work-around code to be as narrow and precise as possible.

It's already present. Not curl-based, but very simplified and cross-platform, I published it in issue #657 description:
https://github.com/Karlson2k/check_system_socket_recv_last
Bug present on Windows 7/Vista. Seems that Win2k is affected as well as WinXP. Win8/8.1/10 not tested yet.

I was trying to add libcurl test, but I'm too deep on fixing curl testsuite errors on W32.

[mback2k]

If this bug is reproducible on different versions of Windows, this should probably be reported to Microsoft. They might come up with a KB or Fixit for this in that case.

[Karlson2k]

@bagder Patch updated with little bit more elegant version. Now passed test 1517 from PR #720 and libmicrohttpd testsuite.
Still WIP, ToDo list:

• Reset buffer when switched connection (move definition to another structure?)
• Replace _WIN32 with USE_WINSOCK_WORKAROUNDS.
• Add different buffers for different sockets (not really relevant for HTTP, but for future-proof).

Could you give me some hint, which place of code is best to add buffer reset for new connection?

[bagder]

What if Curl_send_plain() is called twice (with incoming data present) before Curl_recv_plain() is called? Doesn't it need to append the new data to the temporary buffer?

[Karlson2k]

It's already handled by

          ssize_t recvedbytes = sread(sockfd, st->postponed_buffer +
                                      st->recv_postponed_size, bytestorecv);

Buffer is allocated once, assuming that it will be enough.

May be need better names, or someone can be confused with postponed_buffer_size and recv_postponed_size.

[bagder]

Oh right, when st->recv_postponed_size grows to st->postponed_buffer_size it will ask to read 0 bytes. But is there actually anything that says that is a safe assumption and shouldn't the code at least avoid the case when there's no space left to read data into?

[Karlson2k]

        const size_t bytestorecv = st->postponed_buffer_size -
                                   st->recv_postponed_size;
        if(bytestorecv) {
          ssize_t recvedbytes = sread(sockfd, st->postponed_buffer +
                                      st->recv_postponed_size, bytestorecv);

If st->recv_postponed_size grows to st->postponed_buffer_size then bytestorecv is zero is recv() is skipped.
I'd like to keep this patch as simple as possible and less intrusive.
Don't think that we need to handle any really huge buffers here.

[bagder]

I just thought that if there's no space left to read data, there's no point in doing the Curl_socket_check() call to begin with... But that's not a biggie.

[Karlson2k]

It should be rare situation.
However you're right, Curl_socket_check() is not expensive call, but it produce a system call, so better to avoid it if possible.

[bagder]

The title "Workaround for WinSock bug with destroyed recv() buffer if TCP connection was closed remotely". But how is this closure detected? Isn't it detected by send() failing and we bail out on that?

[Karlson2k]

But how is this closure detected? Isn't it detected by send() failing and we bail out on that?

I could only imagine how WinSock internals work. But I suspect that WinSock detect remote disconnection and store it in some socket-specific structure. If recv() is called after it, WinSock simply return received data. So far everything is correct. But if send() is called for socket marked with "closed by remote side" then WinSock call some error handler for socket (as WinSock unable to send) which free any socket-allocated resources, including received buffer.
So you can't do anything after failed send(), because buffers are already destroyed.

[bagder]

I want to clarify the title of this issue so I'm asking how libcurl detects the remote close in this case. I think it is detected by send() returning a failure. Isn't that what happens?

[Karlson2k]

libcurl doesn't detect remote close, libcurl just saves incoming data before sending.

[bagder]

Sure it detects remote close, by the corresponding call returning 0.

[Karlson2k]

WinSock bug is triggered by calling send() when socket is already disconnected (and received data is present in WinSock buffers).
send() will return -1 in this case.

[bagder]

Here as well.

[bagder]

BTW as the added select() will unconditionally do a system call and then you might do another with the added recv(), isn't it faster to just blindly try a recv() without any select() first ?

[Karlson2k]

@bagder Are sockets always non-blocking?

[Karlson2k]

@bagder Updated.

[Karlson2k]

@bagder Updated again, syntax fixed.

[Karlson2k]

@bagder And if recv() is blindly called each time, then extra buffer will be allocated for each socket, which is system call too.
I'd expect that this code patch will be used rarely, so it worth to make a select() call to save buffer allocation for each socket.

[Karlson2k]

@bagder Updated with proper commit description.

[Karlson2k]

Any chances to merge it somewhen soon?

[bagder]

It doesn't merge cleanly right now, can you rebase it please?

[Karlson2k]

@bagder Rebased.
PR #720 (test) doesn't need rebasing.

[jay]

Was this ever reported to Microsoft and also this workaround seems to be only for plaintext? Apologies if these questions were already answered, it's been a while and a cursory read of this I can't find the answers.

[bagder]

I'm not aware of any report, but I'm also not entirely sure this is against any spec or documentation. I haven't found anything that clearly says this isn't permitted behavior. So the problem is then possibly "just" a libcurl problem when we send() several times in a row without recv()ing and thus risk losing data.