-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
win,tcp: make uv_close work more like unix #3036
Conversation
This reliably fails one benchmark test on libuv-in-node CI, though the error reporting seems quite useless to tell more without more involved investigations:
|
This is an attempt to fix some resource management issues on Windows. Win32 sockets have an issue where it sends an RST packet if there is an outstanding overlapped calls. We can avoid that by being certain to explicitly cancel our read and write requests first. This also removes some conditional cleanup code, since we might as well clean it up eagerly (like unix). Otherwise, it looks to me like these might cause the accept callbacks to be run after the endgame had freed the memory for them. The comment here seems mixed up between send and recv buffers. The default behavior on calling `closesocket` is already to do a graceful shutdown (see https://docs.microsoft.com/en-us/windows/win32/api/winsock/nf-winsock-closesocket with default l_onoff=zero) if it is the last open handle. The expected behavior if there are pending reads in flight is to send an RST packet, notifying the client that the server connection was destroyed before acknowledging the EOF. Refs: libuv#3035 Refs: nodejs/node#35946 Refs: nodejs/node#35904 Fixes: libuv#3034 PR-URL: libuv#3036
The prior attempt was good for reads, but ignored writes. We need to retire those first too, before calling closesocket.
The prior attempt was great for reads and writes, but not for cancelation. The expected behavior is for writes to be canceled (not finished) on uv_close, while in attempt 2 we were letting them finish. We need to explicitly notify Win32 that it is okay to cancel these writes (so it doesn't also generate an RST packet on the wire).
38e2978
to
ab3e742
Compare
First attempt was showing good improvement, but I'd neglected to deal with writes also, which was the cause of that nodejs test now reliably failing. The followup commits should now handle those also. (@mmomtchev) CI ✅: https://ci.nodejs.org/job/libuv-test-commit-windows-cmake/279/ |
@vtjnash I manually applied this as a patch to the 1.40 in Node-master and I still get a |
I am looking at that test now, and am a bit confused. It appears to me that, per the TCP spec, the client should be sending an RST message to the server, so the server learns that the client no longer cares about the data remaining in its send buffer. Sometimes the message may be dropped in transit though as an optimization, if it learns that the server already doesn't care. Thus, there appears to have been a partial workaround / optimization implemented in the PR that added that test, but no actual handling for the condition in that PR (so may be worth removing that patch, to surface the error itself in regular testing). However, http does seem to have an error handler which should do the right thing(tm) and quietly tear down the socket: https://github.com/nodejs/node/blob/e6e64f7a21f5570b5da780c8162fe10dc7c62be9/lib/_http_server.js#L654-L657 But you're sure it's an RST from server -> client that you're now seeing (per comment in your original issue / PR)? |
Okay, yes, I see that sequence in wireshark. Googling it seems to suggest that the RST may be due to a TLS protocol violation, and is warning of some sort of vulnerability to https://en.wikipedia.org/wiki/Transport_Layer_Security#Truncation_attack (I'm unclear on the exact details of that hack). This is clarified by reading the standard https://tools.ietf.org/html/rfc5246#section-7.2.1 however, which notes that attack, and describes the behavior of nodejs as being valid. Here's a sample record of running that test on windows, with TLS decryption (--tls-keylog), in Wireshark, with these works in progress applied:
So what I think appears to be the case is that nodejs is failing to follow this requirement of the standard:
So by my reading, the standard here requires that this read error be ignored for http (as it's after the close_notify) event, and that nodejs https should have already initiate destroy on the socket itself by this time. Other protocols (not https) on top of TLS are allowed to continue receiving data after the close_notify, but in that case, the server would be in violation to have destroyed the socket already. |
To zero'th order, it looks like nodejs deletes the socket 'end' handler after the headers are done. So we can fix your observed bug by putting back a method that just matches does what the standard says here and destroys the socket:
This is isn't necessary here for plain http, since the only way you get an Perhaps this could even be added to the meaning of This patch isn't quite complete however, since it would get added too many times with KeepAlive. Possibly, like the TODO several lines later for handling errors for onSocket, we want this only to be active when it's not being actively used for something else. |
@vtjnash I didn't read the TLS truncation in detail, but it can't be this for sure - the TLS server, which is running Node, has no |
There's many ways that an application can trigger an RST, the I'm not sure what you mean by "active close". Do you mean that the server should wait to destroy the socket until after the close_notify handshake is finished? If so, that's a valid behavior, but isn't mandated (or particularly recommended), so other servers won't be expected to follow that rule, and thus the client would still needs the real patch. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
RFC 5246 section-7.2.1 requires that the implementation must immediately stop reading from the stream, as it is no longer TLS-encrypted. The underlying stream is permitted to still pump events (and errors) to other users, but those are now unencrypted, so we should not process them here. But therefore, we do not want to stop the underlying stream, as there could be another user of it, but we do need to remove ourselves as a listener. Per TLS v1.2, we should have also destroy the TLS state entirely here (including the writing side), but this was revised in TLS v1.3 to permit the stream to continue to flush output. There appears to be some inconsistencies in the way nodejs handles ownership of the underlying stream, with `TLS.close()` on the write side also calling shutdown on the underlying stream (thus assuming other users of the underlying stream are not permitted), while receiving EOF on the read side leaves the underlying channel open. These inconsistencies are left for a later person to resolve, if the extra functionality is needed (as described in nodejs#35904). The current goal here is to the fix the occasional CI exceptions depending on the timing of these kernel messages through the TCP stack. Refs: libuv/libuv#3036 Refs: nodejs#35904 Closes: nodejs#35946 Co-authored-by: Momtchil Momtchev <momtchil@momtchev.com>
Since I submitted the fix to nodejs for their missing error check 6 months ago (nodejs/node#36111), I'd like to merge this now. Anyone willing to review? |
bump. This fixes some connection management issues on Windows, and aligns the code with the correct behaviors that we are doing unix. Anyone willing to approve this? Thanks! |
bump |
bump??? |
daily bump? |
1 similar comment
daily bump? |
daily bump? I've got to stay ahead of stalebot, since it seems pretty eager to close issues that were being actively worked on but are lacking reviewer approvals, like this one. |
daily bump? I'm content with just a SGTM or rubber stamp too, since if there are any additional issues discovered later, I will be around to work on them. Failing that, I suppose I'll merge next week, as I'd like to make a new release. |
daily bump? |
daily bump? I'll plan to merge in a couple days, then start the release process |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SGTM
if (socket != tcp->socket) { | ||
if (reading) | ||
CancelIoEx((HANDLE) socket, &tcp->read_req.u.io.overlapped); | ||
if (writing) | ||
CancelIo((HANDLE) socket); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for my ignorance but, when can this happen?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
socket is the underlying (real) resource while tcp->socket is the user handle (potentially an LSP). It is a deprecated feature in the Window's kernel that these can be different (since it is about as much of an awkward design as it sounds), but still may show up in some ancient firewall products. Those ancient products often may not implement the Windows IO stack correctly either, so we propagate the CancelIO explicitly, rather than relying on them to know about this "modern" functionality.
@@ -203,7 +203,7 @@ static void on_read(uv_stream_t* handle, | |||
/* Make sure that the expected data is correctly multiplexed. */ | |||
ASSERT_MEM_EQ("hello\n", buf->base, nread); | |||
|
|||
outbuf = uv_buf_init("world\n", 6); | |||
outbuf = uv_buf_init("foobar\n", 7); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why this change?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed this because I wanted the read and writes to be different lengths ("hello" and "world" were previously the same number of bytes)
RFC 5246 section-7.2.1 requires that the implementation must immediately stop reading from the stream, as it is no longer TLS-encrypted. The underlying stream is permitted to still pump events (and errors) to other users, but those are now unencrypted, so we should not process them here. But therefore, we do not want to stop the underlying stream, as there could be another user of it, but we do need to remove ourselves as a listener. Per TLS v1.2, we should have also destroy the TLS state entirely here (including the writing side), but this was revised in TLS v1.3 to permit the stream to continue to flush output. There appears to be some inconsistencies in the way nodejs handles ownership of the underlying stream, with `TLS.close()` on the write side also calling shutdown on the underlying stream (thus assuming other users of the underlying stream are not permitted), while receiving EOF on the read side leaves the underlying channel open. These inconsistencies are left for a later person to resolve, if the extra functionality is needed (as described in nodejs#35904). The current goal here is to the fix the occasional CI exceptions depending on the timing of these kernel messages through the TCP stack. Refs: libuv/libuv#3036 Refs: nodejs#35904 Closes: nodejs#35946 Co-authored-by: Momtchil Momtchev <momtchil@momtchev.com>
RFC 5246 section-7.2.1 requires that the implementation must immediately stop reading from the stream, as it is no longer TLS-encrypted. The underlying stream is permitted to still pump events (and errors) to other users, but those are now unencrypted, so we should not process them here. But therefore, we do not want to stop the underlying stream, as there could be another user of it, but we do need to remove ourselves as a listener. Per TLS v1.2, we should have also destroy the TLS state entirely here (including the writing side), but this was revised in TLS v1.3 to permit the stream to continue to flush output. There appears to be some inconsistencies in the way nodejs handles ownership of the underlying stream, with `TLS.close()` on the write side also calling shutdown on the underlying stream (thus assuming other users of the underlying stream are not permitted), while receiving EOF on the read side leaves the underlying channel open. These inconsistencies are left for a later person to resolve, if the extra functionality is needed (as described in #35904). The current goal here is to the fix the occasional CI exceptions depending on the timing of these kernel messages through the TCP stack. PR-URL: #36111 Fixes: #35946 Refs: libuv/libuv#3036 Refs: #35904 Co-authored-by: Momtchil Momtchev <momtchil@momtchev.com> Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
RFC 5246 section-7.2.1 requires that the implementation must immediately stop reading from the stream, as it is no longer TLS-encrypted. The underlying stream is permitted to still pump events (and errors) to other users, but those are now unencrypted, so we should not process them here. But therefore, we do not want to stop the underlying stream, as there could be another user of it, but we do need to remove ourselves as a listener. Per TLS v1.2, we should have also destroy the TLS state entirely here (including the writing side), but this was revised in TLS v1.3 to permit the stream to continue to flush output. There appears to be some inconsistencies in the way nodejs handles ownership of the underlying stream, with `TLS.close()` on the write side also calling shutdown on the underlying stream (thus assuming other users of the underlying stream are not permitted), while receiving EOF on the read side leaves the underlying channel open. These inconsistencies are left for a later person to resolve, if the extra functionality is needed (as described in #35904). The current goal here is to the fix the occasional CI exceptions depending on the timing of these kernel messages through the TCP stack. PR-URL: #36111 Fixes: #35946 Refs: libuv/libuv#3036 Refs: #35904 Co-authored-by: Momtchil Momtchev <momtchil@momtchev.com> Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
RFC 5246 section-7.2.1 requires that the implementation must immediately stop reading from the stream, as it is no longer TLS-encrypted. The underlying stream is permitted to still pump events (and errors) to other users, but those are now unencrypted, so we should not process them here. But therefore, we do not want to stop the underlying stream, as there could be another user of it, but we do need to remove ourselves as a listener. Per TLS v1.2, we should have also destroy the TLS state entirely here (including the writing side), but this was revised in TLS v1.3 to permit the stream to continue to flush output. There appears to be some inconsistencies in the way nodejs handles ownership of the underlying stream, with `TLS.close()` on the write side also calling shutdown on the underlying stream (thus assuming other users of the underlying stream are not permitted), while receiving EOF on the read side leaves the underlying channel open. These inconsistencies are left for a later person to resolve, if the extra functionality is needed (as described in #35904). The current goal here is to the fix the occasional CI exceptions depending on the timing of these kernel messages through the TCP stack. PR-URL: #36111 Fixes: #35946 Refs: libuv/libuv#3036 Refs: #35904 Co-authored-by: Momtchil Momtchev <momtchil@momtchev.com> Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
RFC 5246 section-7.2.1 requires that the implementation must immediately stop reading from the stream, as it is no longer TLS-encrypted. The underlying stream is permitted to still pump events (and errors) to other users, but those are now unencrypted, so we should not process them here. But therefore, we do not want to stop the underlying stream, as there could be another user of it, but we do need to remove ourselves as a listener. Per TLS v1.2, we should have also destroy the TLS state entirely here (including the writing side), but this was revised in TLS v1.3 to permit the stream to continue to flush output. There appears to be some inconsistencies in the way nodejs handles ownership of the underlying stream, with `TLS.close()` on the write side also calling shutdown on the underlying stream (thus assuming other users of the underlying stream are not permitted), while receiving EOF on the read side leaves the underlying channel open. These inconsistencies are left for a later person to resolve, if the extra functionality is needed (as described in #35904). The current goal here is to the fix the occasional CI exceptions depending on the timing of these kernel messages through the TCP stack. PR-URL: #36111 Fixes: #35946 Refs: libuv/libuv#3036 Refs: #35904 Co-authored-by: Momtchil Momtchev <momtchil@momtchev.com> Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
RFC 5246 section-7.2.1 requires that the implementation must immediately stop reading from the stream, as it is no longer TLS-encrypted. The underlying stream is permitted to still pump events (and errors) to other users, but those are now unencrypted, so we should not process them here. But therefore, we do not want to stop the underlying stream, as there could be another user of it, but we do need to remove ourselves as a listener. Per TLS v1.2, we should have also destroy the TLS state entirely here (including the writing side), but this was revised in TLS v1.3 to permit the stream to continue to flush output. There appears to be some inconsistencies in the way nodejs handles ownership of the underlying stream, with `TLS.close()` on the write side also calling shutdown on the underlying stream (thus assuming other users of the underlying stream are not permitted), while receiving EOF on the read side leaves the underlying channel open. These inconsistencies are left for a later person to resolve, if the extra functionality is needed (as described in #35904). The current goal here is to the fix the occasional CI exceptions depending on the timing of these kernel messages through the TCP stack. PR-URL: #36111 Fixes: #35946 Refs: libuv/libuv#3036 Refs: #35904 Co-authored-by: Momtchil Momtchev <momtchil@momtchev.com> Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
RFC 5246 section-7.2.1 requires that the implementation must immediately stop reading from the stream, as it is no longer TLS-encrypted. The underlying stream is permitted to still pump events (and errors) to other users, but those are now unencrypted, so we should not process them here. But therefore, we do not want to stop the underlying stream, as there could be another user of it, but we do need to remove ourselves as a listener. Per TLS v1.2, we should have also destroy the TLS state entirely here (including the writing side), but this was revised in TLS v1.3 to permit the stream to continue to flush output. There appears to be some inconsistencies in the way nodejs handles ownership of the underlying stream, with `TLS.close()` on the write side also calling shutdown on the underlying stream (thus assuming other users of the underlying stream are not permitted), while receiving EOF on the read side leaves the underlying channel open. These inconsistencies are left for a later person to resolve, if the extra functionality is needed (as described in nodejs#35904). The current goal here is to the fix the occasional CI exceptions depending on the timing of these kernel messages through the TCP stack. PR-URL: nodejs#36111 Fixes: nodejs#35946 Refs: libuv/libuv#3036 Refs: nodejs#35904 Co-authored-by: Momtchil Momtchev <momtchil@momtchev.com> Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
RFC 5246 section-7.2.1 requires that the implementation must immediately stop reading from the stream, as it is no longer TLS-encrypted. The underlying stream is permitted to still pump events (and errors) to other users, but those are now unencrypted, so we should not process them here. But therefore, we do not want to stop the underlying stream, as there could be another user of it, but we do need to remove ourselves as a listener. Per TLS v1.2, we should have also destroy the TLS state entirely here (including the writing side), but this was revised in TLS v1.3 to permit the stream to continue to flush output. There appears to be some inconsistencies in the way nodejs handles ownership of the underlying stream, with `TLS.close()` on the write side also calling shutdown on the underlying stream (thus assuming other users of the underlying stream are not permitted), while receiving EOF on the read side leaves the underlying channel open. These inconsistencies are left for a later person to resolve, if the extra functionality is needed (as described in #35904). The current goal here is to the fix the occasional CI exceptions depending on the timing of these kernel messages through the TCP stack. PR-URL: #36111 Fixes: #35946 Refs: libuv/libuv#3036 Refs: #35904 Co-authored-by: Momtchil Momtchev <momtchil@momtchev.com> Reviewed-By: Robert Nagy <ronagy@icloud.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
99eb736#diff-3f206469017f9748c5a5247f9ff4e99ba43f95a8b0e7ee4a946dbdcc2038a100R1487 Hi @vtjnash ! |
Yes, Windows may be violating the TCP spec here, but that spec doesn't say much about the intended behavior of threads. As you see in that part of the diff, we try to instruct Windows not to send the RST packet, but it looks like we don't execute that logic after Actually, that is probably a fairly serious existing bug. It appears we may need to more closely track the lifetime of that pointer that we have passed into the pointer, and make sure we are not going to finalize the uv_close callback until we have confirmation the kernel has stopped tracking that pointer. We have called |
Would you be interested in trying to make a PR to manage those state transitions more accurately? |
@vtjnash |
The main change is to use |
This is an attempt to fix some resource management issues on Windows. Win32 sockets have an issue where it sends an RST packet if there is an outstanding overlapped calls. We can avoid that by being certain to explicitly cancel our read and write requests first. This also removes some conditional cleanup code, since we might as well clean it up eagerly (like unix). Otherwise, it looks to me like these might cause the accept callbacks to be run after the endgame had freed the memory for them. The comment here seems mixed up between send and recv buffers. The default behavior on calling `closesocket` is already to do a graceful shutdown (see https://docs.microsoft.com/en-us/windows/win32/api/winsock/nf-winsock-closesocket with default l_onoff=zero) if it is the last open handle. The expected behavior if there are pending reads in flight is to send an RST packet, notifying the client that the server connection was destroyed before acknowledging the EOF. Additionally, we need to cancel writes explicitly: we need to notify Win32 that it is okay to cancel these writes (so it doesn't also generate an RST packet on the wire). Refs: libuv#3035 Refs: nodejs/node#35946 Refs: nodejs/node#35904 Fixes: libuv#3034 PR-URL: libuv#3036 Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com>
This is an attempt to fix some TCP resource management issues on Windows and avoid a bug where it sends a RST packet that is unwarranted. The Win32 sockets library appear to have a bug where it sends an RST packet if there is an outstanding overlapped WSARecv call, even if no data is received. This would be akin to epoll/kevent sending RST if you close the socket while it was still registered—and so our fix is similar to the existing unix code: calling
CancelIOEx
on Windows is akin to callinguv__io_stop
on unix. Thus we can avoid the observed problem by being certain to explicitly cancel our read request first, and not just buried in unlikely conditional cases. If this approach fails for people (this capability was only added in Vista), we could also switch to usingselect
on a background thread to pump these events, but it would be nice to avoid doing that extra coding work.This also removes some conditional cleanup code, since we might as well clean it up eagerly (like unix). Otherwise, it looks to me like these might cause the accept callbacks to be run after the endgame had freed the memory for them (or not at all)?
The comment here seems to have gotten slightly mixed up over time between send and recv buffers and the expected vs. incorrect failures this code encountered. The default behavior on calling
closesocket
is already to do a graceful shutdown (see https://docs.microsoft.com/en-us/windows/win32/api/winsock/nf-winsock-closesocketwith default l_onoff=zero) if it is the last open handle. So the expected behavior if there are pending reads in flight is to send an RST packet, notifying the client that the server connection was destroyed before acknowledging the EOF. But instead we also observe an RST packet being sent if even though there are no messages in flight.
The only test that I noted already exercises this failure case (after removing the conditionals, so that we aren't injecting a
uv_shutdown
call anymore and the bug is detectable) isipc_tcp_connection
.This PR is passing on CI ✅: https://ci.nodejs.org/job/libuv-test-commit-windows-cmake/273/