-
-
Notifications
You must be signed in to change notification settings - Fork 29.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
os: support blocking functions on Windows #101881
Comments
When WinAPI When WinAPI Footnotes
|
Usage in the io module and test cases will have to be examined for assumptions that partial writes are supported up to the available space in the pipe. On Windows, it's all or nothing in non-blocking mode. If the pipe has 1024 bytes available, then trying to write 2048 bytes will write nothing at all and fail. |
Here's an idea to get around the annoying behavior in non-blocking mode. Modify For example, append the following code to the first // A write that exceeds the available size of a pipe never succeeds in
// non-blocking mode. Limiting writes to the pipe size in this case allows
// a buffered write to succeed eventually, as the pipe is read.
DWORD mode, pipe_size;
HANDLE hfile = _Py_get_osfhandle(fd);
if (hfile == INVALID_HANDLE_VALUE) {
return -1;
}
if (gil_held) {
Py_BEGIN_ALLOW_THREADS
if (GetFileType(hfile) == FILE_TYPE_PIPE &&
GetNamedPipeHandleStateW(hfile, &mode,
NULL, NULL, NULL, NULL, 0) &&
mode & PIPE_NOWAIT)
{
// GetNamedPipeInfo() requires FILE_READ_ATTRIBUTES access.
// CreatePipe() includes this access for the write handle.
if (!GetNamedPipeInfo(hfile, NULL, NULL, &pipe_size, NULL)) {
pipe_size = 4096;
}
if (count > pipe_size) {
count = pipe_size;
}
}
Py_END_ALLOW_THREADS
}
else {
if (GetFileType(hfile) == FILE_TYPE_PIPE &&
GetNamedPipeHandleStateW(hfile, &mode,
NULL, NULL, NULL, NULL, 0) &&
mode & PIPE_NOWAIT)
{
if (!GetNamedPipeInfo(hfile, NULL, NULL, &pipe_size, NULL)) {
pipe_size = 4096;
}
if (count > pipe_size) {
count = pipe_size;
}
}
} |
The documentation in "Doc/library/os.rst" needs to be updated to indicate the new Windows support in 3.12, i.e. For example:
|
Use the GetNamedPipeHandleState and SetNamedPipeHandleState Win32 API functions to add support for os.get_blocking and os.set_blocking.
done, anything else needed? |
Use the GetNamedPipeHandleState and SetNamedPipeHandleState Win32 API functions to add support for os.get_blocking and os.set_blocking.
Use the GetNamedPipeHandleState and SetNamedPipeHandleState Win32 API functions to add support for os.get_blocking and os.set_blocking.
…es (GH-101882) * fileutils: handle non-blocking pipe IO on Windows Handle erroring operations on non-blocking pipes by reading the _doserrno code. Limit writes on non-blocking pipes that are too large. * Support blocking functions on Windows Use the GetNamedPipeHandleState and SetNamedPipeHandleState Win32 API functions to add support for os.get_blocking and os.set_blocking.
Since c6d3cce (pipe_command(): handle ENOSPC when writing to a pipe, 2022-08-17), one `write()` call that results in an `errno` value `ENOSPC` (which typically indicates out of disk space, which makes little sense in the context of a pipe) is treated the same as `EAGAIN`. However, contrary to expectations, as diagnosed in python/cpython#101881 (comment), when writing to a non-blocking pipe on Windows, an `errno` value of `ENOSPC` means something else: the write failed because more data was provided than the internal pipe buffer can handle. Which can be somewhat surprising, considering that `write()` is allowed to write less than the specified amount, e.g. by writing only as much as fits in that buffer. Let's handle this by manually detecting when an `ENOSPC` indicates that a pipe's buffer is smaller than what should be written, and re-try using the buffer size as `size` parameter. It would be plausible to try writing the entire buffer in a loop, feeding pipe buffer-sized chunks, but experiments show that trying to write more than one buffer-sized chunk right after that will immediately fail because the buffer is unlikely to be drained as fast as `write()` could write again. Which means that the logic that determines the pipe's buffer size unfortunately has to be run potentially many times when writing large amounts of data to a non-blocking pipe, as there is no elegant way to cache that information between `write()` calls. This fix is required to let t3701.60 (handle very large filtered diff) pass with the MSYS2 runtime provided by the MSYS2 project. This patch is not required with Git for Windows' variant of the MSYS2 runtime only because that Git for Windows added an ugly work-around specifically to avoid a hang in that test case. The diff is slightly chatty because it extends an already-existing conditional that special-cases a _different_ `errno` value for pipes, and because this patch needs to account for the fact that `_get_osfhandle()` potentially overwrites `errno`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Since c6d3cce (pipe_command(): handle ENOSPC when writing to a pipe, 2022-08-17), one `write()` call that results in an `errno` value `ENOSPC` (which typically indicates out of disk space, which makes little sense in the context of a pipe) is treated the same as `EAGAIN`. However, contrary to expectations, as diagnosed in python/cpython#101881 (comment), when writing to a non-blocking pipe on Windows, an `errno` value of `ENOSPC` means something else: the write _fails_. Completely. Because more data was provided than the internal pipe buffer can handle. Somewhat surprising, considering that `write()` is allowed to write less than the specified amount, e.g. by writing only as much as fits in that buffer. But it doesn't, it writes no byte at all in that instance. Let's handle this by manually detecting when an `ENOSPC` indicates that a pipe's buffer is smaller than what needs to be written, and re-try using the pipe's buffer size as `size` parameter. It would be plausible to try writing the entire buffer in a loop, feeding pipe buffer-sized chunks, but experiments show that trying to write more than one buffer-sized chunk right after that will immediately fail because the buffer is unlikely to be drained as fast as `write()` could write again. And the whole point of a non-blocking pipe is to be non-blocking. Which means that the logic that determines the pipe's buffer size unfortunately has to be run potentially many times when writing large amounts of data to a non-blocking pipe, as there is no elegant way to cache that information between `write()` calls. It's the best we can do, though, so it has to be good enough. This fix is required to let t3701.60 (handle very large filtered diff) pass with the MSYS2 runtime provided by the MSYS2 project: Without this patch, the failed write would result in an infinite loop. This patch is not required with Git for Windows' variant of the MSYS2 runtime only because Git for Windows added an ugly work-around specifically to avoid a hang in that test case. The diff is slightly chatty because it extends an already-existing conditional that special-cases a _different_ `errno` value for pipes, and because this patch needs to account for the fact that `_get_osfhandle()` potentially overwrites `errno`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Since c6d3cce (pipe_command(): handle ENOSPC when writing to a pipe, 2022-08-17), one `write()` call that results in an `errno` value `ENOSPC` (which typically indicates out of disk space, which makes little sense in the context of a pipe) is treated the same as `EAGAIN`. However, contrary to expectations, as diagnosed in python/cpython#101881 (comment), when writing to a non-blocking pipe on Windows, an `errno` value of `ENOSPC` means something else: the write _fails_. Completely. Because more data was provided than the internal pipe buffer can handle. Somewhat surprising, considering that `write()` is allowed to write less than the specified amount, e.g. by writing only as much as fits in that buffer. But it doesn't, it writes no byte at all in that instance. Let's handle this by manually detecting when an `ENOSPC` indicates that a pipe's buffer is smaller than what needs to be written, and re-try using the pipe's buffer size as `size` parameter. It would be plausible to try writing the entire buffer in a loop, feeding pipe buffer-sized chunks, but experiments show that trying to write more than one buffer-sized chunk right after that will immediately fail because the buffer is unlikely to be drained as fast as `write()` could write again. And the whole point of a non-blocking pipe is to be non-blocking. Which means that the logic that determines the pipe's buffer size unfortunately has to be run potentially many times when writing large amounts of data to a non-blocking pipe, as there is no elegant way to cache that information between `write()` calls. It's the best we can do, though, so it has to be good enough. This fix is required to let t3701.60 (handle very large filtered diff) pass with the MSYS2 runtime provided by the MSYS2 project: Without this patch, the failed write would result in an infinite loop. This patch is not required with Git for Windows' variant of the MSYS2 runtime only because Git for Windows added an ugly work-around specifically to avoid a hang in that test case. The diff is slightly chatty because it extends an already-existing conditional that special-cases a _different_ `errno` value for pipes, and because this patch needs to account for the fact that `_get_osfhandle()` potentially overwrites `errno`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Since c6d3cce (pipe_command(): handle ENOSPC when writing to a pipe, 2022-08-17), one `write()` call that results in an `errno` value `ENOSPC` (which typically indicates out of disk space, which makes little sense in the context of a pipe) is treated the same as `EAGAIN`. However, contrary to expectations, as diagnosed in python/cpython#101881 (comment), when writing to a non-blocking pipe on Windows, an `errno` value of `ENOSPC` means something else: the write _fails_. Completely. Because more data was provided than the internal pipe buffer can handle. Somewhat surprising, considering that `write()` is allowed to write less than the specified amount, e.g. by writing only as much as fits in that buffer. But it doesn't, it writes no byte at all in that instance. Let's handle this by manually detecting when an `ENOSPC` indicates that a pipe's buffer is smaller than what needs to be written, and re-try using the pipe's buffer size as `size` parameter. It would be plausible to try writing the entire buffer in a loop, feeding pipe buffer-sized chunks, but experiments show that trying to write more than one buffer-sized chunk right after that will immediately fail because the buffer is unlikely to be drained as fast as `write()` could write again. And the whole point of a non-blocking pipe is to be non-blocking. Which means that the logic that determines the pipe's buffer size unfortunately has to be run potentially many times when writing large amounts of data to a non-blocking pipe, as there is no elegant way to cache that information between `write()` calls. It's the best we can do, though, so it has to be good enough. This fix is required to let t3701.60 (handle very large filtered diff) pass with the MSYS2 runtime provided by the MSYS2 project: Without this patch, the failed write would result in an infinite loop. This patch is not required with Git for Windows' variant of the MSYS2 runtime only because Git for Windows added an ugly work-around specifically to avoid a hang in that test case. The diff is slightly chatty because it extends an already-existing conditional that special-cases a _different_ `errno` value for pipes, and because this patch needs to account for the fact that `_get_osfhandle()` potentially overwrites `errno`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
The os.get_blocking and os.set_blocking functions are only currently supported on Unix.
Linked PRs
The text was updated successfully, but these errors were encountered: