New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

os: support runtime poller with os.File on Windows #19098

Open
ianlancetaylor opened this Issue Feb 15, 2017 · 5 comments

Comments

Projects
None yet
6 participants
@ianlancetaylor
Contributor

ianlancetaylor commented Feb 15, 2017

In order to use the runtime poller with os.File on Windows, the file or pipe will have to be opened with FILE_FLAG_OVERLAPPED. We shouldn't do that unconditionally unless we can remove it if the program calls the Fd method. Programs currently expect the Fd method to return a handle that uses ordinary synchronous I/O.

@ianlancetaylor ianlancetaylor added this to the Go1.9Maybe milestone Feb 15, 2017

@gopherbot

This comment has been minimized.

gopherbot commented Feb 15, 2017

CL https://golang.org/cl/36800 mentions this issue.

@rasky

This comment has been minimized.

Member

rasky commented Feb 15, 2017

ReOpenFile() can do just that, adding/removing FILE_FLAG_OVERLAPPED (https://msdn.microsoft.com/en-us/library/aa365497(VS.85).aspx)

gopherbot pushed a commit that referenced this issue Feb 15, 2017

os: use poller for file I/O
This changes the os package to use the runtime poller for file I/O
where possible. When a system call blocks on a pollable descriptor,
the goroutine will be blocked on the poller but the thread will be
released to run other goroutines. When using a non-pollable
descriptor, the os package will continue to use thread-blocking system
calls as before.

For example, on GNU/Linux, the runtime poller uses epoll. epoll does
not support ordinary disk files, so they will continue to use blocking
I/O as before. The poller will be used for pipes.

Since this means that the poller is used for many more programs, this
modifies the runtime to only block waiting for the poller if there is
some goroutine that is waiting on the poller. Otherwise, there is no
point, as the poller will never make any goroutine ready. This
preserves the runtime's current simple deadlock detection.

This seems to crash FreeBSD systems, so it is disabled on FreeBSD.
This is issue 19093.

Using the poller on Windows requires opening the file with
FILE_FLAG_OVERLAPPED. We should only do that if we can remove that
flag if the program calls the Fd method. This is issue 19098.

Update #6817.
Update #7903.
Update #15021.
Update #18507.
Update #19093.
Update #19098.

Change-Id: Ia5197dcefa7c6fbcca97d19a6f8621b2abcbb1fe
Reviewed-on: https://go-review.googlesource.com/36800
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
@polarina

This comment has been minimized.

Contributor

polarina commented Feb 15, 2017

Overlapped reads on Windows may run synchronously under some conditions. Files using NTFS compression can not be read asynchronously, for example.

See https://support.microsoft.com/en-us/help/156932/asynchronous-disk-i-o-appears-as-synchronous-on-windows

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Feb 15, 2017

Thanks. For our purposes it doesn't matter if I/O is sometimes synchronous. The effect will be that the I/O operation will tie up a thread. That is not ideal, but it is no different than any other blocking syscall.

@jstarks

This comment has been minimized.

jstarks commented May 11, 2017

There's no way to flip the bit on an existing file handle, and ReOpenFile will only work properly for real local files, only if the file does not have the delete on close bit set (I think), and only if they have been opened with compatible sharing flags. And of course Fd() uintptr does not provide a mechanism to return an error if something goes wrong.

I'm afraid this may be impossible to achieve with the current Windows API.

This is probably not such a shame for normal files opened for cached IO, although recent versions of Windows do perform cached reads asynchronously in some cases. But it certainly is limiting for named pipes, files opened with FILE_FLAG_NO_BUFFERING, and other Windows file handle types that reliably perform IO asynchronously.

Maybe this is something that we can improve in Windows. But even if we do, I think for Windows there would still be a lot of value in providing direct access to the runtime poller. There are several APIs that take OVERLAPPED structures that don't fit into os.File (ConnectNamedPipe, MergeVirtualDisk, DeviceIoControl, etc.), and it would be ideal to be able to use the existing poller to avoid blocking threads on these APIs.

That seems at odds with some of the discussion in #18507, unfortunately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment