Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
os: race between File.Close, File.Write, and other Opens #7970
If you have a goroutine calling f.Close at the same time another calls f.Write, it can happen that the f.Close happens, then an unrelated os.Open reuses the fd, then the f.Write writes to the wrong fd. The timing would have to be just right, since Close does set f.fd = -1, but it could happen. We could decide this is not worth protecting against, or we could introduce an rwlock on the File that must be rlocked across all fd-using method implementations and wlocked by Close. Investigate for 1.4. Related to issue #5792.
Using an RWMutex for file.fd will mean that if a Read hangs in the kernel waiting for data, a Close on the same file will hang waiting for the read to complete. I don't think that is desirable.
I think we should postpone this until issue #6817 is fixed.
#6817 might help with this, but I'm not convinced it is always available in all cases. Maybe I'm wrong.
We might be able to just keep living with this. Certainly we don't hear about it much.
If we need to fix things, and AIO is not good enough (and I suspect it's not), one possibility would be to create an unusable fd somehow (for example call socket without binding it anywhere, or maybe call pipe and close the read end and use the write fd) and instead of using Close(fd) use Dup(badfd, fd). That would close the original fd and make f.fd unusable, without leaving the fd available for reuse immediately. The Dup'ed copy of badfd could be properly closed when we know all the i/o has stopped. The problem with this, I suspect, is that Close of a file open for writing can return information about write errors that are as yet unreported, and I suspect Dup onto an existing fd does not return those errors.
Like I said, this is very difficult to fix. I expect that most other languages just live with this the same as Go. I don't know that there's any fix that's not worse than the problem.
I'm not really sure which languages permit concurrent read/close access in any way. I think for most languages it is undefined behavior. For kernel file descriptors it is permitted, but of course the same race is possible.
In the net package we handle the same problem with the code in net/fd_mutex.go. As far as I can see the same approach would work in the os package. This would mean extending the problem of errClosing (#4373) to the os package.
The fix for issue #6817 probably requires some set of helper threads for I/O. AIO is not really possible on GNU/Linux systems. There is no separate supported kernel interface for it; the aio_read functions and friends are implemented by simply using a thread pool for I/O.
Continuing my Dup comment from above, @aclements points out taht we can avoid losing the error from syscall.Close by doing:
Of course, this would mean that if you run out of file descriptors, it becomes impossible to close a file. We could do this only for fd's with pending operations, of course, but then there's a little-exercised code path lurking.
This seems extremely low priority. Going to move to Unplanned until that changes.
CL 31148 added code to protect again simultaneous calls to Close and Wait when using the standard input pipe, to fix the race condition described in issue #9307. That issue is a special case of the race between Close and Write described by issue #7970. Since issue #7970 was not fixed, CL 31148 fixed the problem specific to os/exec. Since then, issue #7970 has been fixed, so the specific fix in os/exec is no longer necessary. Remove it, effectively reverting CL 31148 and followup CL 33298. Updates #7970 Updates #9307 Updates #17647 Change-Id: Ic0b62569cb0aba44b32153cf5f9632bd1f1b411a Reviewed-on: https://go-review.googlesource.com/65490 Run-TryBot: Ian Lance Taylor <firstname.lastname@example.org> Reviewed-by: Daniel Martí <email@example.com> Reviewed-by: Miguel Bernabeu <firstname.lastname@example.org> Reviewed-by: Russ Cox <email@example.com> Reviewed-by: Joe Tsai <firstname.lastname@example.org> TryBot-Result: Gobot Gobot <email@example.com>
This permits the program to reliably know that when the Close method returns, the descriptor has definitely been closed. This matters at least for listeners. Fixes #21856 Updates #7970 Change-Id: I1fd0cfd2333649e6e67c6ae956e19fdff3a35a83 Reviewed-on: https://go-review.googlesource.com/66150 Run-TryBot: Ian Lance Taylor <firstname.lastname@example.org> TryBot-Result: Gobot Gobot <email@example.com> Reviewed-by: Joe Tsai <firstname.lastname@example.org>
os.NewFile doesn't put the fd into non-blocking mode. In most cases, an *os.File returned by os.NewFile is in blocking mode. Updates #7970 Updates #21856 Updates #23111 Change-Id: Iab08432e41f7ac1b5e25aaa8855d478adb7f98ed Reviewed-on: https://go-review.googlesource.com/83995 Reviewed-by: Ian Lance Taylor <email@example.com>