os: File implicitly closed when garbage-collected due to runtime finalizer but this behavior is not documented

[PureWhiteWu]

What version of Go are you using (go version)?

$ go version
go 1.15.2

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

Linux and macOS.

What did you do?

Code: https://play.golang.org/p/YmHIIc7sha4

Note: This does not reproduce in go playground, but can reproduce on my mac/linux.

What did you expect to see?

No panic.

What did you see instead?

panic: bad file descriptor

Reason

I found that this is because there's an implicitly close call when the file object is garbage-collected.

In os/file_unix.go:

func newFile(fd uintptr, name string, kind newFileKind) *File {
	...

	runtime.SetFinalizer(f.file, (*file).close)
	return f
}

What do I expect

I know there's thousands of method to avoid being garbage-collected such as runtime.KeepAlive, but I think this is unreasonable, either this behaviour should be removed or the file object should not be garbage-collected.

If a user forgets to call file.Close, the memory should be leaking.

Also, this behaviour isn't documented anywhere.

[odeke-em]

Thank you for filing this issue @PureWhiteWu!

Unfortunately we can’t remove this behavior, but perhaps we can document it. I worry thought that documenting that files have finalizers might encourage callous behavior in which folks just rely on garbage collection to sort out for them file close problems which will cause undeterministic leaks and bugs. I understand that in between invoking syscall.Write(fd, payload) and the end, a garbage collection cycle might run, but unfortunately that program has a leak that then depends on undefined behavior. How often have you seen such code?

[PureWhiteWu]

@odeke-em Thanks for your reply!

This is a bug we encountered in our real-world program, in which we are doing something like this(simplified version):

// open the file
file, err := os.OpenFile("x", os.O_RDWR, 0666)
...
// get its fd
fd := int(file.Fd())
// do some work using syscall directly forever
for {
    data := getData()
    syscall.Write(fd, data)
    ....
}
syscall.Close(fd)

And we got -9 (bad fd) after some time.

This bug is hard to debug, so I think something should be changed to prevent this.

[odeke-em]

Gotcha, I understand and thank you for the citation. For some remedies:
a) defer file.Close()
b) runtime.KeepAlive(file) although that would mask the problem with the missing file.Close()

Some docs updates to state that a file might be garbage collected perhaps could have helped you in this case then, if we stated the need to keep the file alive for the course of usage or to ensure it is closed, but the finalizer is an implementation/system specific detail. It is definitely inconvenient to try to go down many rabbit holes trying to figure out why a bad fd error occurred.

[odeke-em]

/cc @ianlancetaylor

[cespare]

Also, this behaviour isn't documented anywhere.

It is documented. The os.File.Fd documentation says, in part,

The file descriptor is valid only until f.Close is called or f is garbage collected.

It sounds like you already understand the problem and the workarounds from your original description.

This bug is hard to debug

Perhaps, though the crash seems fairly hard to miss.

In any case, if you're using direct syscalls to write files rather than the usual mechanisms, you should be prepared for some amount of tricky debugging.

[PureWhiteWu]

@cespare Thanks very much, I missed this part.

Perhaps, though the crash seems fairly hard to miss.

In some cases, the for loop may quit before gc, so this may occur occasionally. I think this is really hard to debug for users, because users are not expected to understand how gc works and when will gc execute.

@odeke-em Thanks for your reply, I think defer file.Close() makes sense in this situation.

But maybe we can make things more clearly, such as adding an extra line:

// File represents an open file descriptor.
// Note: File.Close is automatically called if it hasn't been when it is garbage collected. 
type File struct {
	*file // os specific
}

[beoran]

Related issue: #34810

[ianlancetaylor]

The problem here is the call of the Fd method, and the Fd method is the place to document any possible issues with the finalizer. As @cespare points out, the Fd docs already say "The file descriptor is valid only until f.Close is called or f is garbage collected." Can we make that clearer? More prominent?

I do not think we should add any documentation to os.File because there is no problem with ordinary use of os.File. The problem only arises for programs that call the Fd method, which is a rare case. Most programs that need to all Fd should instead be using syscall.Open or the SyscallConn method. Perhaps that we what we should say in the Fd docs.

[andrius4669]

Can we make that clearer?

Perhaps runtime.KeepAlive(f) could be suggested there, to prevent f from being garbage collected.
Currently "or f is garbage collected" doesn't look important enough because many don't know when GC can hit them (importantly, it can get GC'd even before f goes out of scope).
On the other hand if f.Close() or any other method gets called it'd be kept alive too, some maybe these should be documented as well to not cause needless use of runtime.KeepAlive(f).
Maybe one more sentence like "Method call like f.Close() or runtime.KeepAlive(f) can be used to prevent unexpected closure of file descriptor".
I agree with suggestion to use syscall.Open instead too.
Either way documentation could be expanded there, and probably should, but that's just my 2c.

[beoran]

I think that the File.Fd method should be deprecated in favor of RawConn.Control and perhaps a new RawConn.MultiControl. We cannot remove File.Fd due to the Go 1 promise, but it is very hard to use correctly. It is what I would call a "foot gun".