os: document the hard limit of concurrent writes to *File changed in Go1.9

[kokizzu]

What version of Go are you using (go version)?

$ go version
go version go1.12 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/kyz/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/kyz/MEGA/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/go"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build034856733=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Running old code that works on go 1.8 but failed on go 1.12
https://stackoverflow.com/questions/41894046/

What did you expect to see?

no error as previous version of go 1.8

What did you see instead?

$ go build test.go ; for k in 5 50 500 5000 50000 500000; do echo -n $k; time ./test $k > /dev/null; done
5
CPU: 0.00s      Real: 0.00s     RAM: 3972KB
50
CPU: 0.33s      Real: 0.09s     RAM: 49672KB
500
CPU: 4.21s      Real: 0.95s     RAM: 521536KB
5000panic: too many concurrent operations on a single file or socket (max 1048575)

goroutine 3364987 [running]:
internal/poll.(*fdMutex).rwlock(0xc0000680c0, 0x0, 0x0)
        /usr/lib/go/src/internal/poll/fd_mutex.go:147 +0x13f
internal/poll.(*FD).writeLock(...)
        /usr/lib/go/src/internal/poll/fd_mutex.go:239
internal/poll.(*FD).Write(0xc0000680c0, 0xc0b28575d0, 0x8, 0x8, 0x0, 0x0, 0x0)
        /usr/lib/go/src/internal/poll/fd_unix.go:255 +0x4e
os.(*File).write(...)
        /usr/lib/go/src/os/file_unix.go:280
os.(*File).Write(0xc00000e018, 0xc0b28575d0, 0x8, 0x8, 0x0, 0xc0b2b79f60, 0x40890b)
        /usr/lib/go/src/os/file.go:145 +0x76
fmt.Fprintln(0x4d2300, 0xc00000e018, 0xc0b2b79fb0, 0x1, 0x1, 0x0, 0x0, 0x0)
        /usr/lib/go/src/fmt/print.go:266 +0x8b
fmt.Println(...)
        /usr/lib/go/src/fmt/print.go:275
main.main.func1(0xc000014130, 0x33583c)
        /tmp/test.go:24 +0xac
created by main.main
        /tmp/test.go:22 +0xe7
Command exited with non-zero status 2

[ianlancetaylor]

Yes, this did change in Go 1.9. There is now a limit on the number of concurrent writes that can be made to a single os.File. The limit is pretty high, as the error message says. If it makes a difference for a real program, you'll need to use a semaphore of some sort to restrict the number of concurrent writes.

[odeke-em]

@ianlancetaylor I don't think that we documented this change in Go1.9 release notes https://golang.org/doc/go1.9, perhaps this could be a Go1 compatibility violation?
Since we are no longer supporting Go1.9 (I could be mistaken though), how perhaps can we retroactively document this change?

[ianlancetaylor]

I don't think this is a Go 1 compatibility issue. There is also a limitation in 1.8: the number of threads that the OS will let you create. It's just a different path to hitting the limit.

I don't know if there is a sensible way to document this. We don't want to document a specific number; that is going to change depending on the implementation. I'm open to suggestions.

[odeke-em]

Thank you for the explanation @ianlancetaylor!

Perhaps we could document on os.File that

// Note: The maximum number of concurrent operations on a File is subject to OS/system
// specific resource limits. While the limit is very high, it would be useful to bound concurrent
// accesses to the File.

or because this was a natural progression anyways, in that there is a limit that can be reached I think that there is nothing that we can do. This is a very rare scenario as @kokizzu's has been the first report in 2 years.

[ianlancetaylor]

Seems reasonable to me. Thanks.

[gopherbot]

Change https://golang.org/cl/181841 mentions this issue: os: document File concurrent access resource limits