os: add a semaphore in front of the filesystem?

[bradfitz]

I'd like to investigate putting a semaphore in front of the pkg/os filesystem
operations, so massive numbers of goroutines attempting filesystem I/O don't create
threads each and make the operating system cry.

There are deadlock concerns, but perhaps we can do it per filesystem id, or learn the
limits, etc.

[dvyukov]

Comment 1:

What filesystem operations do you mean? There is an opened bug to make Read/Write async.

[bradfitz]

Comment 2:

All of them. Open, Close, Stat, LStat, Read, Write, Link, ...

[dvyukov]

Comment 3:

A common solution to prevent deadlocks is to throttle threads instead of bound them.
That is, you create threads instantly up to some limit; then you start throttling them a
bit; the more threads the more aggressive is the throttling.
I think that scalability will be important here. That is, if we cap fast cached
(in-memory) filesystem operations on a global chan-based semaphore, it can significantly
affect performance.

[rsc]

Comment 4:

Too late for 1.4. Marking Release-None until there's a plan.

Labels changed: added release-none, removed release-go1.4.

[bcmills]

Isn't it generally up to users to apply reasonable flow-control to limit expensive resources and/or bound the footprint of their program?

What makes "using a thread for a filesystem call" significantly different from "using a bunch of live memory in the Go heap" (which we are not - and, I would argue, should not be - proposing to guard with a semaphore or otherwise restrict artificially)?

[rsc]

The operating system no longer cries, as in Brad's earlier message, because we added a fixed limit on the number of threads in the program. Now the Go program crashes instead, which is better than taking down the OS.

I agree with Bryan that we probably don't want to do more here. We chose to crash on thread count instead of trying to do some kind of thread limit exactly for concerns about deadlocks, and those concerns would exist for any kind of file system semaphore as well.

Unless Brad objects, I think we should probably close this.

[bradfitz]

I think I still object. This was the first problem I hit with Go and I continue to hit it. I'd hope we can do better here.

The problem is that with "reasonable" flow control, reasonable often isn't known and implementations of interfaces can be selected at runtime with totally different properties. Camlistore uses a package to do a VFS where different paths are mounted with different file implementations (like google3 files). What's reasonable for /memfile/foo is very different from what's reasonable from /mnt/my-ext3-filesystem/.