New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

os: add a semaphore in front of the filesystem? #7903

Open
bradfitz opened this Issue Apr 30, 2014 · 12 comments

Comments

Projects
None yet
5 participants
@bradfitz
Member

bradfitz commented Apr 30, 2014

I'd like to investigate putting a semaphore in front of the pkg/os filesystem
operations, so massive numbers of goroutines attempting filesystem I/O don't create
threads each and make the operating system cry.

There are deadlock concerns, but perhaps we can do it per filesystem id, or learn the
limits, etc.
@dvyukov

This comment has been minimized.

Member

dvyukov commented Apr 30, 2014

Comment 1:

What filesystem operations do you mean? There is an opened bug to make Read/Write async.
@bradfitz

This comment has been minimized.

Member

bradfitz commented Apr 30, 2014

Comment 2:

All of them. Open, Close, Stat, LStat, Read, Write, Link, ...
@dvyukov

This comment has been minimized.

Member

dvyukov commented Apr 30, 2014

Comment 3:

A common solution to prevent deadlocks is to throttle threads instead of bound them.
That is, you create threads instantly up to some limit; then you start throttling them a
bit; the more threads the more aggressive is the throttling.
I think that scalability will be important here. That is, if we cap fast cached
(in-memory) filesystem operations on a global chan-based semaphore, it can significantly
affect performance.
@rsc

This comment has been minimized.

Contributor

rsc commented Sep 16, 2014

Comment 4:

Too late for 1.4. Marking Release-None until there's a plan.

Labels changed: added release-none, removed release-go1.4.

@rsc rsc added this to the Unplanned milestone Apr 10, 2015

@rsc rsc removed release-none labels Apr 10, 2015

@bcmills

This comment has been minimized.

Member

bcmills commented Feb 18, 2016

Isn't it generally up to users to apply reasonable flow-control to limit expensive resources and/or bound the footprint of their program?

What makes "using a thread for a filesystem call" significantly different from "using a bunch of live memory in the Go heap" (which we are not - and, I would argue, should not be - proposing to guard with a semaphore or otherwise restrict artificially)?

@rsc

This comment has been minimized.

Contributor

rsc commented Feb 19, 2016

The operating system no longer cries, as in Brad's earlier message, because we added a fixed limit on the number of threads in the program. Now the Go program crashes instead, which is better than taking down the OS.

I agree with Bryan that we probably don't want to do more here. We chose to crash on thread count instead of trying to do some kind of thread limit exactly for concerns about deadlocks, and those concerns would exist for any kind of file system semaphore as well.

Unless Brad objects, I think we should probably close this.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Feb 19, 2016

I think I still object. This was the first problem I hit with Go and I continue to hit it. I'd hope we can do better here.

The problem is that with "reasonable" flow control, reasonable often isn't known and implementations of interfaces can be selected at runtime with totally different properties. Camlistore uses a package to do a VFS where different paths are mounted with different file implementations (like google3 files). What's reasonable for /memfile/foo is very different from what's reasonable from /mnt/my-ext3-filesystem/.

@rsc

This comment has been minimized.

Contributor

rsc commented Feb 19, 2016

OK, but a FS semaphore will cause deadlocks when someone does an FS operation that does block. That's not better.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Feb 19, 2016

I acknowledged the deadlock issue in the original issue text. @dvyukov discussed it too in "Comment 3".

Maybe the answer isn't a semaphore in the filesystem but instead a way to provide more information to programs creating work to tell them the current situation. We know how many goroutines there are, but can we cheaply answer how many goroutines are in system calls? Or how many threads there are? That might be enough.

@bcmills

This comment has been minimized.

Member

bcmills commented Feb 19, 2016

@bradfitz:

The problem is that with "reasonable" flow control, reasonable often isn't known

There are two or three directions you can go with that, but they're ~all application-side: there is no change you can make to the runtime that fixes the problem that "reasonable often isn't known". The problem is fundamentally one of balancing resource predictability against peak resource usage, and that depends very much on your execution environment: a reasonable "peak usage" on a small end-user desktop running many programs is fundamentally very different from a large server with strong resource isolation and/or only a few tasks.

and implementations of interfaces can be selected at runtime with totally different properties. Camlistore uses a package to do a VFS where different paths are mounted with different file implementations

That's an application design problem: do you want to maximize throughput, optimize for resource predictability, or both? If you want to optimize for predictability, then you need some kind of estimate of the usage that each implementation is potentially going to need at peak (not just what it is currently using), and then you can implement a global throttle where each implementation says "I'm going to need X resources" and the throttle blocks it until those resources are available. Presumably the /memfile implementation would use a different value for "X" than the ext3 one.

But that's all something you can only apply at the application layer: only you know what the peak resource consumption of each implementation is going to be like.


There is certainly a broad design space for "static analysis of peak resource consumption of a function call", but Go is not exactly designed to be amenable to that kind of analysis.

If you want to do the equivalent dynamic analysis, you basically only have three options: failure (the current behavior when we exhaust the thread limit), blocking (with the associated risks of deadlock and/or priority inversion), or forcible cancellation of pending work (e.g. causing an arbitrary goroutine to panic when we run out of resources, but most Go code is not written to be panic-safe and it's not clear how you choose which goroutine to kill without risking livelock anyway). None of those three options can be applied safely to arbitrary Go programs. All of those options would violate Go 1 compatibility: they could cause previously correct-but-resource-intensive programs to become "incorrect" programs.

The only viable solution for Go as it is today is for application programmers who care about resource footprints to carefully apply their own flow-control at the application layer.

@gopherbot

This comment has been minimized.

gopherbot commented Feb 10, 2017

CL https://golang.org/cl/36799 mentions this issue.

@gopherbot

This comment has been minimized.

gopherbot commented Feb 10, 2017

CL https://golang.org/cl/36800 mentions this issue.

gopherbot pushed a commit that referenced this issue Feb 13, 2017

net: refactor poller into new internal/poll package
This will make it possible to use the poller with the os package.

This is a lot of code movement but the behavior is intended to be
unchanged.

Update #6817.
Update #7903.
Update #15021.
Update #18507.

Change-Id: I1413685928017c32df5654ded73a2643820977ae
Reviewed-on: https://go-review.googlesource.com/36799
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: David Crawshaw <crawshaw@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>

gopherbot pushed a commit that referenced this issue Feb 15, 2017

os: use poller for file I/O
This changes the os package to use the runtime poller for file I/O
where possible. When a system call blocks on a pollable descriptor,
the goroutine will be blocked on the poller but the thread will be
released to run other goroutines. When using a non-pollable
descriptor, the os package will continue to use thread-blocking system
calls as before.

For example, on GNU/Linux, the runtime poller uses epoll. epoll does
not support ordinary disk files, so they will continue to use blocking
I/O as before. The poller will be used for pipes.

Since this means that the poller is used for many more programs, this
modifies the runtime to only block waiting for the poller if there is
some goroutine that is waiting on the poller. Otherwise, there is no
point, as the poller will never make any goroutine ready. This
preserves the runtime's current simple deadlock detection.

This seems to crash FreeBSD systems, so it is disabled on FreeBSD.
This is issue 19093.

Using the poller on Windows requires opening the file with
FILE_FLAG_OVERLAPPED. We should only do that if we can remove that
flag if the program calls the Fd method. This is issue 19098.

Update #6817.
Update #7903.
Update #15021.
Update #18507.
Update #19093.
Update #19098.

Change-Id: Ia5197dcefa7c6fbcca97d19a6f8621b2abcbb1fe
Reviewed-on: https://go-review.googlesource.com/36800
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment