runtime: limit number of operating system threads #4056
Labels
Milestone
Comments
What if GOMAXPROCS > MaxOSThreads? Do we want to limit the number of threads created to run cgo/SWIG, or do we want to limit the total number of threads including those created to run goroutines that make blocking system calls? An absolute limit on all threads is easy to implement and understand but I'm concerned that 256 is too low. |
Thread pool with bounded number of threads known to cause problems. The first problem is system-induced deadlocks. And it's not something that you will uncover during unit-testing. Then, it may cause poor performance, e.g. I may have X threads almost permanently blocked and I do not want them to affect the rest of the program. Then, it's difficult for a user to set it correctly; so users may change the default when they do not need to, or don't change when need to. The new runtime partially solves the problem by "ignoring" short syscalls. On network benchmarks with GOMAXPROCS=16 I see only about 25 threads. It may be further tuned by increasing delay before cpu retake if there are already a lot of threads. If we are committing the new runtime, then I would prefer to see whether it makes things better or not before introducing user control. Do we have a good reproducer for the problem? |
Related: discussion on http://golang.org/cl/6815049 about number of threads blocked in cgo DNS lookups. Cap that too in the net package? |
Attempted fix: https://golang.org/cl/7275049/ |
I had been under the impression that the Go runtime used something similar to libuv or libevent to do non-blocking IO without the use of a threadpool. Do you mean that Go spawns a separate thread for every blocking IO call? Wouldn't that defeat the performance advantages of goroutines, or am I missing something? |
In reference to this a post (https://groups.google.com/d/msg/golang-nuts/jgNKl0Jap_k/BeVBUAuNcBkJ) from the group, here are some error logs that have stack traces just in case anyone finds them useful for later optimizations. The code and logs are from a modified version of Webfront I was using to serve several sites off the same server. https://dl.dropbox.com/u/27496904/frontend/frontend.go https://dl.dropbox.com/u/27496904/frontend/errlogbak https://dl.dropbox.com/u/27496904/frontend/errlogbak2 The first log is about 5mb and the second managed to make it to about 25 before dying. Oddly enough, the Node processes actually serving the sites had no problem keeping up, it was just the proxy that kept crapping itself. I can't rule out the possibility that it's just a bug I couldn't find though. Almost forgot, go version running this was go1.0.3 |
Russ, back in 2011 you seemed against the idea (https://golang.org/issue/1644), too bad I didn't know about this bug before now. However, since a bounded thread pool may create unexpected deadlocks, what about introducing an ever increasing delay for spawning new threads after a certain number of threads? (one of Ms would then become a thread manager, adding more threads) On the other hand, since 2011, I just learned that I always have to limit concurrency where it matters, e.g. never call net.Dial with a random hostname directly, resolve the hostname with max 16 at the same time first and use the resolved address (also limit net.Dials altogethers before they became non-blocking and less of a problem). The issue forces you to think about what you're doing instead of just blindly doing it, and the result is better: if resolves are the problem that's what you have limited, it doesn't harm the rest of the system. |
I think we should do this for Go 1.2. In package runtime/debug: // SetMaxThreads sets the maximum number of operating system // threads that the runtime will create for the current program. // It returns the previous setting. // The default maximum is 1000. func SetMaxThreads(max int) int We should also limit the number of cgo calls that package net makes. |
I am skeptical about this. We do not know how many threads a program needs. So we will unnecessary limit it for large programs running on large boxes; and at the same time allow a "hello world" program to create 1000 threads when it needs only 1. Users must limit number of threads in sys/cgo calls. If that's implemented poorly in net package than that needs to be addressed (most likely we do not want 1000 threads resolving DNS). We can also throttle thread creation in runtime after some threshold. But hard limit is a time bomb. |
Ideally it's still complemented by thread creation throttling, because it's not fully user visible characteristic of the program, and it may be further complicated by interaction of various libraries. On my desktop I can easily create 20000+ threads, and have all them runnable at the same time. |
|
This issue was closed by revision 665feee. Status changed to Fixed. |
This issue was closed.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
The text was updated successfully, but these errors were encountered: