runtime: respect memory ulimit more precisely #5049
Labels
Milestone
Comments
mithro
added a commit
to mithro/go
that referenced
this issue
Mar 16, 2016
----------------------------------------------------------------------------
Currently if the go runtime tries to create a new system thread and is unable
to do so, it will fail with an error like;
```
18:22:18.752169 [go test -timeout 600s -v -race ./common/paniccatcher] was slow: 3m11.377s
runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f54695bdcc9
```
One reason for this occurring is the system have a low "process limit". For a
long time it was fairly common for systems to allow 10k or more, but with
systemd and Linux 4.3 the default limit can be as little as 512.
Most of the code which calls pthread_create in src/runtime/cgo seems to do
something like;
```
err = pthread_create(&p, &attr, threadentry, ts);
pthread_sigmask(SIG_SETMASK, &oset, nil);
if (err != 0) {
fprintf(stderr, "runtime/cgo: pthread_create failed: %s\n", strerror(err));
abort();
}
```
This actually seems reasonable as recovering from thread creation is pretty
hard. As well, creating more then your system's process limit does feel like a
"just don't do that" type things.
However, from what I can see goroutine scheduler will create up to
sched.maxmcount threads and this is set to be initialized to 10k in proc.go at
line 425 (https://github.com/golang/go/blob/master/src/runtime/proc.go#L425).
Linux provides an API for getting the current thread limit, the getrlimit call
with RLIMIT_NPROC (see http://man7.org/linux/man-pages/man2/setrlimit.2.html)
which already seems to be exposed to Go code as syscall.Getrlimit but it is
missing the RLIMIT_NPROC constant needed to get the information.
This is similar to idea of respecting memlimit see
https://github.com/golang/go/blob/master/src/runtime/os1_linux.go#L270 and
probably related to golang#5049
----------------------------------------------------------------------------
|
Timed out. |
|
Incredible timing for that "time out." We just found this issue because we're hitting this exact problem with our services running in Marathon. What can be done to help get this prioritized and fixed? I imagine this is a problem for anyone running Golang apps in an environment such as Marathon or Kubernetes - those environments are only getting more common. |
|
CL https://golang.org/cl/35252 mentions this issue. |
gopherbot
pushed a commit
that referenced
this issue
Feb 7, 2017
mallocinit has evolved organically. Make a pass to clean it up in various ways: 1. Merge the computation of spansSize and bitmapSize. These were computed on every loop iteration of two different loops, but always have the same value, which can be derived directly from _MaxMem. This also avoids over-reserving these on MIPS, were _MaxArena32 is larger than _MaxMem. 2. Remove the ulimit -v logic. It's been disabled for many releases and the dead code paths to support it are even more wrong now than they were when it was first disabled, since now we *must* reserve spans and bitmaps for the full address space. 3. Make it clear that we're using a simple linear allocation to lay out the spans, bitmap, and arena spaces. Previously there were a lot of redundant pointer computations. Now we just bump p1 up as we reserve the spaces. In preparation for #18651. Updates #5049 (respect ulimit). Change-Id: Icbe66570d3a7a17bea227dc54fb3c4978b52a3af Reviewed-on: https://go-review.googlesource.com/35252 Reviewed-by: Russ Cox <rsc@golang.org>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
The text was updated successfully, but these errors were encountered: