runtime: cannot map pages, malloc/free deadlock #1464
Labels
Comments
ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 31072 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) 3448192 open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 31072 virtual memory (kbytes, -v) 4928960 file locks (-x) unlimited uname -a Linux marcel-computer 2.6.34.7-0.7-desktop #1 SMP PREEMPT 2010-12-13 11:13:53 +0100 x86_64 x86_64 x86_64 GNU/Linux |
|
Comment 3 by m@capitanio.org: "virtual memory unlimited" works but
by setting vmem to any value not:
virtual memory (kbytes, -v) 3 000000
virtual memory (kbytes, -v) 10 000000
uname -a
Linux trillian2t 2.6.35-24-generic #42-Ubuntu SMP Thu Dec 2 02:41:37 UTC 2010 x86_64
GNU/Linux
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) 3000000
file locks (-x) unlimited
free
total used free shared buffers cached
Mem: 3798964 3042836 756128 0 152500 509176
-/+ buffers/cache: 2381160 1417804
Swap: 8391904 173744 8218160
make[1]: Entering directory `/data4/soft/go/go/src/pkg/runtime/cgo'
6g -o _go_.6 cgo.go
6c -FVw iscgo.c
6c -FVw callbacks.c
echo 'int main() { return 0; }' >_cgo_main.c
gcc -m64 -g -O2 -fPIC -o amd64.o -c amd64.S
gcc -m64 -g -O2 -fPIC -o linux_amd64.o -c linux_amd64.c
gcc -m64 -g -O2 -fPIC -o util.o -c util.c
gcc -m64 -g -fPIC -O2 -o _cgo_main.o -c _cgo_main.c
gcc -m64 -g -fPIC -O2 -o _cgo1_.o _cgo_main.o amd64.o linux_amd64.o util.o -lpthread
cgo -dynimport _cgo1_.o >__cgo_import.c && mv -f __cgo_import.c _cgo_import.c
throw: runtime: cannot map pages in arena address space
throw: malloc/free - deadlock
double panic
make[1]: *** [_cgo_import.c] Error 3
make[1]: Leaving directory `/data4/soft/go/go/src/pkg/runtime/cgo'
make: *** [runtime/cgo.install] Error 2
|
hg identity 961463ad489e tip uname -a Linux linux-black 2.6.34.7-0.3-desktop #1 SMP PREEMPT 2010-09-20 15:27:38 +0200 x86_64 x86_64 x86_64 GNU/Linux ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 63377 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) 6963028 open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 63377 virtual memory (kbytes, -v) 8230560 file locks (-x) unlimited |
Thanks for all the reports. This is very useful. I am in the middle of debugging something else right now so I can't investigate directly, but here's what's going on in case someone has some insights. We start out by reserving a big chunk of the address space using mmap with prot == 0. On amd64 that chunk is 16 GB. But remember: no memory is being mapped, we are just reserving the right to map it later. Then we allocate memory for ourselves by mapping pages from that arena sequentially as needed. It is this later mapping that is failing, but those are supposed to be quite small, certainly within the 3GB limit that the first poster reported. Any ideas? Running strace on the binary should confirm my description. Maybe I am using mmap wrong. Thanks. Russ |
|
Comment 7 by m@capitanio.org: I found it happens at the end of the gotest script:
+ echo 'func main() {'
+ echo ' testing.Main(__regexp__.MatchString, tests)'
+ echo ' testing.RunBenchmarks(__regexp__.MatchString, benchmarks)'
+ echo '}'
+ 6g -I _test _testmain.go
+ 6l -L _test _testmain.6
+ ./6.out
throw: runtime: cannot map pages in arena address space
throw: malloc/free - deadlock
double panic
strace ./6.out 2> strace.txt
execve("./6.out", ["./6.out"], [/* 40 vars */]) = 0
brk(0) = 0x3c75000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f932be2d000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f932be2c000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f932be2b000
arch_prctl(ARCH_SET_FS, 0x7f932be2c6a0) = 0
arch_prctl(ARCH_SET_FS, 0x616230) = 0
rt_sigaction(SIGQUIT, {0x412055, ~[], SA_RESTORER|SA_STACK|SA_SIGINFO, 0x4120a4}, NULL,
8) = 0
rt_sigaction(SIGILL, {0x412055, ~[], SA_RESTORER|SA_STACK|SA_SIGINFO, 0x4120a4}, NULL,
8) = 0
rt_sigaction(SIGTRAP, {0x412055, ~[], SA_RESTORER|SA_STACK|SA_SIGINFO, 0x4120a4}, NULL,
8) = 0
rt_sigaction(SIGABRT, {0x412055, ~[], SA_RESTORER|SA_STACK|SA_SIGINFO, 0x4120a4}, NULL,
8) = 0
rt_sigaction(SIGBUS, {0x412055, ~[], SA_RESTORER|SA_STACK|SA_SIGINFO, 0x4120a4}, NULL,
8) = 0
rt_sigaction(SIGFPE, {0x412055, ~[], SA_RESTORER|SA_STACK|SA_SIGINFO, 0x4120a4}, NULL,
8) = 0
rt_sigaction(SIGSEGV, {0x412055, ~[], SA_RESTORER|SA_STACK|SA_SIGINFO, 0x4120a4}, NULL,
8) = 0
rt_sigaction(SIGPIPE, {0x4120a3, ~[], SA_RESTORER|SA_STACK|SA_SIGINFO, 0x4120a4}, NULL,
8) = 0
rt_sigaction(SIGSTKFLT, {0x412055, ~[], SA_RESTORER|SA_STACK|SA_SIGINFO, 0x4120a4},
NULL, 8) = 0
rt_sigaction(SIGSYS, {0x412055, ~[], SA_RESTORER|SA_STACK|SA_SIGINFO, 0x4120a4}, NULL,
8) = 0
mmap(0xf800000000, 17179869184, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM
(Cannot allocate memory)
mmap(NULL, 131072, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f932be0b000
mmap(0xc, 1048576, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS,
-1, 0) = -1 EINVAL (Invalid argument)
write(2, "throw: ", 7throw: ) = 7
write(2, "runtime: cannot map pages in are"..., 48runtime: cannot map pages in arena
address space) = 48
write(2, "\n", 1
) = 1
write(2, "\n", 1
) = 1
write(2, "throw: ", 7throw: ) = 7
write(2, "malloc/free - deadlock", 22malloc/free - deadlock) = 22
write(2, "\n", 1
) = 1
write(2, "double panic\n", 13double panic
) = 13
exit_group(3) = ?
|
|
Comment 8 by m@capitanio.org: (Sorry, it's any binary produced by the new release) > mmap(0xf800000000, 17179869184, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory) as rsc mentioned it allocates 16GB virtual memory setting ulimit -v 17179870 lets it run ... |
|
Comment 9 by m@capitanio.org: Looks like there is no other way to stop misbehaved programs rendering the linux computer totally unresponsible, than limit the virtual memory size: http://stackoverflow.com/questions/3360348/why-ulimit-cant-limit-resident-memory-successfully-and-how Is the reserving of 16GB really needed? Or does somebody know a other way to stop a program harvesting all the swap space? |
It's the way the code works, just to make sure no one else steps on the address space. I didn't expect that mapping the address space with no permissions would count toward the ulimit charge. Obviously something will have to change, since incompatibility with ulimit -v is a big deal. Is there some other way to mark address space "don't let someone else use this?". |
|
Comment 11 by geek@duzy.info: >>$ ulimit -a ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) 204800 scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 128622 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) 204800 open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) 3600 max user processes (-u) 115 virtual memory (kbytes, -v) 2097152 file locks (-x) unlimited >>$ uname -a Linux <HOSTNAME> 2.6.32-26.1.BHsmp #1 SMP Fri Nov 26 12:12:01 MST 2010 x86_64 x86_64 x86_64 GNU/Linux |
|
Comment 12 by m@capitanio.org: According to the manuals and if I didn't missed some syscall, the options are: propose a new syscall, write a go kernel driver, set a 16G rlimit or live with the available chunk of virtual memory ;) http://www.kernel.org/doc/man-pages/online/pages/man2/getrlimit.2.html http://www.kernel.org/doc/man-pages/online/pages/man2/mmap.2.html http://www.kernel.org/doc/man-pages/online/pages/man2/mmap2.2.html |
|
Comment 14 by m@capitanio.org: That reminds me of a another option: "What would the advantages of using a GNU/Hurd system be over say a GNU/Linux system?" "There's probably no gigantic advantage that jumps out at the user's face if you're not writing interesting programs. The Hurd offers interesting, powerful capabilities. For instance, you can write your own filesystem, so you could implement any sort of behavior you want and package it as a file." but "The Hurd runs, and missing features are gradually being added. However, for practical use today, you would use a Linux-based version of GNU." |
|
Comment 15 by m@capitanio.org: Just notes if someone outside the go team will try to understand the problem. The change was made in commit http://code.google.com/p/go/source/detail?r=dba56641aa19 -ulimit -v 4000000 +# Linux charges reserved but not mapped addresses to ulimit -v +# so we have to use ulimit -m. +ulimit -m 4000000 as mentioned in #9 this is nop at least on linux The comments in src/pkg/runtime/malloc.goc and src/pkg/runtime/malloc.h explain more what's going on: + if(sizeof(void*) == 8) { + // On a 64-bit machine, allocate from a single contiguous reservation. + // 16 GB should be big enough for now. + // + // The code will work with the reservation at any address, but ask + // SysReserve to use 0x000000f800000000 if possible. + // Allocating a 16 GB region takes away 36 bits, and the amd64 + // doesn't let us choose the top 17 bits, so that leaves the 11 bits + // in the middle of 0x00f8 for us to choose. Choosing 0x00f8 means + // that the valid memory addresses will begin 0x00f8, 0x00f9, 0x00fa, 0x00fb. + // None of the bytes f8 f9 fa fb can appear in valid UTF-8, and + // they are otherwise as far from ff (likely a common byte) as possible. + // Choosing 0x00 for the leading 6 bits was more arbitrary, but it + // is not a common ASCII code point either. Using 0x11f8 instead + // caused out of memory errors on OS X during thread allocations. + // These choices are both for debuggability and to reduce the + // odds of the conservative garbage collector not collecting memory + // because some non-pointer block of memory had a bit pattern + // that matched a memory address. + arena_size = 16LL<<30; + p = runtime·SysReserve((void*)(0x00f8ULL<<32), arena_size); + if(p == nil) + runtime·throw("runtime: cannot reserve arena virtual address space"); + runtime·mheap.arena_start = p; + runtime·mheap.arena_used = p; + runtime·mheap.arena_end = p + arena_size; + } else { + // On a 32-bit machine, we'll take what we can get for each allocation + // and maintain arena_start and arena_end as min, max we've seen. + runtime·mheap.arena_start = (byte*)0xffffffff; + runtime·mheap.arena_end = 0; + } + + // Initialize the rest of the allocator. runtime·MHeap_Init(&runtime·mheap, runtime·SysAlloc); m->mcache = runtime·allocmcache(); ... + // Number of bits in page to span calculations (4k pages). + // On 64-bit, we limit the arena to 16G, so 22 bits suffices. + // On 32-bit, we don't bother limiting anything: 20 bits for 4G. #ifdef _64BIT -#include "mheapmap64.h" + MHeapMap_Bits = 22, #else -#include "mheapmap32.h" + MHeapMap_Bits = 20, #endif |
We've seen about 15 random crashes on our build server that builds Go periodically since around about the time this change was checked in. compress/flate especially likes blowing up. I haven't seen any other weird behavior on the machine, so I think the physical memory is fine. Logs attached. Attachments:
|
|
Comment 18 by m@capitanio.org: Hm, checked on OSX ulimit -m or ulimit -v are both ignored. There is no a RLIMIT_AS only a RLIMIT_RSS and it seems to be a system recommendation, not a limit ... http://developer.apple.com/library/mac/#DOCUMENTATION/Darwin/Reference/ManPages/man2/setrlimit.2.html Most linux distributions doesn't set it too, so maybe just say: set ulimit -v unlimited in linux for go programs? |
#16: different bug. https://golang.org/issue/1479 |
Well, my opinion here is: rollback the modifications that cause this problem. I'm not in favor to always set "ulimit -v unlimited" because on many (most?) Linux system this the only effectivy way to protect memory consumption. Thus this is not a good workaround. To check if your distribution honors the various setting: (ulimit -v 0; godoc) - will terminate immediately on my system (ulimit -m 0; godoc) - runs happily even if I just said: don't allocate real memory |
|
Comment 23 by m@capitanio.org: http://groups.google.com/group/golang-nuts/browse_thread/thread/32b7975e36824438 release.2011-02-01.1 7387 cd ../cmd/hgpatch && make clean rm -rf *.o *.a *.[568vq] [568vq].out goyacc rm -rf *.o *.a *.[568vq] [568vq].out hgpatch CGOPKGPATH= cgo -- gmp.go throw: runtime: cannot reserve memory bitmap virtual address space declare -x GOARCH="386" declare -x GOOS="darwin" |
|
This issue was closed by revision 1cc8c87. Status changed to Fixed. |
This issue was closed.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
multiple reports about errors like: throw: runtime: cannot map pages in arena address space throw: malloc/free - deadlock double panic on Linux amd64. to Linux reporters: please let us know what: ulimit -a uname -a prints on your system. thanks.The text was updated successfully, but these errors were encountered: