New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Out of memory on realloc() #546

Closed
nshalman opened this Issue Jan 21, 2016 · 6 comments

Comments

Projects
None yet
3 participants
@nshalman
Copy link
Contributor

nshalman commented Jan 21, 2016

For @pfmooney

$ docker info | egrep -v '(SDCAccount|ID):'
Containers: 0
Images: 119
Storage Driver: sdc
Execution Driver: sdc-0.3.0
Logging Driver: json-file
Kernel Version: 3.12.0-1-amd64
Operating System: SmartDataCenter
CPUs: 0
Total Memory: 0 B
Name: us-east-3b
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

$ docker run --rm -it -p 9001:9001 --name tor-relay -m 8g jess/tor-relay -f /etc/tor/torrc.bridge
Jan 21 00:22:36.261 [notice] Tor v0.2.7.6 running on Linux with Libevent 2.0.22-stable, OpenSSL 1.0.2e and Zlib 1.2.8.
Jan 21 00:22:36.262 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Jan 21 00:22:36.262 [notice] Read configuration file "/etc/tor/torrc.bridge".
Jan 21 00:22:36.268 [notice] Based on detected system memory, MaxMemInQueues is set to 6144 MB. You can override this by setting MaxMemInQueues by hand.
Jan 21 00:22:36.272 [notice] Wow!  I detected that you have 48 CPUs. I will not autodetect any more than 16, though.  If you want to configure more, set NumCPUs in your torrc
Jan 21 00:22:36.272 [notice] Opening Socks listener on 127.0.0.1:9050
Jan 21 00:22:36.272 [notice] Opening OR listener on 0.0.0.0:9001
Jan 21 00:22:36.000 [notice] Parsing GEOIP IPv4 file /usr/share/tor/geoip.
Jan 21 00:22:36.000 [err] Out of memory on realloc(). Dying.
@pfmooney

This comment has been minimized.

Copy link
Member

pfmooney commented Jan 21, 2016

Out of curiosity, have you tried running with MaxMemInQueues explicitly set to a smaller value?

@nshalman

This comment has been minimized.

Copy link
Contributor

nshalman commented Jan 21, 2016

I have not. That number was automatically scaling along with the memory allocation.

@nshalman

This comment has been minimized.

Copy link
Contributor

nshalman commented Jan 22, 2016

8 gigs of ram, only use 256M for the queues and only use 2 cpus... still fails:

$ docker run --rm -it -p 9001:9001 --name tor-relay -m 8g jess/tor-relay -f /etc/tor/torrc.bridge --MaxMemInQueues 256m --NumCPUs 2
Jan 22 13:28:46.129 [notice] Tor v0.2.7.6 running on Linux with Libevent 2.0.22-stable, OpenSSL 1.0.2e and Zlib 1.2.8.
Jan 22 13:28:46.129 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Jan 22 13:28:46.134 [notice] Read configuration file "/etc/tor/torrc.bridge".
Jan 22 13:28:46.148 [notice] Opening Socks listener on 127.0.0.1:9050
Jan 22 13:28:46.148 [notice] Opening OR listener on 0.0.0.0:9001
Jan 22 13:28:46.000 [notice] Parsing GEOIP IPv4 file /usr/share/tor/geoip.
Jan 22 13:28:46.000 [err] Out of memory on realloc(). Dying.
@nshalman

This comment has been minimized.

Copy link
Contributor

nshalman commented Jan 22, 2016

$ docker run --rm -it -p 9001:9001 --name tor-relay -m 8g --entrypoint=/bin/sh jess/tor-relay
/ $ su -
4aa4ce6aca1f:~# apk update
fetch http://dl-4.alpinelinux.org/alpine/v3.2/main/x86_64/APKINDEX.tar.gz
v3.2.3-165-g80d0343 [http://dl-4.alpinelinux.org/alpine/v3.2/main]
OK: 5293 distinct packages available
4aa4ce6aca1f:~# apk add strace
(1/1) Installing strace (4.9-r1)
Executing busybox-1.23.2-r0.trigger
OK: 13 MiB in 18 packages
4aa4ce6aca1f:~# exit
/ $ strace -f tor -f /etc/tor/torrc.bridge --MaxMemInQueues 256m --NumCPUs 2
<snip>
brk(0x13e000)                           = 0x13e000
readv(9, [{"", 0}, {"O\n1567858688,1567860735,SE\n15678"..., 1024}], 2) = 1024
readv(9, [{"", 0}, {"MD\n1568063488,1568083967,RO\n1568"..., 1024}], 2) = 1024
readv(9, [{"", 0}, {",GP\n1568267264,1568267775,MF\n156"..., 1024}], 2) = 1024
brk(0x13f000)                           = 0x13f000
readv(9, [{"", 0}, {"1,RU\n1570609152,1570611199,PL\n15"..., 1024}], 2) = 1024
readv(9, [{"", 0}, {"67,AM\n1571424768,1571425023,CZ\n1"..., 1024}], 2) = 1024
readv(9, [{"", 0}, {"295,UA\n1571447296,1571447807,RU\n"..., 1024}], 2) = 1024
brk(0x140000)                           = 0x140000
readv(9, [{"", 0}, {"9279,UA\n1571489280,1571489535,BY"..., 1024}], 2) = 1024
readv(9, [{"", 0}, {"40695,RU\n1571540696,1571540696,C"..., 1024}], 2) = 1024
readv(9, [{"", 0}, {"725311,GB\n1571725312,1571727359,"..., 1024}], 2) = 1024
brk(0x141000)                           = 0x141000
readv(9, [{"", 0}, {"1830783,RE\n1571830784,1571831807"..., 1024}], 2) = 1024
readv(9, [{"", 0}, {"72118527,DE\n1572118528,157212671"..., 1024}], 2) = 1024
readv(9, [{"", 0}, {"572413439,RU\n1572413440,15724175"..., 1024}], 2) = 1024
brk(0x142000)                           = 0x142000
readv(9, [{"", 0}, {"1572545439,AU\n1572545440,1572546"..., 1024}], 2) = 1024
mremap(0x7ffffee45000, 266240, 528384, MREMAP_MAYMOVE) = -1 ENOMEM (Out of memory)
clock_gettime(CLOCK_REALTIME, {1453470330, 544601118}) = 0
write(1, "Jan 22 13:45:30.000 [err] Out of"..., 61Jan 22 13:45:30.000 [err] Out of memory on realloc(). Dying.
) = 61
lseek(9, -485, SEEK_CUR)                = 793115
exit_group(1)                           = ?
+++ exited with 1 +++
/ $ /native/usr/bin/truss tor -f /etc/tor/torrc.bridge --MaxMemInQueues 256m --NumCPUs 2
<snip>
sigaction(SIGRTMIN+13, 0x00000000, 0x7FFFFFEFFBB0) = 0
sigaction(SIGRTMIN+14, 0x00000000, 0x7FFFFFEFFBB0) = 0
sigaction(SIGRTMIN+15, 0x00000000, 0x7FFFFFEFFBB0) = 0
sigaction(SIGRTMIN+16, 0x00000000, 0x7FFFFFEFFBB0) = 0
sigaction(SIGRTMAX-15, 0x00000000, 0x7FFFFFEFFBB0) = 0
sigaction(SIGRTMAX-14, 0x00000000, 0x7FFFFFEFFBB0) = 0
sigaction(SIGRTMAX-13, 0x00000000, 0x7FFFFFEFFBB0) = 0
sigaction(SIGRTMAX-12, 0x00000000, 0x7FFFFFEFFBB0) = 0
sigaction(SIGRTMAX-11, 0x00000000, 0x7FFFFFEFFBB0) = 0
sigaction(SIGRTMAX-10, 0x00000000, 0x7FFFFFEFFBB0) = 0
sigaction(SIGRTMAX-9, 0x00000000, 0x7FFFFFEFFBB0) = 0
sigaction(SIGRTMAX-8, 0x00000000, 0x7FFFFFEFFBB0) = 0
sigaction(SIGRTMAX-7, 0x00000000, 0x7FFFFFEFFBB0) = 0
sigaction(SIGRTMAX-6, 0x00000000, 0x7FFFFFEFFBB0) = 0
sigaction(SIGRTMAX-5, 0x00000000, 0x7FFFFFEFFBB0) = 0
sigaction(SIGRTMAX-4, 0x00000000, 0x7FFFFFEFFBB0) = 0
sigaction(SIGRTMAX-3, 0x00000000, 0x7FFFFFEFFBB0) = 0
sigaction(SIGRTMAX-2, 0x00000000, 0x7FFFFFEFFBB0) = 0
sigaction(SIGRTMAX-1, 0x00000000, 0x7FFFFFEFFBB0) = 0
sigaction(SIGRTMAX, 0x00000000, 0x7FFFFFEFFBB0) = 0
sigaction(SIGPWR, 0x7FFFFFEFFB00, 0x00000000)   = 0
lwp_sigmask(SIG_SETMASK, 0x00000000, 0x00000000, 0x00000000, 0x00000000) = 0xFFBFFEFF [0xFFFFFFFF]
mmap(0x00000000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, 4294967295, 0) = 0x7FFFFEF30000
getcontext(0x7FFFFEF30040)
sysconfig(_CONFIG_STACK_PROT)                   = 3
mmap(0x00000000, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, 4294967295, 0) = 0x7FFFFEEEF000
brand(147, 0x7FFFFEF2F000, 0x00000000, 0x00000000, 0x00000000, 0x00000000) = 0
getcontext(0x7FFFFFEFF8B0)
brand(139, 0x7FFFFFEFF8B0, 0x00000000, 0x00000000, 0x00000000, 0x7FFFFFEFFC40) = 0
brand(130, 0x7FFFFEF2ECA0, 0x00000009, 0x7FFFFE800000, 0x00000000, 0x00000002) = 0
brand(130, 0x7FFFFEF2ECA0, 0x00000009, 0x7FFFFEA14000, 0x00000000, 0x00000012) = 0
brand(130, 0x7FFFFEF2ECA0, 0x00000009, 0x7FFFFE400000, 0x00000000, 0x00000002) = 0
brand(130, 0x7FFFFEF2ECA0, 0x00000009, 0x7FFFFE63C000, 0x00000000, 0x00000012) = 0
brand(130, 0x7FFFFEF2ECA0, 0x00000009, 0x7FFFFE000000, 0x00000000, 0x00000002) = 0
brand(130, 0x7FFFFEF2ECA0, 0x00000009, 0x7FFFFE25E000, 0x00000000, 0x00000012) = 0
brand(130, 0x7FFFFEF2ECA0, 0x00000009, 0x7FFFFDA00000, 0x00000000, 0x00000002) = 0
brand(130, 0x7FFFFEF2ECA0, 0x00000009, 0x7FFFFDDF3000, 0x00000000, 0x00000012) = 0
brand(130, 0x7FFFFEF2ECA0, 0x00000009, 0x7FFFFDE1D000, 0x00000000, 0x00000032) = 0
brand(130, 0x7FFFFEF2ECA0, 0x0000000A, 0x00000000, 0x00000000, 0x7FFFFE0003A0) = 0
brand(130, 0x7FFFFEF2ECA0, 0x0000000A, 0x00000000, 0x00000000, 0x7FFFFEC03750) = 0
brand(130, 0x7FFFFEF2ECA0, 0x0000000A, 0x00000000, 0x00000000, 0x7FFFFDA011A0) = 0
brand(130, 0x7FFFFEF2ECA0, 0x0000000A, 0x00000000, 0x00000000, 0x7FFFFEC03750) = 0
brand(130, 0x7FFFFEF2ECA0, 0x0000000A, 0x00000000, 0x00000000, 0x7FFFFEC03750) = 0
brand(130, 0x7FFFFEF2ECA0, 0x0000000A, 0x00000000, 0x00000000, 0x7FFFFEC03750) = 0
brand(130, 0x7FFFFEF2ECA0, 0x00000066, 0x00000064, 0x00000000, 0x00000000) = 0
brand(130, 0x7FFFFEF2ECA0, 0x00000066, 0x00000064, 0x00000000, 0x0002CCBC) = 0
brand(130, 0x7FFFFEF2ECA0, 0x00000009, 0x7FFFFEEE0000, 0x00000000, 0x00000001) = 0
Jan 22 13:47:02.308 [notice] Tor v0.2.7.6 running on Linux with Libevent 2.0.22-stable, OpenSSL 1.0.2e and Zlib 1.2.8.
Jan 22 13:47:02.308 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Jan 22 13:47:02.308 [notice] Read configuration file "/etc/tor/torrc.bridge".
brand(130, 0x7FFFFEF2ECA0, 0x0000000E, 0x00000000, 0x00000000, 0x00000008) = 0
brand(130, 0x7FFFFEF2ECA0, 0x0000000E, 0x00000000, 0x00000000, 0x00000008) = 0
brand(130, 0x7FFFFEF2ECA0, 0x0000000E, 0x00000000, 0x00000000, 0x00000008) = 0
brand(130, 0x7FFFFEF2ECA0, 0x00000035, 0x00000000, 0x00000000, 0x00034A70) = 0
Jan 22 13:47:02.316 [notice] Opening Socks listener on 127.0.0.1:9050
brand(130, 0x7FFFFEF2ECA0, 0x00000032, 0x00000000, 0x00000000, 0x00000000) = 0
Jan 22 13:47:02.316 [notice] Opening OR listener on 0.0.0.0:9001
brand(130, 0x7FFFFEF2ECA0, 0x00000032, 0x00000000, 0x00000000, 0x00000000) = 0
brand(130, 0x7FFFFEF2ECA0, 0x00000066, 0x00000064, 0x00000000, 0x00000002) = 0
brand(130, 0x7FFFFEF2ECA0, 0x00000068, 0x0000FFFD, 0x00000000, 0x00000002) = 0
brand(130, 0x7FFFFEF2ECA0, 0x00000049, 0x00000000, 0x00000000, 0x00000000) = 0
Jan 22 13:47:02.000 [notice] Parsing GEOIP IPv4 file /usr/share/tor/geoip.
brand(130, 0x7FFFFEF2ECA0, 0x00000009, 0x7FFFFEE45000, 0x00000000, 0x00000022) = 0
mmap(0xFFFFFFFFFF7F1000, 528384, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, 4294967295, 0) Err#12 ENOMEM
munmap(0x7FFFFEE40000, 8192)                    = 0
brand(130, 0x7FFFFEF2ECA0, 0x00000019, 0xFFFFFFFFFFFFFFF4, 0x0000000C, 0x00000001) = 0
Jan 22 13:47:02.000 [err] Out of memory on realloc(). Dying.
brand(130, 0x7FFFFEF2ECA0, 0x00000008, 0x000C1A1B, 0x00000000, 0x00000000) Err#793115
setcontext(0x7FFFFEF2E830)
_exit(1)
@RantyDave

This comment has been minimized.

Copy link

RantyDave commented Mar 2, 2016

FWIW Alpine uses a different memory allocator from other distributions, so it may well be that. Try again on Debian?

@nshalman

This comment has been minimized.

Copy link
Contributor

nshalman commented Mar 17, 2016

joyent/illumos-joyent@28e4c4c (OS-5252) appears to have fixed this.
At the very least, I'm now able to launch a tor relay in us-east-3b of the JPC so I'm closing this out.

Thanks for the ping on IRC, @pfmooney.

@nshalman nshalman closed this Mar 17, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment