New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Out of memory error when running on in-ram file system / GCF #225

Closed
sam0x17 opened this Issue Jun 13, 2018 · 10 comments

Comments

Projects
None yet
3 participants
@sam0x17

sam0x17 commented Jun 13, 2018

I'm running into the fabled error:

GC Warning: Out of memory - trying to allocate less
Insufficient memory for GC_all_nils

when running crystal language binaries from within a Google Cloud Function, and when there is tons of free ram (generally 1.7 gigs). GCF uses a 2 GB in-ram file system, meaning if I create a 20 MB file, I am really using up 20 MB of ram, etc. Running crystal --version should not require 2 GB of ram, so something fishy is going on. I've checked the output of free -h right before I run crystal --version and indeed there are like 1.8 gigs of free ram.

For some reason no matter what I do, boehm-gc is throwing this error via crystal. I am wondering if maybe this is a bug? If you give me a portable linux binary I can run it in GCF to try to debug this more.

If you are unfamiliar with crystal, they bind with boehm-gc here: https://github.com/crystal-lang/crystal/blob/master/src/gc/boehm.cr

Here is the same issue posted on the crystal repo:
crystal-lang/crystal#6188

@ivmai

This comment has been minimized.

Owner

ivmai commented Jun 13, 2018

This means GC cannot grab memory from OS (GC uses mmap or sbrk to get more memory depending on the config).

@sam0x17

This comment has been minimized.

sam0x17 commented Jun 14, 2018

@ivmai the folks over in the crystal lang issue thought you guys might be able to make something out from this strace of the crash: https://pastebin.com/hqyuZgS8

@ivmai

This comment has been minimized.

Owner

ivmai commented Sep 27, 2018

Thanks for reporting. Based on strace, I see 2 things that together lead to the error.

brk(NULL)                               = 0x55afd8456000
open("/dev/zero", O_RDONLY)             = 3

This means brk has failed to move the program data boundary for some reason, it might be OK (I don't know who calls mremap and probably there's some relation between mremap EFAULT and brk fail). To handle brk failures, BDWGC switches to using mmap after the first one.

mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE, 3, 0) = -1 ENODEV (No such device)

3 stands for fd of /dev/zero.
I think this means GCF does not allow mapping of /dev/zero. The anonymous way of getting memory works:

mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a9963c9a000

So, to workaround this issue on GCF the BDWGC library should be compiled with "-D USE_MMAP_ANON" passed to CFLAGS. Could you please try this? (I know that you don't need it anymore but it would be nice to check at least.)

Alternatively I'm thinking on the run-time workaround - to switch to anonymous mmap if the first mmap(/dev/zero) failed with ENODEV.

@ivmai

This comment has been minimized.

Owner

ivmai commented Sep 27, 2018

Alternatively I'm thinking on the run-time workaround - to switch to anonymous mmap if the first mmap(/dev/zero) failed with ENODEV.

Run-time detection is not really needed, it is OK to default to MAP_ANONYMOUS/MAP_ANON when available (at compile time).

to workaround this issue on GCF the BDWGC library should be compiled with "-D USE_MMAP_ANON" passed to CFLAGS

Hmm. In case of linux platform, USE_MMAP_ANON is defined by default (if mmap is used).
Which BDWGC version and binary do you use? It's compiled to Linux/x64, right?

@sam0x17

This comment has been minimized.

sam0x17 commented Sep 28, 2018

I really don't know much about how crystal integrates with this project, but if you ask in the crystal issue some people familiar with this will be able to chime in (also there is a lot more information and an strace in there):

crystal-lang/crystal#6188

ivmai added a commit that referenced this issue Oct 3, 2018

Fix memory allocation on GCF (Linux/x64)
Issue #225 (bdwgc).

GCF (Google Cloud Function) seems to have mmap("/dev/zero") operation
unsupported.  The workaround is to use mmap(MAP_ANONYMOUS).

* include/private/gcconfig.h [X86_64 && LINUX && __GLIBC__
&& !__UCLIBC__] (USE_MMAP_ANON): Define macro; add comment.
@ivmai

This comment has been minimized.

Owner

ivmai commented Oct 3, 2018

Note: the above commit (32704bf) is applicable only to libgc-8.0.0 or later.

ivmai added a commit that referenced this issue Oct 3, 2018

Fix memory allocation on GCF (Linux/x64)
(back-port of commit 32704bf from 'master')

Issue #225 (bdwgc).

GCF (Google Cloud Function) seems to have mmap("/dev/zero") operation
unsupported.  The workaround is to use mmap(MAP_ANONYMOUS).

* include/private/gcconfig.h [X86_64 && LINUX && __GLIBC__
&& !__UCLIBC__ && !USE_MMAP] (USE_MMAP): Define macro.
* include/private/gcconfig.h [X86_64 && LINUX && __GLIBC__
&& !__UCLIBC__] (USE_MMAP_ANON): Define macro; add comment.
@ivmai

This comment has been minimized.

Owner

ivmai commented Oct 3, 2018

Commit eed796d should solve the issue in gc-7.6.8

@ivmai ivmai closed this Oct 3, 2018

@bcardiff

This comment has been minimized.

bcardiff commented Oct 3, 2018

@ivmai That commit will be included in the next 7.6 release, right? That would be 7.6.9, since 7.6.8 was released in Aug, 12th.

@ivmai

This comment has been minimized.

Owner

ivmai commented Oct 3, 2018

This patch would be included in libgc-7.6.10 (planned for November).

@sam0x17

This comment has been minimized.

sam0x17 commented Oct 3, 2018

excellent news thanks guys!!

ivmai added a commit that referenced this issue Oct 4, 2018

Fix memory allocation on GCF (Linux/x64)
Issue #225 (bdwgc).

GCF (Google Cloud Function) seems to have mmap("/dev/zero") operation
unsupported.  The workaround is to use mmap(MAP_ANONYMOUS).

* include/private/gcconfig.h [X86_64 && LINUX && __GLIBC__
&& !__UCLIBC__] (USE_MMAP_ANON): Define macro; add comment.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment