Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Tuning Glibc Environment Variables (e.g. MALLOC_ARENA_MAX) #320
This is the other issue we discussed @nebhale.
Having been looking into some Java memory issues recently, I came across the following two articles from Heroku (should I just call them "The unnamable ones"? :P):
Which suggest that tuning
In my own testing I have noticed that setting
There are performance implications from setting
That's my understanding at any rate. So I think most Java apps on CF are basically unaffected for the following reasons:
With all of that said, after digging deeper into this, it seems like setting
I had initially thought that there may be memory concerns here, but I'm not so sure anymore. I think the following scenarios are what would happen if there were a memory leak:
(Actual allocation decisions are a lot more complex - this is heavily simplified. See Understanding glibc malloc for an in-depth explanation).
This would explain why I saw the heap space reported by
With all that said, some allowance for the native memory used by individual threads should probably be made. If it's small enough, it could probably stay in the "native" part of the calculator. Otherwise, since it's relative to the number of threads used, it may need to be calculated somehow.
Regardless of the decision to set it, I do think that documenting it would be very useful (along with how it relates to Java, etc), as would providing a link to other
Have you checked this article written by Evan Jones:
Recently Heroku blogged about tracking down a similar bug in Kafka. GOV UK GDS team's article about debugging native memory leaks as well might also be helpful.
Evan Jones has also blogged about a native memory leak in Java's ByteBuffer.
@lhotari I'm aware of the first one - was not aware of the second issue. As far as I can tell, there is actually a HotSpot bug, and this might be what you're seeing as well: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8164293
I saw the GZIP thing, and even ran my app with JEMalloc, but did not see the same issues that they reported. Given that this issue (a) only appears in Java 8, and (b) disappears when HotSpot is disabled, I suspect that there are multiple issues that could cause apps to increase in memory over time.
@ipsi Thanks for pointing out the JDK bug.
Did you take a look at the assumptions I presented in #163 (comment) ?
I've notice that TieredCompilation is seemingly resulting in continual growth. Disabling TieredCompilation (-XX:-TieredCompilation), has stopped this growth from occurring on several of our applications (all using jdk8). I submitting a bug to Oracle today to evaluation, which could be related to the already mentioned bug. http://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8164293
Just thought I'd mention it, in case this helps you out.