info on debugging memory usage #122
Conversation
| If things are happy usage should return to a small level. This is quite disruptive | ||
| to the system however so use with care. | ||
|
|
||
| The above method can also be configured to run automatically when memory usage |
There was a problem hiding this comment.
I'd go further and say that we discourage this for any production use-case (because it has not had expensive testing).
A better solution for production use-cases is to use-XX:SoftRefLRUPolicyMSPerMB=1. See the discussion in https://issues.apache.org/jira/browse/BROOKLYN-375.
We should also describe the symptoms of BROOKLYN-375 - that the CPU usage goes very high, with almost all of that time being taken up by garbage collection.
We need to do more on BROOKLYN-375 - e.g. implement the cache, to stop using soft references!
| You should also note the task counts in the `brooklyn gc` messages described above, | ||
| and if there are an exceptional number of tasks or tasks are not clearing, | ||
| other log messages will describe what is happening, and the in-product task | ||
| view can indicate issues. `jstack` can also be useful if it is a task problem. |
There was a problem hiding this comment.
Also worth recommending the use of jmap -histo:live <pid> to see how many live instances of each type there are.
| toc: /guide/toc.json | ||
| --- | ||
|
|
||
| ## Memory Usage |
There was a problem hiding this comment.
Worth adding a link to this page from https://github.com/apache/brooklyn-docs/blob/master/guide/ops/server-cli-reference.md#memory-usage
|
Perhaps worth adding links to generic resources, so it's clearer we're telling people to use "normal" tools/approaches - e.g. https://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/memleaks.html |
|
@ahgittin useful tips for the docs - thanks. A few comments. |
|
addressed comments & merging |
|
|
||
| * Note the log lines about `brooklyn gc`, including memory and tasks | ||
| * Do not assume high memory usage alone is an error, as soft reference caches are deliberate; | ||
| use `forceClearSoftReferences()` to clear these |
There was a problem hiding this comment.
@ahgittin (cc @neykov) I thought we were not going to recommend using forceClearSoftReferences() this in any kind of production environment. Can we put in a caveat here about not using it in production. I'd be extremely caution about encouraging real users to call this until devs have been using it in anger themselves a lot.
With the use of the (much safer) -XX:SoftRefLRUPolicyMSPerMB=1, I'd expect the need for calling this would be greatly reduced.
There was a problem hiding this comment.
we say
This call can be disruptive to the system however so use with care.
and
Many JVM authorities discourage interfering with its garbage collector, however,
so use with care and study the particular JVM you are using.
this felt sufficient but if you still want i'll add more caveats.
note the code fails because it tries to allocate a huge block of memory: any parallel code trying to allocate a non-huge amount of memory should be unaffected.
4 participants
No description provided.