Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory usage growth after Jersey 2.25 update #3423

Closed
bernd opened this issue Jan 25, 2017 · 2 comments
Closed

Memory usage growth after Jersey 2.25 update #3423

bernd opened this issue Jan 25, 2017 · 2 comments
Assignees
@bernd @kroepke
Labels
blocker If not finished by release date, the release will be postponed. bug
Milestone
2.2.0

Comments

@bernd
Copy link
Member

bernd commented Jan 25, 2017 •
edited

Expected Behavior

The memory usage should not grow by over 80% after a Jersey update.

Current Behavior

We noticed a memory problem while testing the 2.2 beta. Nodes run into GC problems because the memory in the old gen is getting full. Comparing the 2.1 and 2.2 baseline looks like this:

image

The spikes in the second half of the graphs happens when processing the first messages. This might be a different problem.

So from 2.1 to 2.2 the old gen memory usage after starting Graylog went from around 170MB to over 300MB.

Possible Solution

The memory usage goes back to 2.1 levels when reverting the Jersey 2.25 update that was committed in db98621.

I suggest to downgrade to Jersey 2.24 for 2.2.

Steps to Reproduce (for bugs)

  1. Start Graylog 2.1
  2. Login to web interface and click around
  3. Measure memory usage via jmap -heap <pid>
  4. Start Graylog 2.2 beta
  5. Login to web interface and click around
  6. Measure memory usage via jmap -heap <pid>

Your Environment

  • Graylog Version: Graylog 2.2.0-beta.6
@bernd bernd added blocker If not finished by release date, the release will be postponed. bug labels Jan 25, 2017
@bernd bernd added this to the 2.2.0 milestone Jan 25, 2017
@bernd
Copy link
Member Author

bernd commented Jan 25, 2017

The Jersey 2.25 update pulled in grizzly 2.3.28.

When fixing the grizzly version to 2.3.26 and using Jersey 2.25.1, the memory usage goes down as well. The grizzly version 2.3.27 seems to introduce the memory problem.

@bernd
Copy link
Member Author

bernd commented Jan 25, 2017

In 2.3.27 Grizzly switched to a pooled memory manager with commit https://github.com/GrizzlyNIO/grizzly-mirror/commit/ba9beb2d1

This causes the memory usage to grow because of pre-allocated memory. Setting the -Dorg.glassfish.grizzly.DEFAULT_MEMORY_MANAGER=org.glassfish.grizzly.memory.HeapMemoryManager property brings the memory usage back to 2.1 levels.

@kroepke kroepke mentioned this issue Jan 25, 2017
Merged
@kroepke kroepke self-assigned this Jan 25, 2017
@ghost ghost assigned bernd Jan 25, 2017
@ghost ghost removed the in progress label Jan 25, 2017
bernd pushed a commit that referenced this issue Jan 25, 2017
@kroepke @bernd
Improve base-line memory usage (#3425)
db68c6f 
* force grizzly to use the HeapMemoryManager and not its default

the default switched to PooledMemoryManager which uses 10% of the heap for its pools by default
this creates too much old gen pressure for our purposes, so we switch back to the previous setting

* clear messages in MessageEvent as soon as the processor is done with them

this allows the garbage collector to eagerly collect those references and makes it less likely that the messages are promoted to the old gen (where they typically only increase memory pressure)
in the case of the output buffer the last set of messages in the ring would be help in memory, which can needlessly increase the memory footprint as well

Fixes #3423
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bernd bernd

@kroepke kroepke

Labels
blocker If not finished by release date, the release will be postponed. bug
Projects
None yet
Milestone
2.2.0
Development

No branches or pull requests
2 participants
@bernd @kroepke