-
Notifications
You must be signed in to change notification settings - Fork 892
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable PCBC completionObjects autoShrink to reduce memory usage and gc #3913
Enable PCBC completionObjects autoShrink to reduce memory usage and gc #3913
Conversation
I have a question about this. The memory occupation is not about the array size. The key(CompletionKey) and the value(CompletionValue) is the occupier. As long as the key and the value is removed, the memory occupation will be decrease. |
@horizonzy The layout of the array object in memory, when pointer compression is enabled, includes We have an object array with a size of 16384, and the space it occupies in memory is = 8 + 4 + 4 + 4 * 16384 = 65552 If we can turn on autoShrink, in the case of size=0, the array size will shrink to 24. This way we can reclaim a lot of space in memory. |
Why multiply 16. |
@horizonzy ConcurrentOpenHashMap default DefaultConcurrencyLevel=16 |
Thanks, I got it. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice improvement. LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice Catch!
@merlimat @eolivelli @dlg99 @zymap Can you help take a look at this pr. Thanks. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
#3913) ### Motivation PerChannelBookieClient completionObjects occupy a lot of heap space and cannot be recycled. The figure below shows that the internal table array of ConcurrentOpenHashMap has used space size=0, but the array length is still 16384, and the memory overhead is 65552bytes. ![image](https://user-images.githubusercontent.com/35599757/231114802-db90c49b-d295-46d7-b7db-785035b341f0.png) ![image](https://user-images.githubusercontent.com/35599757/231113930-bd9f3f54-9052-4c0b-9a3f-2fc493632e35.png) ConcurrentOpenHashMap default DefaultConcurrencyLevel=16. We have hundreds of bookie nodes. Due to the feature of bookie polling and writing, the client and server have long connection characteristics, which will as a result, the memory usage of about 65552 * 16 * 1776 = 1.74GB cannot be recycled, and the space take up by these tables is all size=0 (The broker's owner topic has drifted to other brokers due to Full GC). ![image](https://user-images.githubusercontent.com/35599757/231117087-08c80320-fa71-49c2-a199-cfee3d83ddc5.png) When the throughput of the pulsar cluster increases and the bookie cluster expands, these memory usage will also increase. Coupled with the unreasonable memory usage in other aspects of pulsar that we know, this will cause the pulsar broker to continuously generate full gc. ### Changes I think adding autoShrink to completionObjects can reduce this part of memory usage and reduce the frequency of Full GC. (cherry picked from commit ca33b31)
#3913) ### Motivation PerChannelBookieClient completionObjects occupy a lot of heap space and cannot be recycled. The figure below shows that the internal table array of ConcurrentOpenHashMap has used space size=0, but the array length is still 16384, and the memory overhead is 65552bytes. ![image](https://user-images.githubusercontent.com/35599757/231114802-db90c49b-d295-46d7-b7db-785035b341f0.png) ![image](https://user-images.githubusercontent.com/35599757/231113930-bd9f3f54-9052-4c0b-9a3f-2fc493632e35.png) ConcurrentOpenHashMap default DefaultConcurrencyLevel=16. We have hundreds of bookie nodes. Due to the feature of bookie polling and writing, the client and server have long connection characteristics, which will as a result, the memory usage of about 65552 * 16 * 1776 = 1.74GB cannot be recycled, and the space take up by these tables is all size=0 (The broker's owner topic has drifted to other brokers due to Full GC). ![image](https://user-images.githubusercontent.com/35599757/231117087-08c80320-fa71-49c2-a199-cfee3d83ddc5.png) When the throughput of the pulsar cluster increases and the bookie cluster expands, these memory usage will also increase. Coupled with the unreasonable memory usage in other aspects of pulsar that we know, this will cause the pulsar broker to continuously generate full gc. ### Changes I think adding autoShrink to completionObjects can reduce this part of memory usage and reduce the frequency of Full GC. (cherry picked from commit ca33b31)
Motivation
PerChannelBookieClient completionObjects occupy a lot of heap space and cannot be recycled.
The figure below shows that the internal table array of ConcurrentOpenHashMap has used space size=0, but the array length is still 16384, and the memory overhead is 65552bytes.
ConcurrentOpenHashMap default DefaultConcurrencyLevel=16. We have hundreds of bookie nodes. Due to the feature of bookie polling and writing, the client and server have long connection characteristics, which will as a result, the memory usage of about 65552 * 16 * 1776 = 1.74GB cannot be recycled, and the space take up by these tables is all size=0 (The broker's owner topic has drifted to other brokers due to Full GC).
When the throughput of the pulsar cluster increases and the bookie cluster expands, these memory usage will also increase. Coupled with the unreasonable memory usage in other aspects of pulsar that we know, this will cause the pulsar broker to continuously generate full gc.
Changes
I think adding autoShrink to completionObjects can reduce this part of memory usage and reduce the frequency of Full GC.