HBASE-27111 Make Netty channel bytebuf allocator configurable. #4525
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
apurtell
changed the title
|
Update JIRA ID |
|
Unit test |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
In general I do not think we need to extends PooledByteBufAllocator, just make a global instance with 'new PooledByteBufAllocator(false)' is enough. And I do not think we should mark HeapByteBufAllocator as IA.LP(CONFIG), we will use 'heap' to reference it so users do not need to use the name in config file directly, we could just mark it IA.Private. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
Updated. Additional option for
Also updates |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
Unit test failures are related. Back with a fix. |
Support site configuration of the bytebuf allocator that Netty will use for NettyRpcServer channels. Property name is 'hbase.netty.rpcserver.allocator'. Default is no value, which is equivalent to "pooled". Valid values are: - "pooled": use PooledByteBufAllocator - "unpooled": use UnpooledByteBufAllocator - "heap": use HeapByteBufAllocator, which is a PooledByteBufAllocator that preferentially allocates buffers on heap wherever possible - <class>: If the value is none of the recognized labels, treat it as a class name implementing org.apache.hbase.thirdparty.io.netty.buffer.ByteBufAllocator. This allows the user to add a custom implementation, perhaps for debugging. Also updates ReflectionUtils with a new helper method, and replaces a case in Compression where the new ReflectionUtils method duplicates previous code.
|
Rebase, test fix. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
virajjasani
approved these changes
Jun 23, 2022
|
|
||
| @Override | ||
| protected ByteBuf newHeapBuffer(int initialCapacity, int maxCapacity) { | ||
| LOG.info("newHeapBuffer initialCapacity={}, maxCapacity={}", initialCapacity, maxCapacity); |
There was a problem hiding this comment.
Nice, this is good for debugging
asfgit
pushed a commit
that referenced
this pull request
Jun 23, 2022
Support site configuration of the bytebuf allocator that Netty will use for NettyRpcServer channels. Property name is 'hbase.netty.rpcserver.allocator'. Default is no value, which is equivalent to "pooled". Valid values are: - "pooled": use PooledByteBufAllocator - "unpooled": use UnpooledByteBufAllocator - "heap": use HeapByteBufAllocator, which is a PooledByteBufAllocator that preferentially allocates buffers on heap wherever possible - <class>: If the value is none of the recognized labels, treat it as a class name implementing org.apache.hbase.thirdparty.io.netty.buffer.ByteBufAllocator. This allows the user to add a custom implementation, perhaps for debugging. Also updates ReflectionUtils with a new helper method. Signed-off-by: Viraj Jasani <vjasani@apache.org> Conflicts: hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestNettyRpcServer.java
asfgit
pushed a commit
that referenced
this pull request
Jun 23, 2022
Support site configuration of the bytebuf allocator that Netty will use for NettyRpcServer channels. Property name is 'hbase.netty.rpcserver.allocator'. Default is no value, which is equivalent to "pooled". Valid values are: - "pooled": use PooledByteBufAllocator - "unpooled": use UnpooledByteBufAllocator - "heap": use HeapByteBufAllocator, which is a PooledByteBufAllocator that preferentially allocates buffers on heap wherever possible - <class>: If the value is none of the recognized labels, treat it as a class name implementing org.apache.hbase.thirdparty.io.netty.buffer.ByteBufAllocator. This allows the user to add a custom implementation, perhaps for debugging. Also updates ReflectionUtils with a new helper method. Signed-off-by: Viraj Jasani <vjasani@apache.org> Conflicts: hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestNettyRpcServer.java
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Support site configuration of the bytebuf allocator that Netty will use for
NettyRpcServerchannels. We do not allow the site operator to fine tune this but could. It would be particularly useful to allow preference of heap buffers where direct memory may be limited or utilized for other purposes.New property name is
hbase.netty.rpcserver.allocator. Default is no value, which is equivalent to "pooled", which is our current default. Valid values are:pooled: use PooledByteBufAllocatorunpooled: use UnpooledByteBufAllocatorheap: use HeapByteBufAllocator, which is a PooledByteBufAllocator that preferentially allocates buffers on heap wherever possible<class>: If the value is none of the recognized labels, treat it as a class name implementingorg.apache.hbase.thirdparty.io.netty.buffer.ByteBufAllocator. This allows the user to add a custom implementation, perhaps for debugging.Also updates
ReflectionUtilswith a new helper method, and replaces a case inCompressionwhere the newReflectionUtilsmethod duplicates previous code.