-
Notifications
You must be signed in to change notification settings - Fork 24.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Serialize outbound messages on netty buffers #80111
Conversation
Pinging @elastic/es-distributed (Team:Distributed) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I only had a high-level look but it looks good. Being closer to the byte[] where the data is written and using the VarHandles will hopefully make writing data faster.
NetworkStreamOutput looks to me as a better BytesStreamOutput in many cases. E.g. I believe it would also be a better fit than BytesStreamOutput in GeometryDocValuesWriter? If so maybe we should give it a more generic name that doesn't sound like it's only a good fit for network messages?
server/src/main/java/org/elasticsearch/common/io/stream/NetworkStreamOutput.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/common/io/stream/NetworkStreamOutput.java
Outdated
Show resolved
Hide resolved
@jpountz - I made your changes. As I mentioned offline, the stream I created Currently, Or I could look into that in a follow-up. |
Thanks for explaining. Looking into replacing BytesStreamOutput as a follow-up sounds good to me as it will make it easier to evaluate the potential downsides for networking. |
server/src/main/java/org/elasticsearch/common/io/stream/RecyclerBytesStreamOutput.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM on the code changes this looks really nice and IMO is 100% the right thing to do.
The one thing I'm a little worried about is the change in behavior for very large transport messages.
Previously, the BigArrays
would eventually stop pooling bytes and move to un-pooled allocation after going over the configured heap threshold, this isn't the case any longer with this change and we'd potentially pool a lot more memory than we used to after a sporadic spike in required buffers (thinking about things like servicing a CS request for stats with a large cluster state or so). What is Netty's behavior here in terms of downsizing the buffer pool again after dealing with a message of hundreds of MB on a 2G master for example?
Netty immediately releases PoolChunks when they are no longer utilized. PoolChunks are larger that 16KB. However, in the case of large messages you are discussing, there will be minimal waste. With that said, we have obviously implemented our own recycler here that is used for inbound decompression and outbound messages. If we are concerned about overuse, we can add a counter and fallback on JVM allocations once it crosses a certain threshold. We can discuss. |
Today we allocate memory for the cluster state to be sent in a publication using `BigArrays`, but really this memory usage is network-related so (as in elastic#80111) we should allocate its memory via the recycler used in the transport layers (i.e. Netty's allocator) instead. This commit does that.
Today we allocate memory for the cluster state to be sent in a publication using `BigArrays`, but really this memory usage is network-related so (as in #80111) we should allocate its memory via the recycler used in the transport layers (i.e. Netty's allocator) instead. This commit does that.
Same as elastic#80111 but for the REST layer.
Same as #80111 but for the REST layer.
Currently we use BigArrays for serializing outbound messages. Since
these messages are only handled at the network layer, it is better to
use the Netty allocator when using the netty transport. This commit
implements this serialization method when netty is enabled.