Optimize pipe request serialization buffer sizing - #18233
Merged
jt2594838 merged 7 commits intoJul 28, 2026
Conversation
luoluoyuyu
force-pushed
the
pipe-tablet-serialization-size
branch
from
July 17, 2026 06:45
44dc635 to
793b0a6
Compare
jt2594838
reviewed
Jul 20, 2026
jt2594838
left a comment
jt2594838
left a comment
Contributor
There was a problem hiding this comment.
May add some performance results.
jt2594838
reviewed
Jul 23, 2026
Comment on lines
+284
to
+298
| private static void clearDataRegionReplicaSet(final InsertRowNode insertRowNode) { | ||
| insertRowNode.setDataRegionReplicaSet(null); | ||
| } | ||
|
|
||
| private static void clearDataRegionReplicaSet(final InsertRowsNode insertRowsNode) { | ||
| insertRowsNode.setDataRegionReplicaSet(null); | ||
| for (final InsertRowNode insertRowNode : insertRowsNode.getInsertRowNodeList()) { | ||
| clearDataRegionReplicaSet(insertRowNode); | ||
| } | ||
| } | ||
|
|
||
| private static void clearDataRegionReplicaSet(final InsertTabletNode insertTabletNode) { | ||
| insertTabletNode.setDataRegionReplicaSet(null); | ||
| } | ||
|
|
Contributor
There was a problem hiding this comment.
May add planNodeTree.clearUselessFieldAfterDispatch here.
jt2594838
approved these changes
Jul 27, 2026
Member
Author
This optimization pre-calculates the required PublicBAOS capacity before serialization and allocates it once, avoiding repeated Arrays.copyOf calls caused by dynamic buffer growth for large requests.
The change does not alter the data format or processing logic for Insert or Pipe transfers. It reduces transient memory peaks and array-copying overhead caused by the simultaneous retention of old and new byte[] buffers during expansion. The optimization covers serialization for Insert write requests and Pipe Tablet/InsertNode requests sent to sinks, with greater benefits for large Tablets, multi-Tablet requests, and high-volume writes.
Benchmark results show that young-generation GC frequency can be reduced by approximately 1.2x to 2x, indicating fewer short-lived array allocations during serialization. The final request payload still needs to be retained; this optimization targets the extra temporary memory introduced by buffer expansion, rather than the necessary memory footprint of the request data itself.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR has:
for an unfamiliar reader.
for code coverage.
Key changed/added classes (or packages if there are too many classes) in this PR