[improve][build] Upgrade lightproto to 0.8.1 - #26445
Merged
Merged
Conversation
LightProto 0.8.1 fixes the two 0.8.0 regressions that made the 0.8.0 bump (apache#26256) run out of heap in ProxyPatternConsumerBackPressureMultipleConsumersTest: serialization of large messages to direct buffers no longer allocates a full-size heap array per write (messages above 512 bytes are written in place through the buffer's NIO view), and clear() releases the previous message's data on reused instances such as the per-connection BaseCommand in PulsarDecoder. The 0.8.0 serde gains over 0.7.3 are retained.
lhotari
approved these changes
Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Supersedes #26256, which bumped LightProto to 0.8.0 and failed CI:
ProxyPatternConsumerBackPressureMultipleConsumersTestran out of heap. That exposed two regressions in the 0.8.0 generated code, both fixed in LightProto v0.8.1:byte[]scratch before copying it into the target buffer; above 1 MiB the scratch was never retained, so every ~4.6 MBCommandGetTopicsOfNamespaceResponsecost a fresh multi-MB (G1 humongous) allocation — invisible to the broker's and proxy'sAsyncDualMemoryLimiter, which accounts these responses as direct memory. 0.8.1 writes messages above 512 bytes in place through the direct buffer's NIO view: no scratch array, no copy, no allocation at any size.clear()(Release data references in clear() for messages above CLEAR_RETAIN_MAX streamnative/lightproto#19). 0.8.0's O(1)clear()only reset counts and presence bits, so the per-connectionBaseCommandinPulsarDecoderkept the last topic list (~4.6 MB) alive on every connection that had ever received one. 0.8.1 releases those references for messages above 64 KiB.With 0.8.1 the test passes locally (500/500 requests, free-heap floor >50 MB, where 0.8.0 ran out of heap at 275/500).
The performance gains that motivated the upgrade are retained. Measured on Pulsar's protocol messages against 0.7.3 (JDK 26, interleaved A/B, time per operation):
BaseCommandserializeMessageMetadataserializeBaseCommanddeserializeMessageMetadatadeserializeMain changes in the generated code since 0.7.3: O(1)
clear()with presence-guarded getters, presence-bit-driven traversal for union-like messages such asBaseCommand, array/NIO-view serialization with nosun.misc.Unsafein the hot loops (JDK 24+ taxes each call), unchecked Netty reads for 64-bit varints.Modifications
lightprotofrom 0.7.3 to 0.8.1 ingradle/libs.versions.toml; all modules applying the plugin pick it up via the version catalog.Verifying this change
ProxyPatternConsumerBackPressureMultipleConsumersTest(the test that OOMed on 0.8.0) passes locally on this branch with the released 0.8.1 artifacts.Does this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes
io.streamnative.lightproto0.7.3 → 0.8.1 (code generator and Gradle plugin; the generated protocol classes change, the wire format does not)