Skip to content

[improve][build] Upgrade lightproto to 0.8.1 - #26445

Merged
merlimat merged 1 commit into
apache:masterfrom
merlimat:mmerli/lightproto-0.8.1
Sep 2, 2026
Merged

[improve][build] Upgrade lightproto to 0.8.1#26445
merlimat merged 1 commit into
apache:masterfrom
merlimat:mmerli/lightproto-0.8.1

Conversation

@merlimat

@merlimat merlimat commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Motivation

Supersedes #26256, which bumped LightProto to 0.8.0 and failed CI: ProxyPatternConsumerBackPressureMultipleConsumersTest ran out of heap. That exposed two regressions in the 0.8.0 generated code, both fixed in LightProto v0.8.1:

  • Large messages serialized to direct buffers allocated a full-size heap array per write (Write direct buffers in place through their NIO view above 512 bytes streamnative/lightproto#20). 0.8.0 composed every message in a heap byte[] scratch before copying it into the target buffer; above 1 MiB the scratch was never retained, so every ~4.6 MB CommandGetTopicsOfNamespaceResponse cost a fresh multi-MB (G1 humongous) allocation — invisible to the broker's and proxy's AsyncDualMemoryLimiter, which accounts these responses as direct memory. 0.8.1 writes messages above 512 bytes in place through the direct buffer's NIO view: no scratch array, no copy, no allocation at any size.
  • Reused message instances retained the previous message's data after clear() (Release data references in clear() for messages above CLEAR_RETAIN_MAX streamnative/lightproto#19). 0.8.0's O(1) clear() only reset counts and presence bits, so the per-connection BaseCommand in PulsarDecoder kept the last topic list (~4.6 MB) alive on every connection that had ever received one. 0.8.1 releases those references for messages above 64 KiB.

With 0.8.1 the test passes locally (500/500 requests, free-heap floor >50 MB, where 0.8.0 ran out of heap at 275/500).

The performance gains that motivated the upgrade are retained. Measured on Pulsar's protocol messages against 0.7.3 (JDK 26, interleaved A/B, time per operation):

v0.7.3 v0.8.1 Δ
BaseCommand serialize 52.3 ns 23.0 ns −56%
MessageMetadata serialize 81.2 ns 70.4 ns −13%
BaseCommand deserialize 28.5 ns 21.9 ns −23%
MessageMetadata deserialize 72.5 ns 43.0 ns −41%
4.6 MB topic-list response serialize 175 µs 166 µs −5% (0 B allocated)

Main changes in the generated code since 0.7.3: O(1) clear() with presence-guarded getters, presence-bit-driven traversal for union-like messages such as BaseCommand, array/NIO-view serialization with no sun.misc.Unsafe in the hot loops (JDK 24+ taxes each call), unchecked Netty reads for 64-bit varints.

Modifications

  • Bump lightproto from 0.7.3 to 0.8.1 in gradle/libs.versions.toml; all modules applying the plugin pick it up via the version catalog.

Verifying this change

  • Make sure that the change passes the CI checks.

ProxyPatternConsumerBackPressureMultipleConsumersTest (the test that OOMed on 0.8.0) passes locally on this branch with the released 0.8.1 artifacts.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency): io.streamnative.lightproto 0.7.3 → 0.8.1 (code generator and Gradle plugin; the generated protocol classes change, the wire format does not)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

LightProto 0.8.1 fixes the two 0.8.0 regressions that made the 0.8.0 bump
(apache#26256) run out of heap in ProxyPatternConsumerBackPressureMultipleConsumersTest:
serialization of large messages to direct buffers no longer allocates a
full-size heap array per write (messages above 512 bytes are written in
place through the buffer's NIO view), and clear() releases the previous
message's data on reused instances such as the per-connection BaseCommand
in PulsarDecoder. The 0.8.0 serde gains over 0.7.3 are retained.
@lhotari lhotari added this to the 5.0.0-M2 milestone Sep 1, 2026
@merlimat
merlimat merged commit 19c69ee into apache:master Sep 2, 2026
43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants