[improve][build] Upgrade lightproto gradle plugin to 0.8.0 - #26256
Closed
merlimat wants to merge 1 commit into
Closed
[improve][build] Upgrade lightproto gradle plugin to 0.8.0#26256merlimat wants to merge 1 commit into
merlimat wants to merge 1 commit into
Conversation
Assisted-by: Claude Code (Fable 5)
merlimat
marked this pull request as draft
July 31, 2026 14:44
This was referenced Sep 1, 2026
Merged
merlimat
added a commit
to streamnative/lightproto
that referenced
this pull request
Sep 1, 2026
…20) * Add large-message serialize benchmark and non-array target identity sweep LargeMessageBenchmark serializes the Pulsar topic-list shape from 600 B to 4.6 MB, a varint-dense repeated-int64 message at 2 KB and 8 KB, and a 2 MB bytes payload into pooled direct buffers. NonArrayTargetIdentityTest checks that writeTo() to direct, offset and multi-component composite targets is byte-identical to the heap-array path for sizes swept byte by byte across every plausible internal boundary, for built and parsed messages and across repeated writes. * Write direct buffers in place through their NIO view above 512 bytes Since #12, writeTo() to a non-array buffer stages the whole message in a heap byte[] scratch and bulk-copies it. Messages above SCRATCH_RETAIN_MAX (1 MiB) never retain that scratch, so every write allocated a fresh full-size array — multi-MB G1-humongous allocations that OOMed Pulsar's proxy back-pressure test (apache/pulsar#26256, together with the clear() retention fixed in #19). Below the cap the copy itself was still paid. A single-region direct buffer exposes its memory as a java.nio.ByteBuffer through ByteBuf.internalNioBuffer(). Absolute puts on a DirectByteBuffer compile to a bounds check plus a jdk.internal.misc.Unsafe store — which, unlike sun.misc.Unsafe, carries no JDK 24+ deprecation check — so the message can be written in place: no scratch array and no bulk copy, at any size. writeTo() now dispatches heap buffers in place through the backing array (unchanged), single-region direct buffers larger than NIO_WRITE_MIN (512 bytes) through the NIO view, and everything else (small messages; composites and other buffers without a single NIO region) through the scratch path as before. Above the threshold no direct-buffer write touches the scratch, so it only grows past 512 bytes for composite targets. The threshold exists because the view's per-put cost is a fixed tax per message while the copy it saves grows with size. Interleaved JMH on pooled direct buffers (JDK 21/26): the view is 15-19% slower on the ~70-byte varint-dense MessageMetadata, at parity on BaseCommand, 20% faster at 600 bytes, and 35-40% faster from 6 KB to 100 KB; on the 2 MB / 4.6 MB cases it removes the per-write allocation (-70% / -55%) and matches the per-field ByteBuf-API write-through of #18, which it replaces. The field emitters are parameterized over the write sink (WriteSink.ARRAY / WriteSink.NIO): one emitter produces both _writeTo(byte[], int) and _writeTo(ByteBuffer, int), differing only in the sink variable and in how bulk data is copied out of a ByteBuf; every raw writer in LightProtoCodec is overloaded for both sinks. NonArrayTargetIdentityTest sweeps sizes byte by byte across every boundary (64 B .. 1 MiB) on direct, offset and multi-component composite targets, for built and parsed messages, repeated strings incl. non-ASCII, bytes payloads, nested trees and the Pulsar BaseCommand shape. NioWriteTest checks the routing flips exactly at NIO_WRITE_MIN, that composites keep the scratch path, and (via ThreadMXBean.getThreadAllocatedBytes) that 5 writes of a 4.6 MB topic list or a 5 MB payload allocate less than a quarter of one message. LargeMessageBenchmark covers 600 B .. 4.6 MB topic lists, varint-dense 2 KB / 8 KB messages and a 2 MB payload.
Contributor
Author
|
Superseded by #26445: LightProto 0.8.1 fixes the two 0.8.0 regressions behind the |
11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
LightProto v0.8.0 is a systematic performance pass over the generated serialization/deserialization code, benchmarked directly on Pulsar's protocol messages (vs 0.7.3):
MessageMetadatadeserializeBaseCommanddeserializeBaseCommandserializeMessageMetadataserializeMain changes in the generated code:
clear()with presence-guarded getters (also fixes stale-child reachability afterclearX()on a message field, andcopyFrom()now follows protobuf merge semantics for implicit-presence fields)byte[]instead of per-field ByteBuf writes;writeTo(CompositeByteBuf)now worksBaseCommandwith ~50 optional sub-commands)sun.misc.Unsafefrom the write hot loop and the ASCII scan (JDK 24+ taxes each call, and it is on the removal track)Modifications
lightprotofrom 0.7.3 to 0.8.0 ingradle/libs.versions.toml; all modules applying the plugin pick it up via the version catalog.