[improve][build] Upgrade LightProto to 0.6.1

[merlimat]

Motivation

Upgrade the LightProto serialization library from 0.4 to 0.6.1. LightProto is used in Pulsar for high-performance, zero-copy serialization of internal protocol buffers (PulsarApi, transaction metadata, delayed message indexes, resource usage).

LightProto 0.6.x highlights

Full release notes: https://github.com/streamnative/lightproto/releases/tag/v0.6.0

Serialization performance overhaul

Unsafe-based serialization, pre-ensureWritable in writeTo() to eliminate per-field capacity checks, ASCII fast-path for string serialization, native arrays replacing ArrayList for repeated fields, precomputed tag sizes, unrolled VarInt64 decoding, and cached serialized size after parseFrom.

Performance: v0.4 → v0.6

Benchmark	v0.4	v0.6	Speedup
PulsarAPI SerializeBaseCommand	12.4	26.8	+116%
PulsarAPI SerializeMessageMetadata	6.1	10.9	+79%
PulsarAPI DeserializeBaseCommand	39.1	40.3	+3%
PulsarAPI DeserializeMessageMetadata	14.0	14.1	~same
ProtoBenchmark Serialize	8.3	22.6	+172%
ProtoBenchmark FillAndSerialize	5.9	12.8	+117%
ProtoBenchmark Deserialize	16.7	19.2	+15%

LightProto v0.6 vs Google Protobuf (PulsarAPI)

Benchmark	Protobuf	LightProto	Speedup
Serialize MessageMetadata	2.84	10.85	3.8x
Serialize BaseCommand	15.82	26.81	1.7x
Deserialize MessageMetadata	4.18	14.14	3.4x
Deserialize BaseCommand	12.77	40.30	3.2x

Proto3 support

Full proto3 syntax support including implicit field presence, packed-by-default repeated fields, and the optional keyword. This opens up the opportunity to standardize more of Pulsar's serialization on LightProto. Currently, Pulsar uses a mix of Google Protobuf (with its higher allocation overhead) and LightProto. With proto3 support, additional proto definitions — such as those used in the client-broker protocol or managed ledger metadata — could be migrated to LightProto, reducing GC pressure and improving throughput in the hot path.

New field types & features

• map<K, V> fields — complete support for protobuf map fields
• oneof fields — proto2 oneof support with efficient memory layout
• gRPC service stubs — generate *Grpc.java with zero-copy Netty marshaller
• Parser rewrite — replaced protostuff-parser with protoc descriptor sets
• Javadoc generation from proto source comments

Changes in this PR

• Update plugin groupId from com.github.splunk.lightproto to io.streamnative.lightproto
• Bump version from 0.4 to 0.6.1
• Remove the properties-maven-plugin configuration that was only needed for the old LightProto's protostuff-based parser
• Add extraProtoPaths configuration for the transaction coordinator module to resolve cross-module proto imports natively
• Update SpotBugs exclusions for generated code

Documentation

• doc-not-needed

Matching PR in

• https://github.com/streamnative/lightproto

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.76%. Comparing base (3936ce4) to head (3706e8b).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #25332      +/-   ##
============================================
+ Coverage     72.25%   72.76%   +0.51%     
- Complexity    33963    34254     +291     
============================================
  Files          1954     1954              
  Lines        154639   154691      +52     
  Branches      17698    17700       +2     
============================================
+ Hits         111732   112565     +833     
+ Misses        33884    33100     -784     
- Partials       9023     9026       +3     

Flag	Coverage Δ
inttests	25.74% <ø> (-0.06%)	⬇️
systests	22.50% <ø> (-0.05%)	⬇️
unittests	73.74% <ø> (+0.57%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 146 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[lhotari]

One concern about lightproto 0.6.1 is that the optimized String handling might not be compatible with -XX:-CompactStrings. I'm not sure if anyone would be using that setting, but it's a valid JVM config. Perhaps lightproto should fallback to the slow path when compact strings are disabled?
btw. should we enable issues for https://github.com/streamnative/lightproto?

[merlimat]

Good point. I've enabled issues now.

[merlimat]

@lhotari created PR: streamnative/lightproto#1