Skip to content

[feat] PIP-468: Implement V5 client with scalable topic producer and consumer types#25573

Merged
merlimat merged 2 commits intoapache:masterfrom
merlimat:st-v5-client
Apr 24, 2026
Merged

[feat] PIP-468: Implement V5 client with scalable topic producer and consumer types#25573
merlimat merged 2 commits intoapache:masterfrom
merlimat:st-v5-client

Conversation

@merlimat
Copy link
Copy Markdown
Contributor

Stacked on top of #25565 (which is on top of #25564, which is on top of
#25559). Opened as a draft; will be rebased on master once the upstream PRs
land. Reviewers: the new code in this PR is the single commit titled
"PIP-468: Implement V5 client with scalable topic producer and consumer
types" — everything else comes from #25559 / #25564 / #25565.

Summary

Fourth (and final for this series) PR in PIP-468. Adds the
pulsar-client-v5 module that implements the V5 client API (PIP-466) for
scalable topics.

Core pieces

  • PulsarClientV5 + PulsarClientBuilderV5 — wraps the existing
    v4 PulsarClientImpl and adds the scalable-topic routing layer on top.
  • PulsarClientProviderV5 — ServiceLoader SPI implementation for the
    V5 API.
  • ScalableTopicProducer — key-hashing segment routing, round-robin
    for unkeyed messages, per-segment underlying v4 producers.
  • ScalableStreamConsumer — ordered, cumulative-ack consumer that
    receives segment assignments from the controller leader and drives v4
    per-segment consumers.
  • ScalableQueueConsumer — controller-bypassing consumer that
    attaches to every segment and drains them with the receive-loop fix
    from the earlier iterations.
  • ScalableCheckpointConsumer — unmanaged connector-style consumer
    that can resume from a caller-supplied Checkpoint across all
    segments.
  • DagWatchClient — client-side handle for the DAG watch session;
    implements the DagWatchSession callback from the protocol PR.
  • MessageIdV5 — V5 message-id wrapping a v4 id with a per-segment
    position vector for cumulative ack.
  • CheckpointV5 — caller-durable checkpoint including
    EARLIEST / LATEST sentinels and timestamp variants.
  • ClientSegmentLayout — client-side view of the DAG built from a
    ScalableTopicDAG protobuf.
  • SegmentRouter — 16-bit Murmur3 key-hash routing and round-robin
    for empty keys.
  • AuthenticationAdapter, CryptoKeyReaderAdapter,
    SchemaAdapter — bridge V5 API types to the underlying v4
    implementations.
  • V5Utils — small helpers (e.g. asScalableTopicName which rewraps
    a parsed name with the topic:// domain).

Supporting v4 change

  • PulsarClientImpl.createSegmentProducerAsync — internal entry point
    for the V5 client to create producers on segment:// topics,
    bypassing the scalable-domain guard that's meant for user-facing
    producer creation.

Tests

New unit test classes (48 tests)

  • V5UtilsTest (5 tests) — asScalableTopicName on bare name /
    short form / persistent:// / topic:// inputs; also pins the
    caching behavior for the topic:// fast path.
  • SegmentRouterTest (10 tests) — key-routing determinism and
    correctness (hash lands inside the picked segment's range),
    single-segment, empty-segments error, uncovered-hash error,
    round-robin cycling + wrap, hash 16-bit bound, string/bytes agree.
  • MessageIdV5Test (15 tests) — constructor + null-rejection,
    immutable position vector, equals/hashCode (position vector is
    diagnostic, not identity), compareTo by segment then v4 id, foreign-
    type rejection, byte-array roundtrip with and without position
    vector, malformed input handling, EARLIEST/LATEST sentinels,
    toString.
  • ClientSegmentLayoutTest (7 tests) — fromProto filters active
    segments, sorts by hash-range start, builds segment topic names,
    collects broker URLs, propagates the controller URL (plus TLS
    variant), and the returned collections are immutable.
  • CheckpointV5Test (11 tests) — regular checkpoint roundtrip
    with multi-segment positions, empty positions, EARLIEST/LATEST
    sentinel roundtrip (sentinel identity preserved, 1-byte compact
    wire), timestamp variant roundtrip + wire shape, null / empty /
    unknown-type input rejection, immutable positions map.

Already exercised by prior PRs in this stack

Test plan

  • ./gradlew :pulsar-client-v5:test --tests "org.apache.pulsar.client.impl.v5.*" — 48/48 new tests pass.
  • Full compile of all impacted modules
    (pulsar-client-v5, pulsar-client-original, pulsar-broker,
    pulsar-common, pulsar-client-admin-api).
  • Checkstyle clean on all modules touched.

@merlimat merlimat added area/client type/feature The PR added a new feature or issue requested a new feature labels Apr 24, 2026
@merlimat merlimat changed the title PIP-468: Implement V5 client with scalable topic producer and consumer types [feat] PIP-468: Implement V5 client with scalable topic producer and consumer types Apr 24, 2026
Copy link
Copy Markdown
Member

@lhotari lhotari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM as a starting point. Just one comment about possible improvement to explain the MessageIdV5 in the javadoc.

…r types

Add pulsar-client-v5 module implementing the V5 client API. Includes
ScalableTopicProducer with segment routing, ScalableStreamConsumer,
ScalableQueueConsumer, and ScalableCheckpointConsumer. Implement
DagWatchClient for receiving DAG layout updates from broker. Add
PulsarClientV5, builder implementations, schema/auth/crypto adapters,
and ServiceLoader-based provider. Extend PulsarClientImpl with
scalableTopicLookup support.
…r paths

The V4 client's isScalableDomain() helper only matched topic://, so the
internal segment:// domain leaked through the rejection check in
subscribeAsync(), createProducerAsync(), and createReaderAsync(). That
let segment:// topics flow into the normal connect/lookup path, making
testRejectScalableDomainOnConsumer (and its producer/reader siblings)
produce a connection-time exception instead of the expected
InvalidTopicNameException.

Update isScalableDomain() to also match TopicName.isSegment(), and fix
the comment to reflect the new behavior.
@merlimat merlimat marked this pull request as ready for review April 24, 2026 23:06
@merlimat merlimat merged commit f459a1a into apache:master Apr 24, 2026
43 checks passed
@merlimat merlimat deleted the st-v5-client branch April 24, 2026 23:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/client type/feature The PR added a new feature or issue requested a new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants