Skip to content

[Bug] Replicator subscriptions are unusable when the remote cluster name contains a dot #26450

Description

@SEPURI-SAI-KRISHNA

Search before reporting

  • I searched in the issues and found nothing similar.

Read release policy

  • I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

User environment

  • Broker version: 5.0.0-M1-SNAPSHOT, built from master at commit af54f64e18
  • Broker OS / hardware: Linux 6.8.0-84-generic #84-Ubuntu SMP PREEMPT_DYNAMIC x86_64 (Ubuntu 24.04)
  • Broker Java version: openjdk 21.0.12 2026-07-21 (OpenJDK 64-Bit Server VM, 21.0.12+8-1-24.04-Ubuntu)
  • Client library type: Java (pulsar-admin / PulsarAdmin)
  • Client library version: same build, 5.0.0-M1-SNAPSHOT
  • Client OS / hardware and Java version: as above, same machine

This is a code-level defect in the broker rather than an environment-specific one — it reproduces on any broker whose replicated cluster name contains a ., independently of OS, JVM or client language, since the affected parsing is broker-side. It is confirmed by a broker test (see Reproducing the issue).

Issue Description

Cluster names are allowed to contain .NamedEntity.NAMED_ENTITY_PATTERN is ^[-=:.\w]*$, and the comment above it states this applies to "property, namespace, cluster and topic names".

AbstractReplicator builds replicator cursor/subscription names as <replicatorPrefix>.<remoteCluster> in getReplicatorName, but recovers the cluster by splitting on . and taking the last segment (pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractReplicator.java:474):

public static String getRemoteCluster(String remoteCursor) {
    String[] split = remoteCursor.split("\\.");
    return split[split.length - 1];
}

The two are therefore not inverses. For a cluster named remote.east, the cursor pulsar.repl.remote.east resolves to east.

Expected: the cluster name recovered from a replicator cursor is the one that was used to build it, so replicator subscriptions behave the same whether or not the cluster name contains a dot.

Actual: two distinct failures.

1. Every admin operation addressed at a replicator subscription fails.

Six call sites use this parse, then look the replicator up under the wrong cluster name and get null (a seventh, the orphan-cursor sweep, is covered below):

Call site Operation Result
PersistentTopicsBase.java:1982 clear-backlog on a replicator subscription 404 Replicator not found
PersistentTopicsBase.java:2037 skip messages 404 Replicator not found
PersistentTopicsBase.java:4213 expire messages (by position) 404 Replicator not found
PersistentTopicsBase.java:4329 expire messages (by timestamp) 404 Replicator not found
PersistentTopicsBase.java:4767 getReplicatorReference — peek, replicator stats 404 Replicator not found
NamespacesBase.java:2304 namespace-wide clear-backlog for a replicator subscription wrong subscription targeted

Each of these guards with subName.startsWith(replicatorPrefix) on the line immediately above, so the subscription is recognised as a replicator's — only the cluster it belongs to is derived wrongly.

2. The orphan-cursor sweep misidentifies live replicators.

PersistentTopic.removeOrphanReplicationCursors() (PersistentTopic.java:559) runs inside initialize(), i.e. on every topic load:

  1. getRemoteCluster("pulsar.repl.remote.east") returns east (PersistentTopic.java:564).
  2. east is not among the topic's replication clusters, so a live, correctly-configured replicator is declared orphaned and removeReplicator("east") is called (PersistentTopic.java:569).
  3. removeReplicator rebuilds the name as pulsar.repl.east and calls asyncDeleteCursor on it (PersistentTopic.java:2496-2500). No such cursor exists, so ManagedLedgerImpl.java:1092-1095 fails the callback with CursorNotFoundException, and the initialize() chain fails at PersistentTopic.java:509.

The cursor survives only because the reconstructed name is wrong too — the sweep fully intends to delete a cursor that is not orphaned. #22890 documents that wrongly removing a replicator cursor loses the entire replication backlog, so this sits on a code path already known to be destructive when it misfires. The sweep was introduced by #19972 / #22890.

I believe this is a bug rather than intended behaviour because getReplicatorName and getRemoteCluster are documented and used as inverses of one another, and dots are explicitly legal in cluster names.

Error messages

Logged on every load of an affected topic:

WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - Remove the orphan replicator because the cluster does not exist  remoteCluster=east
ERROR org.apache.pulsar.broker.service.persistent.PersistentTopic - Failed to delete cursor  name=pulsar.repl.east
      org.apache.bookkeeper.mledger.ManagedLedgerException$CursorNotFoundException: ManagedCursor not found: pulsar.repl.east

Admin side, for a replicator subscription that plainly exists in topics stats:

Replicator not found
Reason: HTTP 404 Not Found

Reproducing the issue

Any cluster whose name contains a . and participates in geo-replication reproduces this. It reproduces on Pulsar standalone.

  1. Register a cluster with a dot in its name, e.g. remote.east, and add it to the tenant's allowed clusters.
  2. Set it as a replication cluster on a namespace, and create a topic there.
  3. Confirm the replicator exists — pulsar-admin topics stats persistent://tenant/ns/topic lists a remote.east replicator, and pulsar-admin topics subscriptions ... shows pulsar.repl.remote.east.
  4. Address any admin operation at it:
pulsar-admin topics skip -s pulsar.repl.remote.east -n 1 persistent://tenant/ns/topic

Observed: Replicator not found (HTTP 404).
Expected: the operation applies to the remote.east replicator.

Reloading the topic (unload, or restart the broker) additionally logs the orphan-sweep warning and the failed cursor deletion shown above.

The same steps with a cluster named without a dot, e.g. remote, all succeed — which isolates the cause to the name parsing.

The orphan-sweep half also reproduces as a broker test: create a topic with a live replicator cursor pulsar.repl.remote.east for cluster remote.east, call PersistentTopic#initialize(), and observe the Remove the orphan replicator warning for remoteCluster=east.

Additional information

The fix is to make getRemoteCluster the exact inverse of getReplicatorName by stripping the known replicatorPrefix rather than splitting on .. All seven call sites already guard with startsWith(replicatorPrefix) immediately above, so the prefix is in scope at every one and no plumbing is needed beyond passing it in.

Fixing the parsing is preferable to rejecting dotted cluster names: deployments may already use them, and turning those into a validation error would break working clusters on upgrade.

Per pip/README.md, this is a bug-fix and should not need a PIP: no public API, configuration, metric, or wire-format change, and no *ClassName configuration selects a Replicator implementation, so AbstractReplicator is not a plugin/SPI extension point.

Related: #22890 and #19972 introduced the orphan-cursor sweep.

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions