LastMessageID, marker Messages and ReplicateSubscriptionState

[LNA-DC]

We're in the process of establishing a geo-replicated cluster. Because we want to replicate both topics and Subscriptions we have enabled ReplicateSubscriptionState on our subscriptions.

When enabling ReplicateSubscriptionState we noticed that last message id didn't match the ID of the last produced message. I have figured this is due to som "marker messages" used for sending snapshots across clusters, and these messages are (by default) send every 1000ms ..
This is fairly covered by this article: https://github.com/apache/pulsar/wiki/PIP-33%3A-Replicated-subscriptions
So far so good.

My problem here is:
These marker messages are not send to the consumer, but they are still seen as a real message by pulsar and they are assigned a message ID.
Say you have a Topic. with 10 messages - Three real messages, and seven of these marker messages

1. "Hello"
2. (a marker message, not send to the consumer)
3. (a marker message, not send to the consumer)
4. "Word"
5. (a marker message, not send to the consumer)
6. (a marker message, not send to the consumer)
7. "FooBar"
8. (a marker message, not send to the consumer)
9. (a marker message, not send to the consumer)
10. (a marker message, not send to the consumer)

If I ask pulsar to give me the last messageID it will say: 10
This makes perfect sense.

However; seen from an application message 7 should be the last one. On a normal consumer this isn't a problem. After all; it will just consume message in the same order they are produced.

But say you want to create a reader and seek to a specific message, or you want to reset the cursor in a subscription to a specific message ID.
An example of this could be if a reader needs to consume last produced message. Normally I'd run a GetLastMessageID and Seek to that messageID. The result of this; the reader will just sit there and wait for a message the broker will never send.
This is more or less impossible to do and I really can't find any reliable way to do this.

Anybody out there who found a way around this?

[barryhatfield]

• 1
  Following