Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription. #6554

codelipenghui · 2020-03-19T02:25:38Z

Master issue #4077
Is your feature request related to a problem? Please describe.
Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription.

feeblefakie · 2020-03-31T05:29:36Z

Let me add a related issue and some discussion I and @codelipenghui had about this.

Key_Shared consumers with different subscriptions get messages for the same key in an inconsistent order
https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1585527971073400

I think sticky consumers won't really solve it since consumers can be easily down in failure cases and it is pretty hard for applications to manage scalability of consumers.

poulhenriksen · 2020-04-03T10:55:47Z

How come this is marked as a feature and not a bug?

It violates the ordering guarantee of Key_Shared, and basically makes Key_Shared useless if you dynamically add consumers.

I ask, because I would like to ensure this bug gets the proper attention, and is not regarded as a nice-to-have feature.

feeblefakie · 2020-04-14T03:53:25Z

@codelipenghui Could you share updates on this if any ?
Sorry for pushing, but I feel it is a really critical bug and Key_Shared users will have so much troubles if they just use it without doubt.

codelipenghui · 2020-04-14T04:05:57Z

@feeblefakie We are already advancing this matter and I have drafted an approach. I will create a PIP or push a PR soon.

feeblefakie · 2020-04-14T04:59:51Z

@codelipenghui Thank you for sharing the update ! Good to know. :)

merlimat · 2020-04-23T01:26:12Z

So, I think the problem here should be treated by having new consumers joining in a "paused" state.

For example, consider this sequence:

Subscriptions has c1 and c2 consumers
c3 joins. Some of the keys are now supposed to go to c3.
Instead of starting delivering to c3. We mark the current readPosition (let's call it rp0_c3) of the cursor for c3.
Any message that now hashes to c3 and that has messageId >= rp0_c3 will be deferred for later re-delivery
Any message that might get re-delivered (eg: originally sent to c1, but c1 has failed) to c3 and that has messageId < rp0_c3 will be sent to c3
When the markDelete position of the cursor will move past rp0_c3 the restriction on c3 will be lifted.

Essentially, c3 joins but can only receive old messages, until everything that was read before joining gets acked.

feeblefakie · 2020-04-23T03:41:49Z

@merlimat
I'm not fully sure about the solution but does it guarantee key ordering ?
For example, a message X with key "A" is first consumed in c1, and another message Y with key "A" can possibly go to c3 ? (can we make sure message X is consumed and acked before message Y ?)

merlimat · 2020-04-23T04:21:41Z

For example, a message X with key "A" is first consumed in c1, and another message Y with key "A" can possibly go to c3 ? (can we make sure message X is consumed and acked before message Y ?)

Yes, the above proposal takes care of that.

The important point is that X could be re-sent to c3, in case c1 fails just after c3 has joined.

codelipenghui · 2020-04-23T06:07:49Z

@merlimat I like the approach you have mentioned.

codelipenghui · 2020-05-19T02:05:05Z

@pouledodue @feeblefakie @merlimat I have pushed a PR #6977 for this issue. Please help review, thanks.

) ### Motivation Fixes: #6554 Ordering is broken in KeyShared dispatcher if a new consumer `c2` comes in and an existing consumer `c1` goes out. This is because messages with keys previously assigned to `c1` may now route to `c2`. The solution here is to have new consumers joining in a "paused" state. For example, consider this sequence: 1. Subscriptions has `c1` and `c2` consumers 2. `c3` joins. Some of the keys are now supposed to go to `c3`. 3. Instead of starting delivering to `c3`. We mark the current readPosition (let's call it `rp0_c3`) of the cursor for `c3`. 4. Any message that now hashes to `c3` and that has `messageId >= rp0_c3` will be deferred for later re-delivery 5. Any message that might get re-delivered (eg: originally sent to `c1`, but `c1` has failed) to `c3` and that has `messageId < rp0_c3` will be sent to `c3` 6. When the markDelete position of the cursor will move past `rp0_c3` the restriction on `c3` will be lifted. Essentially, `c3` joins but can only receive old messages, until everything that was read before joining gets acked.

codelipenghui · 2020-06-16T05:42:22Z

Reopen it via #5819 (comment).

rocketraman · 2020-07-02T21:21:36Z

PR #7106 referred to by #5819 (comment) is marked as closed and merged, with milestone 2.6.0. Should this issue be closed with 2.6.0 milestone after all?

codelipenghui · 2020-07-03T00:40:24Z

@rocketraman As discussed with @feeblefakie in slack channel, looks there are some problems with auto split mechanism. So I reopen this issue for tracking this problem. And @feeblefakie has verified consistent hash mechanism is work as expected. I will take a look at the auto split mechanism later.

feeblefakie · 2020-07-03T01:08:54Z

@codelipenghui Should I create an open issue for this ?
I realized that I just created how to reproduce doc in my repo and shared it in only the closed issue.

codelipenghui · 2020-07-03T11:24:10Z

@feeblefakie Yes, thanks.

feeblefakie · 2020-07-06T00:24:01Z

@codelipenghui #7455 PTAL.

…ache#7106) ### Motivation Fixes: apache#6554 Ordering is broken in KeyShared dispatcher if a new consumer `c2` comes in and an existing consumer `c1` goes out. This is because messages with keys previously assigned to `c1` may now route to `c2`. The solution here is to have new consumers joining in a "paused" state. For example, consider this sequence: 1. Subscriptions has `c1` and `c2` consumers 2. `c3` joins. Some of the keys are now supposed to go to `c3`. 3. Instead of starting delivering to `c3`. We mark the current readPosition (let's call it `rp0_c3`) of the cursor for `c3`. 4. Any message that now hashes to `c3` and that has `messageId >= rp0_c3` will be deferred for later re-delivery 5. Any message that might get re-delivered (eg: originally sent to `c1`, but `c1` has failed) to `c3` and that has `messageId < rp0_c3` will be sent to `c3` 6. When the markDelete position of the cursor will move past `rp0_c3` the restriction on `c3` will be lifted. Essentially, `c3` joins but can only receive old messages, until everything that was read before joining gets acked.

wolfstudy · 2020-08-14T17:13:08Z

The issue will be tracked in #7455 so move the issue to release/2.6.2

…ache#7106) ### Motivation Fixes: apache#6554 Ordering is broken in KeyShared dispatcher if a new consumer `c2` comes in and an existing consumer `c1` goes out. This is because messages with keys previously assigned to `c1` may now route to `c2`. The solution here is to have new consumers joining in a "paused" state. For example, consider this sequence: 1. Subscriptions has `c1` and `c2` consumers 2. `c3` joins. Some of the keys are now supposed to go to `c3`. 3. Instead of starting delivering to `c3`. We mark the current readPosition (let's call it `rp0_c3`) of the cursor for `c3`. 4. Any message that now hashes to `c3` and that has `messageId >= rp0_c3` will be deferred for later re-delivery 5. Any message that might get re-delivered (eg: originally sent to `c1`, but `c1` has failed) to `c3` and that has `messageId < rp0_c3` will be sent to `c3` 6. When the markDelete position of the cursor will move past `rp0_c3` the restriction on `c3` will be lifted. Essentially, `c3` joins but can only receive old messages, until everything that was read before joining gets acked.

wolfstudy · 2020-10-31T03:00:41Z

It seems that #8292 has solved this problem, let us consider closing this issue.

codelipenghui added the type/feature The PR added a new feature or issue requested a new feature label Mar 19, 2020

codelipenghui mentioned this issue Mar 19, 2020

PIP-34 New subscription type Key_Shared #4077

Closed

12 tasks

codelipenghui changed the title ~~Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages~~ Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription. Mar 19, 2020

sijie mentioned this issue Mar 19, 2020

ISSUE-6554: Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription. streamnative/pulsar-archived#815

Closed

sijie added triage/week-13 area/broker labels Mar 26, 2020

codelipenghui self-assigned this Apr 3, 2020

codelipenghui added this to the 2.6.0 milestone Apr 14, 2020

codelipenghui added the release/2.5.2 label Apr 14, 2020

codelipenghui mentioned this issue Apr 22, 2020

Use consistent hashing in KeyShared distribution #6791

Merged

merlimat added type/bug The PR fixed a bug or issue reported a bug and removed type/feature The PR added a new feature or issue requested a new feature labels Apr 23, 2020

jiazhai removed the release/2.5.2 label May 13, 2020

codelipenghui mentioned this issue May 18, 2020

Introduce messages fence mechanism for Key_Shared subscription #6977

Closed

merlimat mentioned this issue May 29, 2020

Fixed ordering issue in KeyShared dispatcher when adding consumer #7106

Merged

codelipenghui closed this as completed in #7106 Jun 5, 2020

feeblefakie mentioned this issue Jun 16, 2020

[discussion] Pulsar release 2.6.0 #5819

Closed

codelipenghui reopened this Jun 16, 2020

sijie mentioned this issue Jun 16, 2020

ISSUE-6554: Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription. streamnative/pulsar-archived#1098

Closed

codelipenghui modified the milestones: 2.6.0, 2.7.0 Jun 18, 2020

codelipenghui added the release/2.6.1 label Jun 18, 2020

feeblefakie mentioned this issue Jul 6, 2020

Key_Shared consumers with different subscriptions get messages out-of-order in 2.6.0. #7455

Closed

sijie mentioned this issue Jul 6, 2020

ISSUE-7455: Key_Shared consumers with different subscriptions get messages out-of-order in 2.6.0. streamnative/pulsar-archived#1190

Closed

wolfstudy added release/2.6.2 and removed release/2.6.1 labels Aug 14, 2020

wolfstudy closed this as completed Oct 31, 2020

codelipenghui mentioned this issue May 22, 2022

New KeyShared consumers will not get any messages until a consumer that did get messages disconnects or acks/nacks some messages #15705

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription. #6554

Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription. #6554

codelipenghui commented Mar 19, 2020 •

edited

Loading

feeblefakie commented Mar 31, 2020

poulhenriksen commented Apr 3, 2020

feeblefakie commented Apr 14, 2020

codelipenghui commented Apr 14, 2020 •

edited

Loading

feeblefakie commented Apr 14, 2020

merlimat commented Apr 23, 2020

feeblefakie commented Apr 23, 2020

merlimat commented Apr 23, 2020

codelipenghui commented Apr 23, 2020

codelipenghui commented May 19, 2020 •

edited

Loading

codelipenghui commented Jun 16, 2020

rocketraman commented Jul 2, 2020

codelipenghui commented Jul 3, 2020

feeblefakie commented Jul 3, 2020 •

edited

Loading

codelipenghui commented Jul 3, 2020

feeblefakie commented Jul 6, 2020

wolfstudy commented Aug 14, 2020

wolfstudy commented Oct 31, 2020

Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription. #6554

Dispatch messages to the new consumer after exist consumer acknowledged all dispatched messages in Key_Shared Subscription. #6554

Comments

codelipenghui commented Mar 19, 2020 • edited Loading

feeblefakie commented Mar 31, 2020

poulhenriksen commented Apr 3, 2020

feeblefakie commented Apr 14, 2020

codelipenghui commented Apr 14, 2020 • edited Loading

feeblefakie commented Apr 14, 2020

merlimat commented Apr 23, 2020

feeblefakie commented Apr 23, 2020

merlimat commented Apr 23, 2020

codelipenghui commented Apr 23, 2020

codelipenghui commented May 19, 2020 • edited Loading

codelipenghui commented Jun 16, 2020

rocketraman commented Jul 2, 2020

codelipenghui commented Jul 3, 2020

feeblefakie commented Jul 3, 2020 • edited Loading

codelipenghui commented Jul 3, 2020

feeblefakie commented Jul 6, 2020

wolfstudy commented Aug 14, 2020

wolfstudy commented Oct 31, 2020

codelipenghui commented Mar 19, 2020 •

edited

Loading

codelipenghui commented Apr 14, 2020 •

edited

Loading

codelipenghui commented May 19, 2020 •

edited

Loading

feeblefakie commented Jul 3, 2020 •

edited

Loading