Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broker considers fail-over consumer priority-level #2954

Merged
merged 2 commits into from
Mar 5, 2019

Conversation

rdhabalia
Copy link
Contributor

Motivation

Right now, Broker selects failover consumer based on lexicographical sorting on consumer name. However, one of the user has usecase to switch fail-over consumer (based on hot/cold colo)in a deterministic way without changing/managing consumer-name. Consumer already has priority-level but it's not implemented in Failover consumer which can be useful in certain usecase.

Modifications

Broker also considers consumer's priority-level while selecting active consumer.

@rdhabalia rdhabalia added this to the 2.2.1 milestone Nov 7, 2018
@rdhabalia rdhabalia self-assigned this Nov 7, 2018
Copy link
Contributor

@merlimat merlimat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Can you also clarify on the ConsumerBuilder javadoc how the priority is being used?

@rdhabalia
Copy link
Contributor Author

Can you also clarify on the ConsumerBuilder javadoc how the priority is being used?

yes, added.

@rdhabalia
Copy link
Contributor Author

run java8 tests

@merlimat
Copy link
Contributor

merlimat commented Nov 8, 2018

@rdhabalia One more question: how would this impact the assignment across multiple partitions?

@rdhabalia
Copy link
Contributor Author

how would this impact the assignment across multiple partitions?

priority-level gets configured at consumer level so, each partitioned-consumer will present its priority-level to broker and broker chooses active consumer for each partition. Each partitioned will be a separate topic so, there will not be any behavior difference in case of multiple partitions.

@merlimat
Copy link
Contributor

merlimat commented Nov 8, 2018

The question is that broker is using ordinal position for consumers to ensure even distribution.

Simple example: 4 partitions, 2 consumers. With equal priorities, they

partition consumer
t-partition-0 c-1
t-partition-1 c-2
t-partition-2 c-1
t-partition-3 c-2

If I set c-1 with higher priority, I would expect to have c1 on all the partitions. however the priority will only affect the sorting, so each partition will see the same list of consumers [c1, c2], ordered by priority, but will keep doing the same assigment as above, given that on partition-3 with chose the consumer index with N % len(consumers)

@rdhabalia
Copy link
Contributor Author

The question is that broker is using ordinal position for consumers to ensure even distribution.
If I set c-1 with higher priority, I would expect to have c1 on all the partitions.

that's true, I missed it. if we have C1 (priority=1), C2 (priority=0), C3 (priority=0) then C2,C3 consumers with highest priority should serve partitions evenly. let me fix it.

@rdhabalia
Copy link
Contributor Author

@merlimat fixed it.

@rdhabalia
Copy link
Contributor Author

rerun integration tests

@merlimat merlimat modified the milestones: 2.2.1, 2.3.0 Dec 14, 2018
@sijie
Copy link
Member

sijie commented Dec 27, 2018

@rdhabalia can you rebase this issue to latest master?

add java doc

fix partitioned-topic distribution
@merlimat merlimat modified the milestones: 2.3.0, 2.4.0 Feb 14, 2019
@rdhabalia
Copy link
Contributor Author

retest this please

@rdhabalia
Copy link
Contributor Author

rerun java8 tests

1 similar comment
@rdhabalia
Copy link
Contributor Author

rerun java8 tests

@rdhabalia rdhabalia merged commit c39e7d1 into apache:master Mar 5, 2019
jiazhai pushed a commit that referenced this pull request Jun 4, 2020
…er mode (#7136)

This doc is updated for the Pulsar 2.4.0 released PR: #2954.

The doc is updated for the master right now. once it is reviewed and approved, it will be added to the previous releases.
Huanli-Meng added a commit to Huanli-Meng/pulsar that referenced this pull request Jun 4, 2020
These doc updates is for the PR: apache#2954

Docs for master is updated and merged.
Update the following releases based on the master updates:
2.5.2, 2.5.1, 2.5.0, 2.4.2, 2.4.1, 2.4.0
codelipenghui pushed a commit that referenced this pull request Jun 5, 2020
…7172)

These doc updates is for the PR: #2954

Docs for master is updated and merged.
Update the following releases based on the master updates:
2.5.2, 2.5.1, 2.5.0, 2.4.2, 2.4.1, 2.4.0
cdbartholomew pushed a commit to kafkaesque-io/pulsar that referenced this pull request Jul 24, 2020
…er mode (apache#7136)

This doc is updated for the Pulsar 2.4.0 released PR: apache#2954.

The doc is updated for the master right now. once it is reviewed and approved, it will be added to the previous releases.
cdbartholomew pushed a commit to kafkaesque-io/pulsar that referenced this pull request Jul 24, 2020
…pache#7172)

These doc updates is for the PR: apache#2954

Docs for master is updated and merged.
Update the following releases based on the master updates:
2.5.2, 2.5.1, 2.5.0, 2.4.2, 2.4.1, 2.4.0
huangdx0726 pushed a commit to huangdx0726/pulsar that referenced this pull request Aug 24, 2020
…er mode (apache#7136)

This doc is updated for the Pulsar 2.4.0 released PR: apache#2954.

The doc is updated for the master right now. once it is reviewed and approved, it will be added to the previous releases.
huangdx0726 pushed a commit to huangdx0726/pulsar that referenced this pull request Aug 24, 2020
…pache#7172)

These doc updates is for the PR: apache#2954

Docs for master is updated and merged.
Update the following releases based on the master updates:
2.5.2, 2.5.1, 2.5.0, 2.4.2, 2.4.1, 2.4.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants