Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[client] Support seek at partitioned topic #3643

Closed
sijie opened this issue Feb 21, 2019 · 2 comments · Fixed by #7518
Closed

[client] Support seek at partitioned topic #3643

sijie opened this issue Feb 21, 2019 · 2 comments · Fixed by #7518
Labels
area/client help wanted type/feature The PR added a new feature or issue requested a new feature

Comments

@sijie
Copy link
Member

sijie commented Feb 21, 2019

Is your feature request related to a problem? Please describe.

Currently seeking is not supported on partitioned topics.

Describe the solution you'd like

It should be doable to support seek on partitioned topics.

Because:

  1. seeking by earliest / latest can just seek on all partitions.
  2. seeking by messageId is also doable. because messageId provides partition id.

Describe alternatives you've considered

No other alternative. We can just decide not to support it.

Additional context

It would be much easier for people to use if we support seek at partitioned topics.
@sijie sijie added the type/feature The PR added a new feature or issue requested a new feature label Feb 21, 2019
@merlimat
Copy link
Contributor

  • seeking by earliest / latest can just seek on all partitions.

Agree. It doesn't make sense not to support earliest/latest...

2. seeking by messageId is also doable. because messageId provides partition id.

If the seek operation only affects the cursor in one partition, it might become confusing to a user, since it's a different behavior from the above.

In general, I'm not a big fan of seek by message id on the consumer interface, since the purpose of Consumer should be that the reading position is "managed" by the system. If an application wants to position to a particular message id, it's more of manual handling and maybe Reader is a better option there.

@ltamber
Copy link
Contributor

ltamber commented Dec 19, 2019

@sijie Is this feature expected to have time to do?

@sijie sijie mentioned this issue Dec 27, 2019
Closed
@yoniiny yoniiny mentioned this issue Jun 12, 2020
Closed
@sijie sijie mentioned this issue Jun 12, 2020
Closed
@315157973 315157973 mentioned this issue Jul 21, 2020
Merged
codelipenghui pushed a commit that referenced this issue Nov 17, 2020
@315157973
Support partitioned topics in the Reader (#7518)
12ef7c9 
Fixes #3643 #7265


### Motivation
Support partitioned topics in the Reader

### Modifications
class relationship：
PulsarClientImpl -> MultiTopicsReaderImpl -> MultiTopicsConsumerImpl -> ConsumerImpl

PulsarClientImpl support build MultiTopicsReader
MultiTopicsReader wraps one MultiTopicsConsumerImpl 
MultiTopicsConsumerImpl contains multiple consumerImpl

Make MultiTopicsConsumerImpl support seek by messageId

seek by time:
All consumerImpl seek by time, Reader can get all the messages returned by each partition

seek by message：
1）When seek by latest/earliest,all partitions seek by latest/earliest,Reader can get all the messages returned by each partition
2）When the messageId contains explicit partition information, it will only receive messages from this partition, and will not receive messages from other partitions.To avoid other partitions returning messages in inclusive mode, I make other consumer seek to latest and add a ignoredConsumersSet to ignore messages from other partitions .
flowchartsman pushed a commit to flowchartsman/pulsar that referenced this issue Nov 17, 2020
@315157973 @flowchartsman
Support partitioned topics in the Reader (apache#7518)
8b946de 
Fixes apache#3643 apache#7265


### Motivation
Support partitioned topics in the Reader

### Modifications
class relationship：
PulsarClientImpl -> MultiTopicsReaderImpl -> MultiTopicsConsumerImpl -> ConsumerImpl

PulsarClientImpl support build MultiTopicsReader
MultiTopicsReader wraps one MultiTopicsConsumerImpl 
MultiTopicsConsumerImpl contains multiple consumerImpl

Make MultiTopicsConsumerImpl support seek by messageId

seek by time:
All consumerImpl seek by time, Reader can get all the messages returned by each partition

seek by message：
1）When seek by latest/earliest,all partitions seek by latest/earliest,Reader can get all the messages returned by each partition
2）When the messageId contains explicit partition information, it will only receive messages from this partition, and will not receive messages from other partitions.To avoid other partitions returning messages in inclusive mode, I make other consumer seek to latest and add a ignoredConsumersSet to ignore messages from other partitions .
@Samreay Samreay mentioned this issue May 16, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/client help wanted type/feature The PR added a new feature or issue requested a new feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support partitioned topics in the Reader 315157973/pulsar
3 participants
@merlimat @sijie @ltamber