Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

expect always earliest offset on empty initial offset for the records #180

Merged
merged 4 commits into from
Dec 3, 2019

Conversation

lanwen
Copy link
Collaborator

@lanwen lanwen commented Sep 3, 2019

Pulsar by default caches the result of the seek, so in case you do seek,
then create new subscription with the same group name without seek and
earliest subscription type, it will get back to the last seeked offset
what is different from the Kafka and should be aligned in the liiklus

@lanwen lanwen requested a review from bsideup September 3, 2019 15:15
@lanwen
Copy link
Collaborator Author

lanwen commented Sep 3, 2019

Tried this:

.delaySubscription(
        initialOffset.map(PulsarRecordsStorage::fromOffset)
                .switchIfEmpty(Mono.justOrEmpty(autoOffsetReset
                        .map(it -> {
                            switch (it) {
                                case "earliest":
                                    return MessageId.earliest;
                                case "latest":
                                    return MessageId.latest;
                                default:
                                    return null;
                            }
                        })
                ))
                .defaultIfEmpty(MessageId.earliest)
                .flatMap(messageId -> {
                    return Mono.fromCompletionStage(consumer.seekAsync(messageId));
                })
);

it fixes the test, but fails with testMultipleGroups and testExclusiveRecordDistribution

@lanwen
Copy link
Collaborator Author

lanwen commented Sep 3, 2019

with this fix, testExclusiveRecordDistribution seems fails because after seek it sticks to the first subscription doing that and completely ignores another one, so regardless of the partition all new records will go to the first one

@lanwen
Copy link
Collaborator Author

lanwen commented Oct 8, 2019

@bsideup disabled the test for pulsar, so that PR could be merged on its own

Pulsar by default caches the result of the seek, so in case you do seek,
then create new subscription with the same group name without seek and
`earliest` subscription type, it will get back to the last seeked offset
what is different from the Kafka and should be aligned in the liiklus
@bsideup bsideup merged commit bdef814 into master Dec 3, 2019
@delete-merged-branch delete-merged-branch bot deleted the reprocessing branch December 3, 2019 10:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants