Skip to content

Conversation

@alisaifee
Copy link
Contributor

Clients of the multi-process consumer currently have no way of enforcing that the consumer resumes from the last committed offset. SimpleConsumer automatically receives this behaviour when auto_commit is enabled and can explicitly request the offset using fetch_last_known_offsets.

To ensure backward compatibility a start_from_beginning parameter has been added to the base consumer which defaults to True.

Reference: #347

mp consumer does not resume from the last comitted
offset.
If auto_commit is set on the mp consumers or an explicit
start_from_beginning is set to False, the child workers will
request the last known offsets as well.
@alisaifee
Copy link
Contributor Author

The failing test is unrelated. Theres a new PR #354 which pretty much does the same thing (with inverted logic for the default value of the kwarg for SimpleConsumer). I'm ok with closing this if #354 is preferred.

@vshlapakov
Copy link
Contributor

Oops, my apologies @alisaifee, I missed your PR (searched by word MultiProcessConsumer and found earlier hanging PR #252). They both do practically the same and I'll be fine too if your PR is preferred, it's be just great to add this functionality asap.

@dpkp
Copy link
Owner

dpkp commented Mar 30, 2015

I think there's a better approach to this -- see #356 . I cherry-picked your test commits :)

@alisaifee
Copy link
Contributor Author

👍
closing this PR.
Thanks!

@alisaifee alisaifee closed this Mar 31, 2015
@alisaifee alisaifee deleted the mp_offset_resume branch March 31, 2015 05:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants