Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reject offsets that are older than the group expiration time #330

Merged
merged 2 commits into from
Jan 19, 2018

Conversation

toddpalino
Copy link
Contributor

Currently, Burrow will accept any offset for storage. Especially in the case of Zookeeper offsets, it's possible to have stale data (topics that are no longer consumed, or old offsets that have been moved over to Kafka) that will get pulled in and cause false alerts.

This PR adds a check into the storage module to reject offsets if they are older than the expire-group configuration. The reasoning is that if the offset is old enough that we would expire a group with just that offset, we shouldn't accept it at all. This should greatly reduce false alerts due to stale Zookeeper data, and it shouldn't impact current offsets at all.

This also updates the tests to support this and properly test min-distance and offset expiration.

Closes #320

@coveralls
Copy link

coveralls commented Jan 19, 2018

Coverage Status

Coverage decreased (-0.05%) to 73.786% when pulling d522778 on toddpalino:drop-old-offsets into b6184ff on linkedin:master.

@coveralls
Copy link

coveralls commented Jan 19, 2018

Coverage Status

Coverage decreased (-0.05%) to 73.786% when pulling 240838c on toddpalino:drop-old-offsets into b6184ff on linkedin:master.

@toddpalino toddpalino merged commit cacf05e into linkedin:master Jan 19, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants