producer: bugfix for aggregators getting stuck #368
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In circumstances where Flush.Messages and/or Flush.Bytes were set but Flush.Frequency was not, the producer's aggregator could get stuck on a retry because a metadata-only chaser message would not be enough on its own to trigger a flush, and so it would sit in limbo forever. Always trigger a flush in the aggregator when the message is a chaser. This has the additional benefit of reducing retry latency when Flush.Frequency *is* set. Add a test for this case.
eapache
force-pushed
the
flush-retry-recovery
branch
from
191d912
to
a381b36
Compare
|
This PR also DRYs up a pattern in the tests that was annoying me |
| } | ||
| } | ||
| } | ||
|
|
|
This makes sense to me; if the test case went from red to green I am happy. Would writing these tests be easier if we would have a mock Client? |
|
This test in particular was straightforward - I'd have to think about the other two. |
eapache
added a commit
that referenced
this pull request
Mar 19, 2015
producer: bugfix for aggregators getting stuck
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In circumstances where Flush.Messages and/or Flush.Bytes were set but
Flush.Frequency was not, the producer's aggregator could get stuck on a retry
because a metadata-only chaser message would not be enough on its own to trigger
a flush, and so it would sit in limbo forever.
Always trigger a flush in the aggregator when the message is a chaser. This has
the additional benefit of reducing retry latency when Flush.Frequency is set.
Add a test for this case.
@Shopify/kafka