Race Condition in KinesisDataFetcher #231
Labels
Comments
pfifer
added a commit
that referenced
this issue
Sep 26, 2017
* Release 1.8.5 of the Amazon Kinesis Client for Java Release 1.8.5 (September 26, 2017) * Only advance the shard iterator for the accepted response. This fixes a race condition in the `KinesisDataFetcher` when it's being used to make asynchronous requests. The shard iterator is now only advanced when the retriever calls `DataFetcherResult#accept()`. * PR #230 * Issue #231
sahilpalvia
pushed a commit
that referenced
this issue
Sep 27, 2017
* Only advance the shard iterator when we accept a result to return This changes the retriever strategy to only accept the shard iterator when we have accepted a result to return. This is for the asynchronous retriever where multiple threads may contend for the same iterator slot. This ensures only the one selected for the response will advance the shard iterator. * Release 1.8.5 of the Amazon Kinesis Client for Java (#232) * Release 1.8.5 of the Amazon Kinesis Client for Java Release 1.8.5 (September 26, 2017) * Only advance the shard iterator for the accepted response. This fixes a race condition in the `KinesisDataFetcher` when it's being used to make asynchronous requests. The shard iterator is now only advanced when the retriever calls `DataFetcherResult#accept()`. * PR #230 * Issue #231 * Change the TerminalResult to return an empty GetRecordsResult Changes the TerminalResult to provide an empty GetRecordsResult, which components downstream depend on. * Fix unit test for behavior change of TerminalResult Got distracted, and forgot to run the unit tests.
sahilpalvia
added a commit
that referenced
this issue
Oct 16, 2017
* Only advance the shard iterator when we accept a result to return This changes the retriever strategy to only accept the shard iterator when we have accepted a result to return. This is for the asynchronous retriever where multiple threads may contend for the same iterator slot. This ensures only the one selected for the response will advance the shard iterator. * Release 1.8.5 of the Amazon Kinesis Client for Java (#232) * Release 1.8.5 of the Amazon Kinesis Client for Java Release 1.8.5 (September 26, 2017) * Only advance the shard iterator for the accepted response. This fixes a race condition in the `KinesisDataFetcher` when it's being used to make asynchronous requests. The shard iterator is now only advanced when the retriever calls `DataFetcherResult#accept()`. * PR #230 * Issue #231
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When using the asynchronous retriever it's possible for the retriever to start multiple requests. When the requests are completed they update the next shard iterator in the
KinesisDataFetcher. If multiple requests are initiated it's possible for there to be race to set the next shard iterator. This could cause records to be replayed or skipped.The text was updated successfully, but these errors were encountered: