Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-6004] [kinesis] Allow FlinkKinesisConsumer to skip non-deserializable records #5269

Closed
wants to merge 4 commits into from
Closed

[FLINK-6004] [kinesis] Allow FlinkKinesisConsumer to skip non-deserializable records #5269

wants to merge 4 commits into from

Commits on Jan 31, 2018

  1. [FLINK-8398] [kinesis, tests] Cleanup confusing implementations in Ki… 

    …nesisDataFetcherTest and related classes

The previous implementation of the TestableKinesisDataFetcher was
confusing in various ways, causing it hard to be re-used for other
tests. This commit contains the following various cleaups:

- Remove confusing mocks of source context and checkpoint lock. We now
  allow users of the TestableKinesisDataFetcher to provide a source
  context, which should provide the checkpoint lock.
- Remove override of emitRecordAndUpdateState(). Strictly speaking, that
  method should be final. It was previously overriden to allow
  verifying how many records were output by the fetcher. That
  verification would be better implemented within a mock source context.
- Properly parameterize the output type for the
  TestableKinesisDataFetcher.
- Remove use of PowerMockito in KinesisDataFetcherTest.
- Use CheckedThreads to properly capture any exceptions in fetcher /
  consumer threads in unit tests.
- Use assertEquals / assertNull instead of assertTrue where-ever
  appropriate.
    @tzulitai
    tzulitai committed Jan 31, 2018
    Configuration menu
    Copy the full SHA
    d334908 View commit details
    Browse the repository at this point in the history

  2. [FLINK-8398] [kinesis, tests] Stabilize flaky KinesisDataFetcherTests 

    Prior to this commit, several unit tests in KinesisDataFetcherTest
relied on sleeps to wait until a certain operation happens, in order for
the test to pass.

This commit removes those sleeps and replaces the test behaviours with
OneShotLatches.
    @tzulitai
    tzulitai committed Jan 31, 2018
    Configuration menu
    Copy the full SHA
    e100364 View commit details
    Browse the repository at this point in the history

  3. [FLINK-6004] [kinesis] Allow FlinkKinesisConsumer to skip records 

    This commit acknowledges that null can be returned from the
deserialization schema, if the message cannot be deserialized. If null
is returned for a Kinesis record, no output is produced for that record,
while the sequence number in the shard state is still advanced so that
the record is effectively accounted as processed.
    @tzulitai
    tzulitai committed Jan 31, 2018
    Configuration menu
    Copy the full SHA
    14a51f0 View commit details
    Browse the repository at this point in the history

  4. fixup! Add log warning if a record is skipped

    @tzulitai
    tzulitai committed Jan 31, 2018
    Configuration menu
    Copy the full SHA
    1832606 View commit details
    Browse the repository at this point in the history