[SPARK-11198][STREAMING][KINESIS] Support de-aggregation of records during recovery

[brkyvz]

While the KCL handles de-aggregation during the regular operation, during recovery we use the lower level api, and therefore need to de-aggregate the records.

@tdas Testing is an issue, we need protobuf magic to do the aggregated records. Maybe we could depend on KPL for tests?

[SparkQA]

Test build #44784 has finished for PR 9403 at commit de3e91b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * case class RepartitionByExpression(\n

[brkyvz]

Tested that this patch successfully de-aggregates in recovery as well.

[SparkQA]

Test build #44941 has finished for PR 9403 at commit 6b0b290.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44944 has finished for PR 9403 at commit 023055c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44956 has finished for PR 9403 at commit 5f7a763.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44975 has finished for PR 9403 at commit 7c1edd7.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tdas]

test this again.

[tdas]

Isn it easier to have a parameter in the constructor? Less code while subclassing class WithAggregationKinesisStreamSuite extends KinesisStreamTests(aggregateTestData = true)

[brkyvz]

That can also work :)

[tdas]

overall the code looks fine, but Kinesis tests are not passing

[SparkQA]

Test build #45049 has finished for PR 9403 at commit 83a0f13.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #45149 has finished for PR 9403 at commit 8bcd6c3.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[brkyvz]

test this please

[zsxwing]

The failure may be because protobuf 2.6.1 was not enabled. BTW, if 2.6.1 is enabled, shouldn't protobuf 2.6.1 break other stuff in Spark and make the end-to-end test fail?

[brkyvz]

@zsxwing probably not. Locally tests pass. They fail on jenkins for some reason. I've enabled a lot of the logging to look deeper into it.

[SparkQA]

Test build #45181 has finished for PR 9403 at commit 4acf47e.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #45249 has started for PR 9403 at commit 66fade4.

[brkyvz]

test this please

[SparkQA]

Test build #45255 has finished for PR 9403 at commit 66fade4.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #45343 has finished for PR 9403 at commit 2ef1717.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[brkyvz]

Huh. Didn't change much but the tests passed this time. I wonder if it was a Java 7 vs. 8 mismatch... Just to be sure, will re-run tests

[brkyvz]

test this please

[SparkQA]

Test build #45377 has finished for PR 9403 at commit 2ef1717.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[brkyvz]

Aww man. The Kinesis tests passed, hive tests failed. Re-running

[brkyvz]

test this please

[SparkQA]

Test build #45391 has finished for PR 9403 at commit 2ef1717.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[brkyvz]

test this please

[brkyvz]

test this please

[SparkQA]

Test build #45422 has finished for PR 9403 at commit 932485a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tdas]

LGTM. Merging this to master and 1.6. Thanks @brkyvz

[SparkQA]

Test build #45440 has finished for PR 9403 at commit 932485a.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[lordnynex]

Despite being closed, I'm confused about why this causes hive unit tests to fail. Is this unrelated?Also, the publisher in the unit test looks like it's publishing an intentionally non-aggregated stream. Is that expected? Am I reading this incorrectly?

[brkyvz]

Hi @lordnynex. Why did you think this causes Hive tests to fail? That was most probably a flaky test.
Regarding publishing a non-aggregated stream, yes that is correct. Not everyone uses KPL, therefore we need to make sure we support both non-aggregated and aggregated streams :)

[zsxwing]

KinesisStreamTests in test.py is broken because of this PR. See https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46896/testReport/(root)/KinesisStreamTests/test_kinesis_stream/

Is the new dependency the failure cause?

The PR builds for this PR actually didn't report the Python failure because of #9669.