[SPARK-29677][DStreams] amazon-kinesis-client 1.12.0

[etspaceman]

What changes were proposed in this pull request?

Upgrading the amazon-kinesis-client dependency to 1.12.0.

Why are the changes needed?

The current amazon-kinesis-client version is 1.8.10. This version depends on the use of describeStream, which has a hard limit on an AWS account (10 reqs / second). Versions 1.9.0 and up leverage listShards, which has no such limit. For large customers, this can be a major problem.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing tests

[dongjoon-hyun]

ok to test

[dongjoon-hyun]

Thank you for your first contribution, @etspaceman .

[SparkQA]

Test build #112990 has finished for PR 26333 at commit a859ce4.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[etspaceman]

The unit test failures seem unrelated to the change. I see that the kinesis package has all passing tests.

[dongjoon-hyun]

Retest this please.

[SparkQA]

Test build #113035 has finished for PR 26333 at commit a859ce4.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[etspaceman]

Seems to have had a separate unrelated error...

[srowen]

Looks OK if tests pass (yes those failures are unrelated).
BTW there are a lot of deprecation warnings from the current code; anyone interested in this module is welcome to fix them up.
Do we need to update the kinesis producer too? it's just used in tests though.

[SparkQA]

Test build #4911 has finished for PR 26333 at commit a859ce4.

• This patch fails build dependency tests.
• This patch merges cleanly.
• This patch adds no public classes.

[etspaceman]

The KPL doesn't currently have a release that uses listShards I'm afraid. Though I've heard that's coming soon.

Still, if it's just for tests, I'm not too worried about that.

Just rebased on the current master.

[SparkQA]

Test build #113091 has finished for PR 26333 at commit ddf4931.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #4912 has finished for PR 26333 at commit a859ce4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[etspaceman]

Anything else required before this is merged?

[srowen]

I usually leave it open for a day or two if it's not urgent, esp at the weekend, just to see if anyone has further comments. It's probably fine.

[etspaceman]

This one is quite urgent for our team at Disney Streaming Services. We are going through a massive Kinesis scale-up and the DescribeStream limits have been a major problem. Some of our apps use this module, so it is critical for us to have this merged.

We are Databricks customers and this was brought up in a conversation w/ our support staff w/ them last week.

If possible, we'd also really appreciate a hotfix release of the spark-streaming-kinesis-asl module.

[srowen]

Understood, and I expect it's fine to merge this to 2.4 as well. While I'm also at Databricks, we all wear our ASF project 'hat' here. Databricks support is separate, and you'd have to ask for updates to that distribution (including any maintenance releases) there. I don't think the OSS project here would cut new releases just for this -- relatively speaking it's not a critical issue for most users. But, can be in the next maintenance release.

[srowen]

Merged to master/2.4

[gatorsmile]

Upgrading from amazon-kinesis-client version from 1.8.10 to 1.12.0 is risky in the maintenance release of Spark 2.4. We always avoid such a dependence upgrade in the maintenance releases.

I would suggest to revert this from Spark 2.4 release. cc @srowen @dongjoon-hyun @zsxwing @yhuai

[srowen]

I don't really have a strong opinion on it. It appears to be a clean dependency update for a secondary module. @etspaceman describes a reasonably compelling reason to update it, and I don't know of a problem it causes - do we otherwise have a concern about it?