-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-29677][DStreams] amazon-kinesis-client 1.12.0 #26333
Conversation
ok to test |
Thank you for your first contribution, @etspaceman . |
Test build #112990 has finished for PR 26333 at commit
|
The unit test failures seem unrelated to the change. I see that the kinesis package has all passing tests. |
Retest this please. |
Test build #113035 has finished for PR 26333 at commit
|
Seems to have had a separate unrelated error... |
Looks OK if tests pass (yes those failures are unrelated). |
Test build #4911 has finished for PR 26333 at commit
|
a859ce4
to
ddf4931
Compare
The KPL doesn't currently have a release that uses listShards I'm afraid. Though I've heard that's coming soon. Still, if it's just for tests, I'm not too worried about that. Just rebased on the current master. |
Test build #113091 has finished for PR 26333 at commit
|
Test build #4912 has finished for PR 26333 at commit
|
Anything else required before this is merged? |
I usually leave it open for a day or two if it's not urgent, esp at the weekend, just to see if anyone has further comments. It's probably fine. |
This one is quite urgent for our team at Disney Streaming Services. We are going through a massive Kinesis scale-up and the DescribeStream limits have been a major problem. Some of our apps use this module, so it is critical for us to have this merged. We are Databricks customers and this was brought up in a conversation w/ our support staff w/ them last week. If possible, we'd also really appreciate a hotfix release of the |
Understood, and I expect it's fine to merge this to 2.4 as well. While I'm also at Databricks, we all wear our ASF project 'hat' here. Databricks support is separate, and you'd have to ask for updates to that distribution (including any maintenance releases) there. I don't think the OSS project here would cut new releases just for this -- relatively speaking it's not a critical issue for most users. But, can be in the next maintenance release. |
### What changes were proposed in this pull request? Upgrading the amazon-kinesis-client dependency to 1.12.0. ### Why are the changes needed? The current amazon-kinesis-client version is 1.8.10. This version depends on the use of `describeStream`, which has a hard limit on an AWS account (10 reqs / second). Versions 1.9.0 and up leverage `listShards`, which has no such limit. For large customers, this can be a major problem. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Existing tests Closes #26333 from etspaceman/kclUpgrade. Authored-by: Eric Meisel <eric.steven.meisel@gmail.com> Signed-off-by: Sean Owen <sean.owen@databricks.com> (cherry picked from commit be022d9) Signed-off-by: Sean Owen <sean.owen@databricks.com>
Merged to master/2.4 |
Upgrading from amazon-kinesis-client version from 1.8.10 to 1.12.0 is risky in the maintenance release of Spark 2.4. We always avoid such a dependence upgrade in the maintenance releases. I would suggest to revert this from Spark 2.4 release. cc @srowen @dongjoon-hyun @zsxwing @yhuai |
I don't really have a strong opinion on it. It appears to be a clean dependency update for a secondary module. @etspaceman describes a reasonably compelling reason to update it, and I don't know of a problem it causes - do we otherwise have a concern about it? |
### What changes were proposed in this pull request? This PR aims to upgrade `aws-java-sdk-sts` to `1.11.655`. ### Why are the changes needed? [SPARK-29677](#26333) upgrades AWS Kinesis Client to 1.12.0 for Apache Spark 2.4.5 and 3.0.0. Since AWS Kinesis Client 1.12.0 is using AWS SDK 1.11.665, `aws-java-sdk-sts` should be consistent with Kinesis client dependency. - https://github.com/awslabs/amazon-kinesis-client/releases/tag/v1.12.0 ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Pass the Jenkins. Closes #27720 from dongjoon-hyun/SPARK-30968. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
### What changes were proposed in this pull request? This PR aims to upgrade `aws-java-sdk-sts` to `1.11.655`. ### Why are the changes needed? [SPARK-29677](#26333) upgrades AWS Kinesis Client to 1.12.0 for Apache Spark 2.4.5 and 3.0.0. Since AWS Kinesis Client 1.12.0 is using AWS SDK 1.11.665, `aws-java-sdk-sts` should be consistent with Kinesis client dependency. - https://github.com/awslabs/amazon-kinesis-client/releases/tag/v1.12.0 ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Pass the Jenkins. Closes #27720 from dongjoon-hyun/SPARK-30968. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 3995728) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
### What changes were proposed in this pull request? This PR aims to upgrade `aws-java-sdk-sts` to `1.11.655`. ### Why are the changes needed? [SPARK-29677](#26333) upgrades AWS Kinesis Client to 1.12.0 for Apache Spark 2.4.5 and 3.0.0. Since AWS Kinesis Client 1.12.0 is using AWS SDK 1.11.665, `aws-java-sdk-sts` should be consistent with Kinesis client dependency. - https://github.com/awslabs/amazon-kinesis-client/releases/tag/v1.12.0 ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Pass the Jenkins. Closes #27720 from dongjoon-hyun/SPARK-30968. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 3995728) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
### What changes were proposed in this pull request? This PR aims to upgrade `aws-java-sdk-sts` to `1.11.655`. ### Why are the changes needed? [SPARK-29677](apache#26333) upgrades AWS Kinesis Client to 1.12.0 for Apache Spark 2.4.5 and 3.0.0. Since AWS Kinesis Client 1.12.0 is using AWS SDK 1.11.665, `aws-java-sdk-sts` should be consistent with Kinesis client dependency. - https://github.com/awslabs/amazon-kinesis-client/releases/tag/v1.12.0 ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Pass the Jenkins. Closes apache#27720 from dongjoon-hyun/SPARK-30968. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
What changes were proposed in this pull request?
Upgrading the amazon-kinesis-client dependency to 1.12.0.
Why are the changes needed?
The current amazon-kinesis-client version is 1.8.10. This version depends on the use of
describeStream
, which has a hard limit on an AWS account (10 reqs / second). Versions 1.9.0 and up leveragelistShards
, which has no such limit. For large customers, this can be a major problem.Does this PR introduce any user-facing change?
No
How was this patch tested?
Existing tests