[BEAM-12845] Add AWS services as a runtime dependency to Spark Job Server to support S3 #15461

meowcakes · 2021-09-06T08:06:14Z

This accomplishes the same thing as #15313, except for the Spark job server. The AWS services runtime dependency is necessary to use S3 for staging artifacts.

Additionally, I added software.amazon.awssdk:sts as a dependency to the amazon-web-services2 SDK. This is necessary for authentication from EKS when using IAM roles for service accounts.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

`ValidatesRunner` compliance status (on master branch)

Lang	ULR	Twister2
Go	---	---
Java
Python	---	---
XLang		---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

codecov · 2021-09-06T08:27:05Z

Codecov Report

Merging #15461 (9627cb3) into master (bcb56b7) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #15461      +/-   ##
==========================================
- Coverage   83.76%   83.75%   -0.02%     
==========================================
  Files         443      443              
  Lines       60077    60077              
==========================================
- Hits        50324    50316       -8     
- Misses       9753     9761       +8

Impacted Files	Coverage Δ
...hon/apache_beam/runners/direct/test_stream_impl.py	`94.02% <0.00%> (-2.24%)`	⬇️
...pache_beam/runners/interactive/interactive_beam.py	`75.53% <0.00%> (-1.07%)`	⬇️
...runners/interactive/display/pcoll_visualization.py	`85.26% <0.00%> (-0.53%)`	⬇️
...hon/apache_beam/runners/worker/bundle_processor.py	`93.39% <0.00%> (-0.25%)`	⬇️
sdks/python/apache_beam/transforms/util.py	`95.81% <0.00%> (-0.17%)`	⬇️
sdks/python/apache_beam/runners/common.py	`88.73% <0.00%> (-0.15%)`	⬇️
sdks/python/apache_beam/runners/direct/executor.py	`96.79% <0.00%> (+0.53%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bcb56b7...9627cb3. Read the comment docs.

aromanenko-dev · 2021-09-06T11:41:23Z

Is it related to the same Jira as #15313 ?

UPD: seems it requires a new Jira

aromanenko-dev

Thanks for contribution!
Please, don't merge master branch into your feature branch, just use rebase instead

sdks/java/io/amazon-web-services2/build.gradle

aromanenko-dev · 2021-09-06T11:45:30Z

CC: @ibzib ptal

ibzib

LGTM

ibzib · 2021-09-08T16:43:16Z

sdks/java/io/amazon-web-services2/build.gradle

@@ -39,6 +39,7 @@ dependencies {
  compile library.java.aws_java_sdk2_sdk_core
  compile library.java.aws_java_sdk2_sns
  compile library.java.aws_java_sdk2_sqs
+  runtime library.java.aws_java_sdk2_sts


Nit: let's list different types of dependencies (compile, testCompile, etc.) contiguously so it's easier to read.

ibzib · 2021-09-08T16:44:26Z

Run Java_Examples_Dataflow PreCommit

meowcakes · 2021-09-09T01:39:17Z

Run Java_Examples_Dataflow PreCommit

meowcakes · 2021-09-09T01:40:18Z

Run Java_Examples_Dataflow_Java11 PreCommit

pabloem · 2021-09-09T23:56:47Z

should this be merged?

ibzib · 2021-09-10T00:00:58Z

should this be merged?

I'm fine with merging it, @aromanenko-dev what do you think?

aromanenko-dev · 2021-09-10T14:25:42Z

Run Java PostCommit

aromanenko-dev · 2021-09-10T14:41:22Z

@ibzib I'm fine too, just let's wait for Java PostCommit results

aromanenko-dev · 2021-09-10T14:42:23Z

And we need to squash the commits before merge.

aromanenko-dev · 2021-09-14T15:44:11Z

Run Java PostCommit

…rver to support S3 (apache#15461) * enable S3 file system for spark job server * make sts a runtime dependency * contiguous dependency types

aromanenko-dev reviewed Sep 6, 2021

View reviewed changes

sdks/java/io/amazon-web-services2/build.gradle Outdated Show resolved Hide resolved

meowcakes force-pushed the spark-job-server-aws branch from b02895c to 36a4f6e Compare September 6, 2021 12:37

enable S3 file system for spark job server

e6dcdc1

meowcakes force-pushed the spark-job-server-aws branch from 36a4f6e to e6dcdc1 Compare September 6, 2021 13:00

meowcakes changed the title ~~Add AWS services as a runtime dependency to Spark Job Server to support S3~~ [BEAM-12845] Add AWS services as a runtime dependency to Spark Job Server to support S3 Sep 7, 2021

make sts a runtime dependency

6e2ef0f

meowcakes requested a review from ibzib September 8, 2021 04:10

ibzib approved these changes Sep 8, 2021

View reviewed changes

contiguous dependency types

9627cb3

pabloem requested review from aromanenko-dev and ibzib September 9, 2021 23:56

aromanenko-dev approved these changes Sep 10, 2021

View reviewed changes

aromanenko-dev merged commit f5afff1 into apache:master Sep 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-12845] Add AWS services as a runtime dependency to Spark Job Server to support S3 #15461

[BEAM-12845] Add AWS services as a runtime dependency to Spark Job Server to support S3 #15461

meowcakes commented Sep 6, 2021

codecov bot commented Sep 6, 2021 •

edited

Loading

aromanenko-dev commented Sep 6, 2021 •

edited

Loading

aromanenko-dev left a comment

aromanenko-dev commented Sep 6, 2021 •

edited

Loading

ibzib left a comment

ibzib Sep 8, 2021

ibzib commented Sep 8, 2021

meowcakes commented Sep 9, 2021

meowcakes commented Sep 9, 2021

pabloem commented Sep 9, 2021

ibzib commented Sep 10, 2021

aromanenko-dev commented Sep 10, 2021

aromanenko-dev commented Sep 10, 2021

aromanenko-dev commented Sep 10, 2021

aromanenko-dev commented Sep 14, 2021

[BEAM-12845] Add AWS services as a runtime dependency to Spark Job Server to support S3 #15461

[BEAM-12845] Add AWS services as a runtime dependency to Spark Job Server to support S3 #15461

Conversation

meowcakes commented Sep 6, 2021

ValidatesRunner compliance status (on master branch)

Examples testing status on various runners

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

codecov bot commented Sep 6, 2021 • edited Loading

Codecov Report

aromanenko-dev commented Sep 6, 2021 • edited Loading

aromanenko-dev left a comment

Choose a reason for hiding this comment

aromanenko-dev commented Sep 6, 2021 • edited Loading

ibzib left a comment

Choose a reason for hiding this comment

ibzib Sep 8, 2021

Choose a reason for hiding this comment

ibzib commented Sep 8, 2021

meowcakes commented Sep 9, 2021

meowcakes commented Sep 9, 2021

pabloem commented Sep 9, 2021

ibzib commented Sep 10, 2021

aromanenko-dev commented Sep 10, 2021

aromanenko-dev commented Sep 10, 2021

aromanenko-dev commented Sep 10, 2021

aromanenko-dev commented Sep 14, 2021

`ValidatesRunner` compliance status (on master branch)

codecov bot commented Sep 6, 2021 •

edited

Loading

aromanenko-dev commented Sep 6, 2021 •

edited

Loading

aromanenko-dev commented Sep 6, 2021 •

edited

Loading