[SPARK-32119][CORE][3.0] ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars #29621

sarutak · 2020-09-01T21:29:08Z

What changes were proposed in this pull request?

This is a backport PR for branch-3.0.

This PR changes Executor to load jars and files added by --jars and --files on Executor initialization.
To avoid downloading those jars/files twice, they are assosiated with startTime as their uploaded timestamp.

Why are the changes needed?

ExecutorPlugin can't work with Standalone Cluster and Kubernetes
when a jar which contains plugins and files used by the plugins are added by --jars and --files option with spark-submit.

This is because jars and files added by --jars and --files are not loaded on Executor initialization.
I confirmed it works with YARN because jars/files are distributed as distributed cache.

Does this PR introduce any user-facing change?

Yes. jars/files added by --jars and --files are downloaded on each executor on initialization.

How was this patch tested?

Added a new testcase.

…er and Kubernetes with --jars ### What changes were proposed in this pull request? This PR changes Executor to load jars and files added by --jars and --files on Executor initialization. To avoid downloading those jars/files twice, they are assosiated with `startTime` as their uploaded timestamp. ### Why are the changes needed? ExecutorPlugin can't work with Standalone Cluster and Kubernetes when a jar which contains plugins and files used by the plugins are added by --jars and --files option with spark-submit. This is because jars and files added by --jars and --files are not loaded on Executor initialization. I confirmed it works with YARN because jars/files are distributed as distributed cache. ### Does this PR introduce _any_ user-facing change? Yes. jars/files added by --jars and --files are downloaded on each executor on initialization. ### How was this patch tested? Added a new testcase. Closes apache#28939 from sarutak/fix-plugin-issue. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>

…ovement in SPARK-32119 ### What changes were proposed in this pull request? Update monitoring doc following the improvement/fix in SPARK-32119. ### Why are the changes needed? SPARK-32119 removes the limitations listed in the monitoring doc "Distribution of the jar files containing the plugin code is currently not done by Spark." ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Not relevant Closes apache#29463 from LucaCanali/followupSPARK32119. Authored-by: Luca Canali <luca.canali@cern.ch> Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.com>

sarutak · 2020-09-01T21:29:51Z

cc: @tgravescs @mridulm @LucaCanali

SparkQA · 2020-09-01T21:32:21Z

Test build #128166 has finished for PR 29621 at commit f76814c.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-09-01T22:13:58Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32791/

SparkQA · 2020-09-01T22:36:01Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32791/

HyukjinKwon · 2020-09-02T00:47:01Z

retest this please

HyukjinKwon · 2020-09-02T00:47:41Z

cc @zhengruifeng since you're the release manager of 3.0.1

SparkQA · 2020-09-02T01:03:44Z

Test build #128172 has finished for PR 29621 at commit f76814c.

This patch fails Java style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-09-02T01:34:57Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32797/

SparkQA · 2020-09-02T01:55:12Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32797/

zhengruifeng · 2020-09-02T02:16:44Z

@HyukjinKwon Thanks for letting me know this. It seems that this issue may also exist in 2.4? I think you can vote -1 in this active vote.

HyukjinKwon · 2020-09-02T02:33:15Z

Assuming from the JIRA, seems like it's not a blocker. I just cc'ed you just as a FYI :-). but @sarutak can you confirm if this blocks the release?

sarutak · 2020-09-02T02:54:26Z

I don't think this is a blocker. We can't deploy plugins and its required files with --jars/--files but as @tgravescs mentioned here, we have a workaround by installing plugins on all worker nodes beforehand and specifying them with extraClassPath.

dongjoon-hyun · 2020-09-02T03:56:15Z

Ya. I agree that this is not a blocker for 3.0.1 release.

dongjoon-hyun · 2020-10-28T17:36:51Z

Retest this please

SparkQA · 2020-10-28T18:23:43Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34981/

SparkQA · 2020-10-28T18:50:22Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34981/

dongjoon-hyun · 2020-10-28T19:26:12Z

Retest this please

SparkQA · 2020-10-28T19:34:12Z

Test build #130380 has finished for PR 29621 at commit f76814c.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-10-28T19:38:26Z

Retest this please.

SparkQA · 2020-10-28T19:41:58Z

Test build #130381 has finished for PR 29621 at commit f76814c.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-10-28T19:43:10Z

AmbLap Server seems to have a broken Maven local repository again.

========================================================================
Running build tests
========================================================================
exec: curl -s -L https://downloads.lightbend.com/zinc/0.3.15/zinc-0.3.15.tgz
exec: curl -s -L https://downloads.lightbend.com/scala/2.12.10/scala-2.12.10.tgz
Using `mvn` from path: /home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.6.3/bin/mvn
Using `mvn` from path: /home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.6.3/bin/mvn
Performing Maven install for hadoop-2.7-hive-1.2
Using `mvn` from path: /home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.6.3/bin/mvn
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-install-plugin:3.0.0-M1:install (default-cli) on project spark-launcher_2.12: ArtifactInstallerException: Failed to install metadata org.apache.spark:spark-launcher_2.12/maven-metadata.xml: Could not parse metadata /home/jenkins/.m2/repository/org/apache/spark/spark-launcher_2.12/maven-metadata-local.xml: in epilog non whitespace content is not allowed but got > (position: END_TAG seen ...</metadata>\n>... @13:2) -> [Help 1]
[ERROR]

SparkQA · 2020-10-28T20:11:47Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34983/

SparkQA · 2020-10-28T20:19:37Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34984/

SparkQA · 2020-10-28T20:41:04Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34984/

SparkQA · 2020-10-28T20:41:56Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34983/

SparkQA · 2020-10-28T20:46:00Z

Test build #130378 has finished for PR 29621 at commit f76814c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-10-28T21:20:36Z

This PR doesn't pass K8s IT since September.

[SPARK-32119][CORE][3.0] ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars #29621 (comment)
[SPARK-32119][CORE][3.0] ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars #29621 (comment)
[SPARK-32119][CORE][3.0] ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars #29621 (comment)
[SPARK-32119][CORE][3.0] ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars #29621 (comment)
[SPARK-32119][CORE][3.0] ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars #29621 (comment)

The failures are on the following test cases. Recently, Jenkins K8s IT seems to be unstable.

Run SparkPi with no resources (Failed always, but known to be flaky)
Run SparkPi with a very long application name. (failed once in the above)
Run SparkR on simple dataframe.R example (Failed always, but known to be flaky

So, I verified manually on my laptop.

ubernetesSuite:
- Run SparkPi with no resources
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- All pods have the same service account by default
- Run extraJVMOptions check on driver
- Run SparkRemoteFileTest using a remote data file
- Run SparkPi with env and mount secrets.
- Run PySpark on simple pi.py example
- Run PySpark with Python2 to test a pyfiles example
- Run PySpark with Python3 to test a pyfiles example
- Run PySpark with memory customization
- Run in client mode.
- Start pod creation from template
- PVs with local storage
- Launcher client dependencies
- Run SparkR on simple dataframe.R example
Run completed in 10 minutes, 17 seconds.

For the K8s IT failure, I've been taking a look independently.

[SPARK-33408][SPARK-32354][K8S][R] Use R 3.6.3 in K8s R image and re-enable RTestsSuite #30130

dongjoon-hyun

+1, LGTM. Merged to branch-3.0 for Apache Spark 3.0.2.

…Cluster and Kubernetes with --jars ### What changes were proposed in this pull request? This is a backport PR for branch-3.0. This PR changes Executor to load jars and files added by --jars and --files on Executor initialization. To avoid downloading those jars/files twice, they are assosiated with `startTime` as their uploaded timestamp. ### Why are the changes needed? ExecutorPlugin can't work with Standalone Cluster and Kubernetes when a jar which contains plugins and files used by the plugins are added by --jars and --files option with spark-submit. This is because jars and files added by --jars and --files are not loaded on Executor initialization. I confirmed it works with YARN because jars/files are distributed as distributed cache. ### Does this PR introduce _any_ user-facing change? Yes. jars/files added by --jars and --files are downloaded on each executor on initialization. ### How was this patch tested? Added a new testcase. Closes #29621 from sarutak/fix-plugin-issue-3.0. Lead-authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Co-authored-by: Luca Canali <luca.canali@cern.ch> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

sarutak · 2020-10-29T07:48:51Z

@dongjoon-hyun @tgravescs Thank you for taking a look at this PR again!

sarutak and others added 2 commits September 2, 2020 05:54

probot-autolabeler bot added CORE DOCS labels Sep 1, 2020

tgravescs approved these changes Oct 28, 2020

View reviewed changes

dongjoon-hyun added the KUBERNETES label Oct 28, 2020

dongjoon-hyun approved these changes Oct 28, 2020

View reviewed changes

dongjoon-hyun closed this Oct 28, 2020

sarutak deleted the fix-plugin-issue-3.0 branch June 4, 2021 20:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-32119][CORE][3.0] ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars #29621

[SPARK-32119][CORE][3.0] ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars #29621

sarutak commented Sep 1, 2020

sarutak commented Sep 1, 2020

SparkQA commented Sep 1, 2020

SparkQA commented Sep 1, 2020

SparkQA commented Sep 1, 2020

HyukjinKwon commented Sep 2, 2020

HyukjinKwon commented Sep 2, 2020

SparkQA commented Sep 2, 2020

SparkQA commented Sep 2, 2020

SparkQA commented Sep 2, 2020

zhengruifeng commented Sep 2, 2020

HyukjinKwon commented Sep 2, 2020

sarutak commented Sep 2, 2020 •

edited

Loading

dongjoon-hyun commented Sep 2, 2020

dongjoon-hyun commented Oct 28, 2020

SparkQA commented Oct 28, 2020

SparkQA commented Oct 28, 2020

dongjoon-hyun commented Oct 28, 2020

SparkQA commented Oct 28, 2020

dongjoon-hyun commented Oct 28, 2020

SparkQA commented Oct 28, 2020

dongjoon-hyun commented Oct 28, 2020

SparkQA commented Oct 28, 2020

SparkQA commented Oct 28, 2020

SparkQA commented Oct 28, 2020

SparkQA commented Oct 28, 2020

SparkQA commented Oct 28, 2020

dongjoon-hyun commented Oct 28, 2020

dongjoon-hyun left a comment

sarutak commented Oct 29, 2020 •

edited

Loading

[SPARK-32119][CORE][3.0] ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars #29621

[SPARK-32119][CORE][3.0] ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars #29621

Conversation

sarutak commented Sep 1, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

sarutak commented Sep 1, 2020

SparkQA commented Sep 1, 2020

SparkQA commented Sep 1, 2020

SparkQA commented Sep 1, 2020

HyukjinKwon commented Sep 2, 2020

HyukjinKwon commented Sep 2, 2020

SparkQA commented Sep 2, 2020

SparkQA commented Sep 2, 2020

SparkQA commented Sep 2, 2020

zhengruifeng commented Sep 2, 2020

HyukjinKwon commented Sep 2, 2020

sarutak commented Sep 2, 2020 • edited Loading

dongjoon-hyun commented Sep 2, 2020

dongjoon-hyun commented Oct 28, 2020

SparkQA commented Oct 28, 2020

SparkQA commented Oct 28, 2020

dongjoon-hyun commented Oct 28, 2020

SparkQA commented Oct 28, 2020

dongjoon-hyun commented Oct 28, 2020

SparkQA commented Oct 28, 2020

dongjoon-hyun commented Oct 28, 2020

SparkQA commented Oct 28, 2020

SparkQA commented Oct 28, 2020

SparkQA commented Oct 28, 2020

SparkQA commented Oct 28, 2020

SparkQA commented Oct 28, 2020

dongjoon-hyun commented Oct 28, 2020

dongjoon-hyun left a comment

Choose a reason for hiding this comment

sarutak commented Oct 29, 2020 • edited Loading

sarutak commented Sep 2, 2020 •

edited

Loading

sarutak commented Oct 29, 2020 •

edited

Loading