[SPARK-32119][CORE] ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars #28939

sarutak · 2020-06-28T18:48:54Z

What changes were proposed in this pull request?

This PR changes Executor to load jars and files added by --jars and --files on Executor initialization.
To avoid downloading those jars/files twice, they are assosiated with startTime as their uploaded timestamp.

Why are the changes needed?

ExecutorPlugin can't work with Standalone Cluster and Kubernetes
when a jar which contains plugins and files used by the plugins are added by --jars and --files option with spark-submit.

This is because jars and files added by --jars and --files are not loaded on Executor initialization.
I confirmed it works with YARN because jars/files are distributed as distributed cache.

Does this PR introduce any user-facing change?

Yes. jars/files added by --jars and --files are downloaded on each executor on initialization.

How was this patch tested?

Added a new testcase.

dongjoon-hyun · 2020-06-28T19:34:16Z

Hi, @sarutak . Do you think we can consider this as a Bug for 3.0.1 and 2.4.7? Currently, it's an improvement JIRA for 3.1.0.

sarutak · 2020-06-28T19:47:12Z

Marked as improvement is just a mistake. I've modified it to bug.
The current plugin feature is introduced in 3.0.0 so we can consider for 3.0.1.
For 2.4, this bug may exist but the implementation of plugin feature is different from the one of 3.0 so it might be difficult to backport this change to 2.4 straightforward.

dongjoon-hyun · 2020-06-28T20:02:15Z

Thanks, @sarutak . That's enough~

SparkQA · 2020-06-28T21:00:04Z

Test build #124605 has finished for PR 28939 at commit 69a9be1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-06-28T21:55:56Z

The failure seems relevant.

org.apache.spark.deploy.SparkSubmitSuite.SPARK-32119:
Jars and files should be loaded when Executors launch for plugins

HyukjinKwon · 2020-06-29T02:09:31Z

cc @vanzin FYI

dongjoon-hyun · 2020-06-29T04:31:06Z

core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala

@@ -1254,7 +1255,7 @@ class SparkSubmitSuite
        |public void init(PluginContext ctx, Map<String, String> extraConf) {
        |  String str = null;
        |  try (BufferedReader reader =
-        |    new BufferedReader(new InputStreamReader(new FileInputStream($tempFileName)))) {
+        |    new BufferedReader(new InputStreamReader(new FileInputStream("$tempFileName")))) {


Oh, interesting. Previously, this test case succeeds with "?

Actually, I replaced "test.txt" with $tempFileName just before the first push so, it's a miss-replacement.

SparkQA · 2020-06-29T05:26:15Z

Test build #124622 has finished for PR 28939 at commit b571f4c.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-06-29T07:14:19Z

retest this please.

SparkQA · 2020-06-29T11:50:31Z

Test build #124628 has finished for PR 28939 at commit b571f4c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-06-30T16:54:20Z

Those failed tests passed on my laptop so I'll extend the timeout of the newly added testcase to 30 secs just in case.

SparkQA · 2020-06-30T22:57:47Z

Test build #124682 has finished for PR 28939 at commit 2a47155.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-07-02T06:37:03Z

cc: @LucaCanali

LucaCanali · 2020-07-02T09:26:26Z

This appears to have issues with Kubernetes in my tests, when adding --jars it throws "Unable to create executor due to ".

sarutak · 2020-07-02T13:23:19Z

@LucaCanali Thank you for the feedback. I'll look into the cause.

SparkQA · 2020-07-05T19:56:42Z

Test build #124941 has started for PR 28939 at commit 21720ac.

sarutak · 2020-07-06T17:57:39Z

retest this please.

SparkQA · 2020-07-06T17:58:26Z

Test build #125113 has started for PR 28939 at commit 21720ac.

sarutak · 2020-07-07T21:32:37Z

retest this please.

SparkQA · 2020-07-07T21:40:04Z

Test build #125260 has started for PR 28939 at commit 21720ac.

sarutak · 2020-07-09T08:55:07Z

retest this please.

SparkQA · 2020-07-09T09:00:48Z

Test build #125458 has started for PR 28939 at commit 21720ac.

sarutak · 2020-07-09T17:13:17Z

Hmm, how flaky it is...

sarutak · 2020-07-09T17:13:30Z

retest this please.

SparkQA · 2020-08-14T00:13:36Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32046/

mridulm · 2020-08-14T01:41:44Z

This looks good to me - once Tom's comment is addressed.

SparkQA · 2020-08-14T01:52:54Z

Test build #127425 has finished for PR 28939 at commit 5d65caf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-08-14T06:06:01Z

This looks good to me - once Tom's comment is addressed.

Oops, I've forgotten to remove the unused code. Thanks.

SparkQA · 2020-08-14T06:52:22Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32059/

SparkQA · 2020-08-14T07:05:01Z

Test build #127438 has finished for PR 28939 at commit e57d053.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-08-14T07:14:01Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32059/

sarutak · 2020-08-14T07:20:18Z

retest this please.

SparkQA · 2020-08-14T08:05:50Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32066/

SparkQA · 2020-08-14T08:26:06Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32066/

sarutak · 2020-08-14T08:44:47Z

retest this please.

SparkQA · 2020-08-14T09:31:34Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32070/

SparkQA · 2020-08-14T09:37:38Z

Test build #127445 has finished for PR 28939 at commit e57d053.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-08-14T09:51:40Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32070/

SparkQA · 2020-08-14T11:04:44Z

Test build #127449 has finished for PR 28939 at commit e57d053.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-08-14T11:30:38Z

retest this please.

SparkQA · 2020-08-14T12:20:59Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32078/

SparkQA · 2020-08-14T12:46:25Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32078/

tgravescs · 2020-08-14T13:12:15Z

test this please

SparkQA · 2020-08-14T13:59:20Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32080/

SparkQA · 2020-08-14T14:18:40Z

Test build #127457 has finished for PR 28939 at commit e57d053.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-08-14T14:29:15Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/32080/

SparkQA · 2020-08-14T16:04:35Z

Test build #127459 has finished for PR 28939 at commit e57d053.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

mridulm · 2020-08-14T22:12:59Z

Thanks @sarutak !

…er and Kubernetes with --jars ### What changes were proposed in this pull request? This PR changes Executor to load jars and files added by --jars and --files on Executor initialization. To avoid downloading those jars/files twice, they are assosiated with `startTime` as their uploaded timestamp. ### Why are the changes needed? ExecutorPlugin can't work with Standalone Cluster and Kubernetes when a jar which contains plugins and files used by the plugins are added by --jars and --files option with spark-submit. This is because jars and files added by --jars and --files are not loaded on Executor initialization. I confirmed it works with YARN because jars/files are distributed as distributed cache. ### Does this PR introduce _any_ user-facing change? Yes. jars/files added by --jars and --files are downloaded on each executor on initialization. ### How was this patch tested? Added a new testcase. Closes apache#28939 from sarutak/fix-plugin-issue. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>

…er and Kubernetes with --jars This PR changes Executor to load jars and files added by --jars and --files on Executor initialization. To avoid downloading those jars/files twice, they are assosiated with `startTime` as their uploaded timestamp. ExecutorPlugin can't work with Standalone Cluster and Kubernetes when a jar which contains plugins and files used by the plugins are added by --jars and --files option with spark-submit. This is because jars and files added by --jars and --files are not loaded on Executor initialization. I confirmed it works with YARN because jars/files are distributed as distributed cache. Yes. jars/files added by --jars and --files are downloaded on each executor on initialization. Added a new testcase. Closes apache#28939 from sarutak/fix-plugin-issue. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>

Fixed for ExecutorPlugin working with Standalone Cluster.

69a9be1

probot-autolabeler bot added the CORE label Jun 28, 2020

Modified testcases.

b571f4c

dongjoon-hyun reviewed Jun 29, 2020

View reviewed changes

Extended the timeout to 30 secs.

2a47155

sarutak added 2 commits July 6, 2020 02:59

Fixed url issue.

c5c69d1

Replaced java.io.File with File simply.

21720ac

Removed unused getUserJars method from Utils.scala.

e57d053

tgravescs approved these changes Aug 14, 2020

View reviewed changes

asfgit closed this in 1a4c8f7 Aug 14, 2020

[SPARK-32119][CORE] ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars #28939

[SPARK-32119][CORE] ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars #28939

Conversation

sarutak commented Jun 28, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

dongjoon-hyun commented Jun 28, 2020

sarutak commented Jun 28, 2020 • edited Loading

dongjoon-hyun commented Jun 28, 2020

SparkQA commented Jun 28, 2020

dongjoon-hyun commented Jun 28, 2020

HyukjinKwon commented Jun 29, 2020

dongjoon-hyun Jun 29, 2020

Choose a reason for hiding this comment

sarutak Jun 29, 2020 • edited Loading

Choose a reason for hiding this comment

SparkQA commented Jun 29, 2020

sarutak commented Jun 29, 2020

SparkQA commented Jun 29, 2020

sarutak commented Jun 30, 2020 • edited Loading

SparkQA commented Jun 30, 2020

sarutak commented Jul 2, 2020

LucaCanali commented Jul 2, 2020

sarutak commented Jul 2, 2020

SparkQA commented Jul 5, 2020

sarutak commented Jul 6, 2020

SparkQA commented Jul 6, 2020

sarutak commented Jul 7, 2020

SparkQA commented Jul 7, 2020

sarutak commented Jul 9, 2020

SparkQA commented Jul 9, 2020

sarutak commented Jul 9, 2020

sarutak commented Jul 9, 2020

SparkQA commented Aug 14, 2020

mridulm commented Aug 14, 2020

SparkQA commented Aug 14, 2020

sarutak commented Aug 14, 2020 • edited Loading

SparkQA commented Aug 14, 2020

SparkQA commented Aug 14, 2020

SparkQA commented Aug 14, 2020

sarutak commented Aug 14, 2020

SparkQA commented Aug 14, 2020

SparkQA commented Aug 14, 2020

sarutak commented Aug 14, 2020

SparkQA commented Aug 14, 2020

SparkQA commented Aug 14, 2020

SparkQA commented Aug 14, 2020

SparkQA commented Aug 14, 2020

sarutak commented Aug 14, 2020

SparkQA commented Aug 14, 2020

SparkQA commented Aug 14, 2020

tgravescs commented Aug 14, 2020

SparkQA commented Aug 14, 2020

SparkQA commented Aug 14, 2020

SparkQA commented Aug 14, 2020

SparkQA commented Aug 14, 2020

mridulm commented Aug 14, 2020

sarutak commented Jun 28, 2020 •

edited

Loading

sarutak commented Jun 28, 2020 •

edited

Loading

sarutak Jun 29, 2020 •

edited

Loading

sarutak commented Jun 30, 2020 •

edited

Loading

sarutak commented Aug 14, 2020 •

edited

Loading