[SPARK-21714][CORE][BACKPORT-2.2] Avoiding re-uploading remote resources in yarn client mode #19074

jerryshao · 2017-08-29T05:59:01Z

What changes were proposed in this pull request?

This is a backport PR to fix issue of re-uploading remote resource in yarn client mode. The original PR is #18962.

How was this patch tested?

Tested in local UT.

…arn client mode With SPARK-10643, Spark supports download resources from remote in client deploy mode. But the implementation overrides variables which representing added resources (like `args.jars`, `args.pyFiles`) to local path, And yarn client leverage this local path to re-upload resources to distributed cache. This is unnecessary to break the semantics of putting resources in a shared FS. So here proposed to fix it. This is manually verified with jars, pyFiles in local and remote storage, both in client and cluster mode. Author: jerryshao <sshao@hortonworks.com> Closes apache#18962 from jerryshao/SPARK-21714. (cherry picked from commit 1813c4a) Signed-off-by: jerryshao <sshao@hortonworks.com> Change-Id: Ib2e8cb056707b362bc1c496002bac1472dc78ea7

SparkQA · 2017-08-29T06:02:23Z

Test build #81201 has started for PR 19074 at commit 9c5b562.

jerryshao · 2017-08-29T06:02:54Z

CC @vanzin @tgravescs please review. Since 2.2 and master are quite different in this part of code, so backporting changes a lot.

jerryshao · 2017-08-29T08:02:39Z

Jenkins, retest this please.

SparkQA · 2017-08-29T10:56:39Z

Test build #81205 has finished for PR 19074 at commit 9c5b562.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
* (4) the main class for the child

vanzin · 2017-08-29T17:49:33Z

LGTM. Merging to 2.2.

vanzin · 2017-08-29T17:50:25Z

Merged. @jerryshao please close manually.

…ces in yarn client mode ## What changes were proposed in this pull request? This is a backport PR to fix issue of re-uploading remote resource in yarn client mode. The original PR is #18962. ## How was this patch tested? Tested in local UT. Author: jerryshao <sshao@hortonworks.com> Closes #19074 from jerryshao/SPARK-21714-2.2-backport.

vanzin · 2017-08-29T19:53:15Z

@jerryshao looks like this breaks the 2.10 build:
59529b2#commitcomment-23954704

I reverted the patch waiting for the fix.

jerryshao · 2017-08-30T00:53:19Z

Ohh, sorry about it, I forgot to fix it in 2.10 repl code. I will push a fix soon.

BTW how do we trigger scala 2.10 build on Jenkins?

vanzin · 2017-08-30T01:18:04Z

I don't think there's a way to do that (the build Sean mentions happens after PRs are committed).

jerryshao · 2017-08-30T01:19:39Z

OK, so I will do the test locally.

Change-Id: Ic7ae6152187ce7ee2e527340ecc7e178da970d67

jerryshao · 2017-08-30T02:30:33Z

@vanzin @srowen pushed another commit to change 2.10 repl code, I tested locally with 2.10 code, please review.

SparkQA · 2017-08-30T05:09:36Z

Test build #81242 has finished for PR 19074 at commit 871a005.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-08-30T05:13:59Z

Test build #81243 has finished for PR 19074 at commit 6ba3367.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2017-08-30T19:30:01Z

Alright, let's try again. Merging to 2.2; make sure to close this PR manually.

…ces in yarn client mode ## What changes were proposed in this pull request? This is a backport PR to fix issue of re-uploading remote resource in yarn client mode. The original PR is #18962. ## How was this patch tested? Tested in local UT. Author: jerryshao <sshao@hortonworks.com> Closes #19074 from jerryshao/SPARK-21714-2.2-backport.

jerryshao · 2017-08-31T01:13:11Z

Thanks @vanzin , it should be passed now 😄 .

loneknightpy

@jerryshao @gatorsmile This PR broke spark submit file downloading in standalone cluster mode. Basically, it doesn't use local file consistently after download. Can you take a look?

loneknightpy · 2017-09-15T20:52:44Z

core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

@@ -366,7 +376,7 @@ object SparkSubmit extends CommandLineUtils {
        // If a python file is provided, add it to the child arguments and list of files to deploy.
        // Usage: PythonAppRunner <main python file> <extra python files> [app arguments]
        args.mainClass = "org.apache.spark.deploy.PythonRunner"
-        args.childArgs = ArrayBuffer(args.primaryResource, args.pyFiles) ++ args.childArgs
+        args.childArgs = ArrayBuffer(localPrimaryResource, localPyFiles) ++ args.childArgs
        if (clusterManager != YARN) {
          // The YARN backend distributes the primary file differently, so don't merge it.
          args.files = mergeFileLists(args.files, args.primaryResource)


It is a behavior change here. I think we should use localFiles and localPrimaryResource instead of args.files and args.primaryResource.

Why do you think it should use local files/primary resource here? args.files will be assigned to "spark.files" for non-yarn deploy, and Spark's fileserver will download them to local for all the executors, so it should be fine to keep as remote resources.

Also here the changes restricted to client deploy mode, I'm not sure why it is related to standalone cluster mode?

loneknightpy · 2017-09-15T20:55:15Z

core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

@@ -376,8 +386,8 @@ object SparkSubmit extends CommandLineUtils {
        // The YARN backend handles python files differently, so don't merge the lists.
        args.files = mergeFileLists(args.files, args.pyFiles)


The same reason as above.

loneknightpy · 2017-09-15T20:57:49Z

core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

+    var localPrimaryResource: String = null
+    var localJars: String = null
+    var localPyFiles: String = null
+    var localFiles: String = null
    if (deployMode == CLIENT) {


If you want to avoid download for yarn, can we just check the cluster mode it here?

vanzin · 2017-09-15T21:07:28Z

@loneknightpy can you file a new bug instead of comment on a closed PR?

jiangxb1987 · 2017-09-18T23:03:52Z

ping @jerryshao

jerryshao · 2017-09-19T01:21:42Z

@loneknightpy did you open a new JIRA about this issue?

AFAIK, downloading resources to local disk is not supported for cluster mode even from beginning, would you please elaborate the issue in JIRA, thanks!

jerryshao · 2017-09-21T06:53:16Z

@loneknightpy can you please elaborate more about the issue?

I believe you brought this remote resources support in #18078. It doesn't support cluster mode from beginning. Also your original works brings in regression in running on yarn, that's why I made a change here.

Maybe there's some changes for standalone cluster mode which assumes local resources, but I don't think it's a expected behavior.

…ces in yarn client mode ## What changes were proposed in this pull request? This is a backport PR to fix issue of re-uploading remote resource in yarn client mode. The original PR is apache#18962. ## How was this patch tested? Tested in local UT. Author: jerryshao <sshao@hortonworks.com> Closes apache#19074 from jerryshao/SPARK-21714-2.2-backport.

Add missing scala 2.10 repl changes

6ba3367

Change-Id: Ic7ae6152187ce7ee2e527340ecc7e178da970d67

jerryshao force-pushed the SPARK-21714-2.2-backport branch from 871a005 to 6ba3367 Compare August 30, 2017 02:28

jerryshao closed this Aug 31, 2017

loneknightpy reviewed Sep 15, 2017

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-21714][CORE][BACKPORT-2.2] Avoiding re-uploading remote resources in yarn client mode #19074

[SPARK-21714][CORE][BACKPORT-2.2] Avoiding re-uploading remote resources in yarn client mode #19074

jerryshao commented Aug 29, 2017

SparkQA commented Aug 29, 2017

jerryshao commented Aug 29, 2017

jerryshao commented Aug 29, 2017

SparkQA commented Aug 29, 2017

vanzin commented Aug 29, 2017

vanzin commented Aug 29, 2017

vanzin commented Aug 29, 2017

jerryshao commented Aug 30, 2017

vanzin commented Aug 30, 2017

jerryshao commented Aug 30, 2017

jerryshao commented Aug 30, 2017

SparkQA commented Aug 30, 2017

SparkQA commented Aug 30, 2017

vanzin commented Aug 30, 2017

jerryshao commented Aug 31, 2017

loneknightpy left a comment

loneknightpy Sep 15, 2017

jerryshao Sep 21, 2017

jerryshao Sep 21, 2017

loneknightpy Sep 15, 2017

jerryshao Sep 21, 2017

loneknightpy Sep 15, 2017

vanzin commented Sep 15, 2017

jiangxb1987 commented Sep 18, 2017

jerryshao commented Sep 19, 2017

jerryshao commented Sep 21, 2017

		@@ -376,8 +386,8 @@ object SparkSubmit extends CommandLineUtils {
		// The YARN backend handles python files differently, so don't merge the lists.
		args.files = mergeFileLists(args.files, args.pyFiles)

[SPARK-21714][CORE][BACKPORT-2.2] Avoiding re-uploading remote resources in yarn client mode #19074

[SPARK-21714][CORE][BACKPORT-2.2] Avoiding re-uploading remote resources in yarn client mode #19074

Conversation

jerryshao commented Aug 29, 2017

What changes were proposed in this pull request?

How was this patch tested?

SparkQA commented Aug 29, 2017

jerryshao commented Aug 29, 2017

jerryshao commented Aug 29, 2017

SparkQA commented Aug 29, 2017

vanzin commented Aug 29, 2017

vanzin commented Aug 29, 2017

vanzin commented Aug 29, 2017

jerryshao commented Aug 30, 2017

vanzin commented Aug 30, 2017

jerryshao commented Aug 30, 2017

jerryshao commented Aug 30, 2017

SparkQA commented Aug 30, 2017

SparkQA commented Aug 30, 2017

vanzin commented Aug 30, 2017

jerryshao commented Aug 31, 2017

loneknightpy left a comment

Choose a reason for hiding this comment

loneknightpy Sep 15, 2017

Choose a reason for hiding this comment

jerryshao Sep 21, 2017

Choose a reason for hiding this comment

jerryshao Sep 21, 2017

Choose a reason for hiding this comment

loneknightpy Sep 15, 2017

Choose a reason for hiding this comment

jerryshao Sep 21, 2017

Choose a reason for hiding this comment

loneknightpy Sep 15, 2017

Choose a reason for hiding this comment

vanzin commented Sep 15, 2017

jiangxb1987 commented Sep 18, 2017

jerryshao commented Sep 19, 2017

jerryshao commented Sep 21, 2017