[SPARK-14423][YARN] Avoid same name files added to distributed cache again

[jerryshao]

What changes were proposed in this pull request?

In the current implementation of assembly-free spark deployment, jars under assembly/target/scala-xxx/jars will be uploaded to distributed cache by default, there's a chance these jars' name will be conflicted with name of jars specified in --jars, this will introduce exception when starting application:

client token: N/A
     diagnostics: Application application_1459907402325_0004 failed 2 times due to AM Container for appattempt_1459907402325_0004_000002 exited with  exitCode: -1000
For more detailed output, check application tracking page:http://hw12100.local:8088/proxy/application_1459907402325_0004/Then, click on links to logs of each attempt.
Diagnostics: Resource hdfs://localhost:8020/user/sshao/.sparkStaging/application_1459907402325_0004/avro-mapred-1.7.7-hadoop2.jar changed on src filesystem (expected 1459909780508, was 1459909782590
java.io.IOException: Resource hdfs://localhost:8020/user/sshao/.sparkStaging/application_1459907402325_0004/avro-mapred-1.7.7-hadoop2.jar changed on src filesystem (expected 1459909780508, was 1459909782590
    at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)
    at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
    at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
    at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
    at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

So here by checking the name of file to avoid same name files uploaded again.

How was this patch tested?

Unit test and manual integrated test is done locally.

[SparkQA]

Test build #55105 has finished for PR 12203 at commit 7ff58be.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

what is this change about?

[jerryshao]

@andrewor14 , in the previous code we assume all the files will be uploaded into distributed cache, so this localizedPath should not be null. But here with my change, some duplicated files will be neglected, this is return localizedPath as null instead, so here I change to this way.

[andrewor14]

Looks good.

[vanzin]

Change looks ok (should mostly affect local builds where example jars have duplicates), but could you add a small unit test in ClientSuite.scala?

[jerryshao]

Sure, will do.

[SparkQA]

Test build #55302 has finished for PR 12203 at commit e1b09c4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jerryshao]

@vanzin , please help to review again, thanks a lot.

[vanzin]

You could have used java.nio.file.Files.copy, but no need to change that now.

[vanzin]

LGTM, merging to master.

[RicoGit]

Hi guys, it is possible to apply this patch to version 1.6? What can I do for this?

[jerryshao]

@RicoGit This is a behavior change for jars uploading to distributed cache, I'm not sure if it is suitable to back-port to branch 1.6. Also this problem is not so severe in 1.6 since we do the assembly for packaging.

[RicoGit]

Thanks for reply. I have problem with running spark job with oozie. This patch solves my problem. I applied this path to spark 1.6, built (spark-yarn_2.10-1.6.0-cdh5.7.0.jar) and put into sharedLibs of oozie.

[jerryshao]

Can you make sure the problem you met is exactly the same as what this PR solved? Since the exception stack you pasted in the StackOverFlow is different from What I pasted here before. From you exception stack, what I could guess is that same jar (same path with same file name) added twice, this is a little different from this PR's mentioned problem.

[RicoGit]

Thanks, i understand this is different problems. What will you advice me? I think that this is not good solution: require(localizedPath != null) just falling with exception message "requirements fails".It is better skip adding to the distributed cache and log warning. How do you think it is enough to open issue?

[jerryshao]

Maybe as you mentioned - skip adding to distributed cache and log warning - is enough, throwing exception will fail the application and this is actually not a fatal problem. I'm OK to change the current behavior for this, what do you think @vanzin ?

[vanzin]

I think there was a version of Oozie that triggered that assert; so maybe upgrading Oozie fixes the problem. It's also probably fine to remove that assert since we haven't seen many people hit it, meaning this situation should be rare.

And, btw, please avoid long discussions on closed PRs. That's why we have mailing lists and JIRA.

[kishorvpatil]

@vanzin, @jerryshao
Sorry for breaking this functionality. I have the patch available with more unit tests added to ensure positive test case ensuring submission continues if unique files/archives are mentioned.

#15810