[SPARK-23669] Executors fetch jars and name the jars with md5 prefix

[jinxing64]

What changes were proposed in this pull request?

In our cluster, there are lots of UDF jars, some of them have the same filename but different path, for example:

hdfs://A/B/udf.jar  -> udfA
hdfs://C/D/udf.jar  -> udfB

When user uses udfA and udfB in same sql, executor will fetch both hdfs://A/B/udf.jar and hdfs://C/D/udf.jar to local. There will be a conflict for the same name.

Can we config to fetch jars and save with a filename with MD5 prefix, so there will be no conflict.

How was this patch tested?

UT

[SparkQA]

Test build #88208 has finished for PR 20812 at commit f78c273.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jinxing64]

@vanzin @zsxwing @jerryshao
How do you think about this ?

[jiangxb1987]

shall we still need the localName here?

[jinxing64]

Well, I think we don't need it . The File returned by Utils.fetchFile is the local file. We don't need localName to initialize the local file here.

[jiangxb1987]

How about s"${DigestUtils.md5Hex(url)}-${decodeFileNameInURI(new URI(url))}"?

[jiangxb1987]

should we have this conf in internal/conf ?

[jiangxb1987]

The idea looks good, just a few comments.

[jinxing64]

@jiangxb1987
Thanks a lot for review. I will refine soon !

[jerryshao]

Does it only fix the jars added by sc.addJar or using non-yarn mode? Because yarn uses distributed cache at start, so it has a different code path, right?

[jinxing64]

@jerryshao
Thanks for comment;
Yes, this change is only for sc.addJar and the jars will be named with a prefix when executor updateDependencies.

[SparkQA]

Test build #88303 has finished for PR 20812 at commit 4473878.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #88302 has finished for PR 20812 at commit 4ac1e8e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jerryshao]

@jinxing64 , I think using same name jars which contains different classes seems practically not a best practice. Ideally different udfs should be packaged in different jars with different name/version. That will be easy for user to manage. Also same name jars could easily cause classpath issue usually.

As you always has a workaround for this issue out of Spark. So I would suggest not to fix it, since this is a quite user specific issue.

[jinxing64]

@jerryshao
Understood, Ideally different udfs should be packaged in different jars with different name/version. True. But we are faced with tons of udf/jars migrating from other engine. I made this pr not just for merging, but also want to know you experts ideas.

@jiangxb1987 @jerryshao Thanks again for your comments.

[jiangxb1987]

Should we close this then? @jinxing64 @jerryshao