[SPARK-7205] Support .ivy2/local and .m2/repositories/ in --packages
#5755
Conversation
|
Test build #31167 has finished for PR 5755 at commit
|
| // We need a chain resolver if we want to check multiple repositories | ||
| val cr = new ChainResolver | ||
| cr.setName("list") | ||
|
|
||
| val localM2 = new IBiblioResolver | ||
| localM2.setM2compatible(true) | ||
| val m2Path = ".m2" + File.separator + "repository" + File.separator |
There was a problem hiding this comment.
Are you sure that you can point ivy to a maven repository like this? Ivy and maven use different format for laying out directory structures (does setUsepoms just make this work?). I'd make sure you explicitly test this locally with the .m2 folder.
There was a problem hiding this comment.
ah I guess that comes with setM2compatible.
|
LGTM - but just to be sure, since the unit tests don't actually cover loading a jar from the local maven or ivy cache, have you tested it locally? If you wanted to go above and beyond, you could create unit tests that generate a maven or ivy dependency in temporary folder and then test that you can load it. |
|
I tested it locally. I can add unit tests. In fact I have a |
|
Okay thanks - we have SPARK-7224 for dealing with better automated testing. I'll merge this now. |
In addition, I made a small change that will allow users to import 2 different artifacts with the same name. That change is made in `[organization]_[artifact]-[revision].[ext]`. This used to be only `[artifact].[ext]` which might have caused collisions between artifacts with the same artifactId, but different groupId's. cc pwendell Author: Burak Yavuz <brkyvz@gmail.com> Closes apache#5755 from brkyvz/local-caches and squashes the following commits: c47c9c5 [Burak Yavuz] Small fixes to --packages
In addition, I made a small change that will allow users to import 2 different artifacts with the same name. That change is made in `[organization]_[artifact]-[revision].[ext]`. This used to be only `[artifact].[ext]` which might have caused collisions between artifacts with the same artifactId, but different groupId's. cc pwendell Author: Burak Yavuz <brkyvz@gmail.com> Closes apache#5755 from brkyvz/local-caches and squashes the following commits: c47c9c5 [Burak Yavuz] Small fixes to --packages Conflicts: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
In addition, I made a small change that will allow users to import 2 different artifacts with the same name. That change is made in `[organization]_[artifact]-[revision].[ext]`. This used to be only `[artifact].[ext]` which might have caused collisions between artifacts with the same artifactId, but different groupId's. cc pwendell Author: Burak Yavuz <brkyvz@gmail.com> Closes apache#5755 from brkyvz/local-caches and squashes the following commits: c47c9c5 [Burak Yavuz] Small fixes to --packages
In addition, I made a small change that will allow users to import 2 different artifacts with the same name. That change is made in `[organization]_[artifact]-[revision].[ext]`. This used to be only `[artifact].[ext]` which might have caused collisions between artifacts with the same artifactId, but different groupId's. cc pwendell Author: Burak Yavuz <brkyvz@gmail.com> Closes apache#5755 from brkyvz/local-caches and squashes the following commits: c47c9c5 [Burak Yavuz] Small fixes to --packages
| @@ -899,7 +916,8 @@ private[deploy] object SparkSubmitUtils { | |||
| } | |||
| // retrieve all resolved dependencies | |||
| ivy.retrieve(rr.getModuleDescriptor.getModuleRevisionId, | |||
| packagesDirectory.getAbsolutePath + File.separator + "[artifact](-[classifier]).[ext]", | |||
| packagesDirectory.getAbsolutePath + File.separator + | |||
| "[organization]_[artifact]-[revision].[ext]", | |||
There was a problem hiding this comment.
What is the reason we got rid of (-[classifier])?
I saw some discussions about this. For example,
If this pattern for instance doesn't has the [type] or [classifier] token, Ivy will download the source/javadoc artifacts to the same file as the regular jar.
## What changes were proposed in this pull request? In the previous PR #5755 (comment), we dropped `(-[classifier])` from the retrieval pattern. We should add it back; otherwise, > If this pattern for instance doesn't has the [type] or [classifier] token, Ivy will download the source/javadoc artifacts to the same file as the regular jar. ## How was this patch tested? The existing tests Author: gatorsmile <gatorsmile@gmail.com> Closes #20037 from gatorsmile/addClassifier.
In addition, I made a small change that will allow users to import 2 different artifacts with the same name. That change is made in
[organization]_[artifact]-[revision].[ext]. This used to be only[artifact].[ext]which might have caused collisions between artifacts with the same artifactId, but different groupId's.cc @pwendell