[SPARK-22849] ivy.retrieve pattern should also consider classifier

[gatorsmile]

What changes were proposed in this pull request?

In the previous PR #5755 (comment), we dropped (-[classifier]) from the retrieval pattern. We should add it back; otherwise,

If this pattern for instance doesn't has the [type] or [classifier] token, Ivy will download the source/javadoc artifacts to the same file as the regular jar.

How was this patch tested?

The existing tests

[gatorsmile]

@srowen I think we are resolving a different issue. Your PR #17416 is trying to resolve the issues raised by spark-submit et al via --packages, in which users explicitly specify the classifier.

However, we still face another scenario, the package has the external dependencies. In the dependencies, they could have the same name but different classifiers. For example, zookeeper 3.4.6 has tests.jar and regular jar. In the current solution, we will download both to the same file and ivy will return an exception.

zookeeper-jar: {artifact=zookeeper, ext=jar, module=zookeeper, classifier=tests, organisation=org.apache.zookeeper, type=test-jar, revision=3.4.6}
zookeeper-jar: {artifact=zookeeper, ext=jar, module=zookeeper, organisation=org.apache.zookeeper, type=jar, revision=3.4.6}

[gatorsmile]

In my example,

zookeeper-jar: {artifact=zookeeper, ext=jar, module=zookeeper, classifier=tests, organisation=org.apache.zookeeper, type=test-jar, revision=3.4.6}
zookeeper-jar: {artifact=zookeeper, ext=jar, module=zookeeper, organisation=org.apache.zookeeper, type=jar, revision=3.4.6}

Both dependencies will have the same name org.apache.zookeeper_zookeeper-3.4.6.jar and cause the collision. After my PR, they will be different names
org.apache.zookeeper_zookeeper-3.4.6.jar
org.apache.zookeeper_zookeeper-3.4.6-tests.jar

[gatorsmile]

The reason why I am putting classifier at the end. I am just following the default artifact partern in the Apache IVY.

[srowen]

FWIW this looks fine; I needed a similar change as part of srowen@4126702
It's possible you may need some other changes from that WIP commit; not sure.

[gatorsmile]

Thanks!

[yhuai]

I tried it today. Somehow, I only got the test jar downloaded. Have you guys seen this issue?

[zsxwing]

LGTM

[gatorsmile]

Thanks! Merged to master.

[SparkQA]

Test build #85197 has finished for PR 20037 at commit 331ba33.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.