Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coursier eviction is "backwards" #418

Open
djspiewak opened this issue Jan 4, 2017 · 3 comments
Labels

Comments

@djspiewak
Copy link

@djspiewak djspiewak commented Jan 4, 2017

I don't have a nice minimal example here, and I'm actually fairly certain this is a bug in Ivy and not a bug in Coursier, but I thought I would toss up the issue just so that other people didn't have to Google far to find the answers.

Coursier appears to evict minor versions with the opposite ordering from what Ivy will do. As an example, if you transitively depend on com.example:exemplar:3.1.1 and com.example:exemplar:3.1.7, Ivy will generally (though not always) evict in favor of 3.1.1 while Coursier will always (so far as I can tell) evict in favor of 3.1.7. The most direct place that this is noticeable is in the unfortunately-ubiquitous "a suffix convention" in the Scalaz ecosystem, where equivalent versions of the same library with differing Scalaz (or scalaz-stream) dependency versions are given identical version numbers but with/without an a suffix. Knobs is a decent example of this. Ivy and Coursier apparently both consider 3.1a to be "higher" than 3.1, but since they evict with opposite semantics, you will end up with an entirely different classpath with each: Ivy will give you 3.1 (and thus the lower version of Scalaz) while Coursier will give you 3.1a (and the higher version). This can in turn generate direct and indirect conflicts with artifacts in other branches of the transitive DAG.

Again, I don't actually think this is a bug! At least, not a bug in Coursier. Coursier's behavior seems saner and more consistent than Ivy's in this regard. But it's something to be aware of, especially when converting projects with non-trivial transitive dependencies or extensive use of version ranges.

@or-shachar

This comment has been minimized.

Copy link

@or-shachar or-shachar commented Jul 11, 2017

I also noticed that on with maven. I couldn't find a simple example as well.
https://maven.apache.org/plugins/maven-dependency-plugin/examples/resolving-conflicts-using-the-dependency-tree.html

Maven will prefer dependencies that are closer to the root of the tree and it seems like coursier takes those that are deeper. I couldn't find any place in code I can configure that behavior.

@OlegYch

This comment has been minimized.

Copy link

@OlegYch OlegYch commented Jan 19, 2018

fwiw i've never seen ivy select 3.1.1 over 3.1.7

@eed3si9n

This comment has been minimized.

Copy link
Contributor

@eed3si9n eed3si9n commented Apr 30, 2019

latest-wins vs nearest-wins conflict resolution

Upon a dependency conflict (that is multiple version candidates are found for com.example:foo within a deps graph), by default Ivy uses a conflict manager with latest-wins strategy:

If this container is not present, a default conflict manager is used for all modules.
The current default conflict manager is the "latest-revision" conflict manager.

Thus latest-wins semantics will be used if you published your module locally using publishLocal, and resolved it out of ivy.xml.

HOWEVER, Maven uses nearest-wins strategy to resolve the conflicts:

  • Dependency mediation - this determines what version of an artifact will be chosen when multiple versions are encountered as dependencies. Maven picks the "nearest definition". That is, it uses the version of the closest dependency to your project in the tree of dependencies. You can always guarantee a version by declaring it explicitly in your project's POM. Note that if two dependency versions are at the same depth in the dependency tree, the first declaration wins.
    • "nearest definition" means that the version used will be the closest one to your project in the tree of dependencies. For example, if dependencies for A, B, and C are defined as
      A -> B -> C -> D 2.0 and A -> E -> D 1.0, then D 1.0 will be used when building A because the path from A to D through E is shorter. You could explicitly add a dependency to D 2.0 in A to force the use of D 2.0.

Ivy tries to emulate this behavior by putting force="true" attribute on the ivy.xml in Ivy cache when it translates from POM file. See for example ~/.ivy2/cache/io.netty/netty/ivy-3.10.6.Final.xml:

		<dependency org="io.netty" name="netty-tcnative" rev="1.1.30.Fork2" force="true" conf="optional->compile(*),master(compile)">
			<artifact name="netty-tcnative" type="jar" ext="jar" conf="" m:classifier="windows-x86_64"/>
		</dependency>
		<dependency org="org.jboss.marshalling" name="jboss-marshalling" rev="1.3.14.GA" force="true" conf="optional->compile(*),master(compile)"/>
		<dependency org="com.google.protobuf" name="protobuf-java" rev="2.5.0" force="true" conf="optional->compile(*),master(compile)"/>
...

Given the fact that most of the libraries are published to Maven Central, the conflict resolution of library dependencies would often happen based on this Maven-emulated nearest-wins semantics.

If Coursier chooses not to adopt Maven emulation, and uses latest-wins strategy I think that's a valid choice, but the conflict resolution semantics should be clearly documented to the users would know what to expect.

version ordering

Ivy and Coursier apparently both consider 3.1a to be "higher" than 3.1

This should be tested. Coursier uses version ordering semantics inspired by Maven as it says so in the comment, whereas Ivy generally waves its hand towards the direction of PHP.

scala> val strategy = new org.apache.ivy.plugins.latest.LatestRevisionStrategy
strategy: org.apache.ivy.plugins.latest.LatestRevisionStrategy = latest-revision

scala> case class MockArtifactInfo(version: String) extends org.apache.ivy.plugins.latest.ArtifactInfo {
         def getRevision: String = version
         def getLastModified: Long = -1
       }
defined class MockArtifactInfo

scala> def sortVersionsIvy(versions: String*): List[String] = {
         import scala.collection.JavaConverters._
         strategy.sort(versions.toArray map MockArtifactInfo)
           .asScala.toList map { case MockArtifactInfo(v) => v }
       }
sortVersionsIvy: (versions: String*)List[String]

scala> sortVersionsIvy("1.0", "2.0")
res0: List[String] = List(1.0, 2.0)

scala> import coursier.core.Version
import coursier.core.Version

scala> def sortVersionsCoursier(versions: String*): List[String] =
         versions.toList.map(Version.apply).sorted.map(_.repr)
sortVersionsCoursier: (versions: String*)List[String]

scala> sortVersionsCoursier("1.0", "2.0")
res0: List[String] = List(1.0, 2.0)

Now we can compare the version ordering implementations.

scala> sortVersionsIvy("3.1", "3.1a")
res1: List[String] = List(3.1a, 3.1)

scala> sortVersionsCoursier("3.1", "3.1a")
res1: List[String] = List(3.1, 3.1a)

As we can see Ivy and Coursier differ here. What if we add a dash?

scala> sortVersionsIvy("3.1", "3.1-a")
res2: List[String] = List(3.1-a, 3.1)

scala> sortVersionsCoursier("3.1", "3.1-a")
res2: List[String] = List(3.1, 3.1-a)

No difference. This might be a bit more surprising since Semantic Versioning says that a dash denotes a prerelease. Let's try more traditional -M1 or -beta1:

scala> sortVersionsIvy("3.1", "3.1-M1")
res3: List[String] = List(3.1-M1, 3.1)

scala> sortVersionsIvy("3.1", "3.1-beta1")
res4: List[String] = List(3.1-beta1, 3.1)

scala> sortVersionsCoursier("3.1", "3.1-M1")
res3: List[String] = List(3.1-M1, 3.1)

scala> sortVersionsCoursier("3.1", "3.1-beta1")
res4: List[String] = List(3.1-beta1, 3.1)

Coursier orders version numbers slightly differently from Ivy. Maybe this is something that should be documented as well. If someone wants to release a beta, they probably should stick to M1 or beta1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.