Add Coursier as a library management implementation #270

andreaTP · 2018-10-16T11:38:56Z

This is a revamp of the great effort done first by @leonardehrenfried in #190

Now we can reproduce the test cases using the scripted tests implementation.
Current status is that normal project dependencies are properly downloaded and resolved, while sbt plugins gets downloaded but not discovered by Sbt itself.

I'm not sure if the resolution of sbt-plugins is mandatory to start integrating this PR.
I've noticed that now coursier is able to publish artifacts, but I think this can be added in a second stage.

I will ask for help and review by @alexarchambault and @eed3si9n

andreaTP · 2018-10-16T14:05:03Z

Essentially, this is a port of the current sbt-coursier sbt plugin, I just revamped the previous implementation and added integration tests with sbt/sbt.

The only current problem is that using coursier to fetch sbtplugins (with addSbtPlugin) result in having the artifacts fetched but sbt do not load the plugin itself.

leonardehrenfried · 2018-10-16T17:19:52Z

This is a fantastic step in the right direction. I'm not sure I can do a meaningful review but I'm going to try anyway.

However, I also got stuck on the sbt plugins and that was one of the reasons I couldn't complete the PR, as I ran out of ideas.

One question: when you set the resolver to coursier does the sbt launcher also fetch the sbt jars themselves using coursier or ivy? On a CI environment this is often the slowest part of the build.

andreaTP · 2018-10-16T17:52:15Z

One question: when you set the resolver to coursier does the sbt launcher also fetch the sbt jars themselves using coursier or ivy? On a CI environment this is often the slowest part of the build.

I'm pretty sure this is not happening, nor I have a clue on how this can be done, maybe @eed3si9n can help?
As you can see on the hack for testing the sbt-plugins integration also going in that direction starts to become tricky https://github.com/sbt/librarymanagement/pull/270/files#diff-3a3697897c85dadd0627ea7d643e5b31R3

eed3si9n · 2018-10-16T18:14:47Z

One question: when you set the resolver to coursier does the sbt launcher also fetch the sbt jars themselves using coursier or ivy? On a CI environment this is often the slowest part of the build.

sbt launcher is its own small program that has its own Ivy embedded. It might be good to at least parallelize download of that like sbt 1's Ivy in lm.

leonardehrenfried · 2018-10-17T11:35:02Z

sbt launcher is its own small program that has its own Ivy embedded. It might be good to at least parallelize download of that like sbt 1's Ivy in lm.

I experimented with using coursier for this too: leonardehrenfried/sbt@4d41125#diff-2d60fc47edf4d9f1c9de681e52e5cd47R95

I downloaded the dependencies but then the JVM couldn't find or load them. In the end I gave up.

IMO, we can however solve one problem at a time and this one can be tackled later. In any case, the change would need to happen to the sbt/sbt repo.

alexarchambault · 2018-10-17T12:53:36Z

Thanks for your efforts @andreaTP!

Have you thought about adding this to the sbt-coursier repository instead? It already has plenty of scripted tests. These currently ensure the sbt-coursier plugin is fine, but I guess they could be tweaked to run against a custom version of sbt, depending on lm-coursier, like you do here. These tests should be rather thorough, so this could help catch bugs in lm-coursier (and debug the issue with sbt plugins I guess).

I can help move that there.

alexarchambault · 2018-10-18T15:04:10Z

build.sbt

@@ -259,6 +260,22 @@ lazy val lmIvy = (project in file("ivy"))
    ),
  )

+lazy val lmCoursier = (project in file("coursier"))
+  .enablePlugins(ContrabandPlugin, JsonCodecPlugin)


Are these plugins necessary?

Suggested change

.enablePlugins(ContrabandPlugin, JsonCodecPlugin)

kept Contraband that is now used and removed the unused JsonCodec

alexarchambault · 2018-10-18T15:04:50Z

build.sbt

+    commonSettings,
+    crossScalaVersions := Seq(scala212, scala211),
+    name := "librarymanagement-coursier",
+    libraryDependencies ++= Seq(coursier, coursierCache, scalaTest, scalaCheck),


Only depend on scalaTest and scalaCheck for the tests?

Suggested change

libraryDependencies ++= Seq(coursier, coursierCache, scalaTest, scalaCheck),

libraryDependencies ++= Seq(coursier, coursierCache, scalaTest % Test, scalaCheck % Test),

correct thanks

alexarchambault · 2018-10-18T15:08:08Z

build.sbt

+    name := "librarymanagement-coursier",
+    libraryDependencies ++= Seq(coursier, coursierCache, scalaTest, scalaCheck),
+    managedSourceDirectories in Compile +=
+      baseDirectory.value / "src" / "main" / "contraband-scala",


Like above, is this needed? (unless you plan to add contraband stuff to lmCoursier?)

alexarchambault · 2018-10-18T15:18:32Z

coursier/src/main/scala/sbt/librarymanagement/coursier/CoursierDependencyResolution.scala

+
+case class CoursierModuleSettings() extends ModuleSettings
+
+private[sbt] class CoursierDependencyResolution(resolvers: Vector[Resolver])


Couldn't this be like a case class, or made behave like one? That way, fields corresponding to the various sbt-coursier keys could correspond to fields here, that could be customized via a .copy(…).

Users could then easily tweak various parameters of the resolution.

And it might also be interesting to have a next version of sbt-coursier just set and / or customize dependencyResolution, updating the various fields. (That would provide a migration path sbt-coursier -> lm-coursier).

Using a simple case class might pose some binary compatibility issues though… Maybe contraband could help here? (Maybe via a separate class to keep all the parameters.)

All the keys don't necessarily need to be here in a first time, just something case-class like, fine with binary compatibility if we add fields, would allow to add the various fields later.

I'm adding a contraband CoursierSettings class, this also affect previous comments around contraband itself

alexarchambault · 2018-10-18T15:20:12Z

coursier/src/main/scala/sbt/librarymanagement/coursier/CoursierDependencyResolution.scala

+      val localArtifacts: Map[Artifact, Either[FileError, File]] = Gather[Task]
+        .gather(
+          resolution.artifacts.map { a =>
+            Cache.file[Task](a).run.map((a, _))


You might want to pass a logger here too, for progress bars and all to appear.

alexarchambault · 2018-10-18T15:23:08Z

coursier/src/main/scala/sbt/librarymanagement/coursier/CoursierDependencyResolution.scala

+
+    import scala.concurrent.ExecutionContext.Implicits.global
+
+    val fetch = Fetch.from(repositories, Cache.fetch[Task](logger = Some(createLogger())))


Logger (TermDisplay at least) should be stopped (via TermDisplay.stop()). This might result in spurious terminal state else (some progress bar updates not being flushed in particular).

andreaTP · 2018-10-19T09:57:19Z

@alexarchambault first round done, ready for the next 😃

andreaTP · 2018-10-24T17:27:47Z

@alexarchambault finally solved the issue with the meta-build! 🥇

I reworked it again a few times until being able to compile sbt itself with coursier as LM.
@alexarchambault would you mind to make a new review? thanks in advance!

typesafe-tools · 2018-10-25T12:55:11Z

The validator has checked the following projects, tested using dbuild, projects built on top of each other.

Project	Reference	Commit
sbt	pull/4430/head	sbt/sbt@`8a8e33b`
zinc	develop	sbt/zinc@`c8e1f53`
io	develop	sbt/io@`e6529c0`
librarymanagement	pull/270/head	`abc6d69`
util	develop	sbt/util@`cd67959`

✅ The result is: SUCCESS

The binaries are at https://dl.bintray.com/sbt/maven-snapshots
The new sbt version is: 1.3.0-bin-20181025T102644
(restart)

typesafe-tools · 2018-10-25T15:16:53Z

A validation involving this pull request is in progress...

typesafe-tools · 2018-10-25T17:14:45Z

The validator has checked the following projects, tested using dbuild, projects built on top of each other.

Project	Reference	Commit
sbt	pull/4430/head	sbt/sbt@`8a8e33b`
zinc	develop	sbt/zinc@`c8e1f53`
io	develop	sbt/io@`e6529c0`
librarymanagement	pull/270/head	`789a4cc`
util	develop	sbt/util@`cd67959`

✅ The result is: SUCCESS

The binaries are at https://dl.bintray.com/sbt/maven-snapshots
The new sbt version is: 1.3.0-bin-20181025T151651
(restart)

typesafe-tools · 2018-10-26T08:45:06Z

A validation involving this pull request is in progress...

typesafe-tools · 2018-10-26T10:42:06Z

The validator has checked the following projects, tested using dbuild, projects built on top of each other.

Project	Reference	Commit
sbt	pull/4430/head	sbt/sbt@`1ad89a4`
zinc	develop	sbt/zinc@`c8e1f53`
io	develop	sbt/io@`e6529c0`
librarymanagement	pull/270/head	`abc6d69`
util	develop	sbt/util@`cd67959`

✅ The result is: SUCCESS

The binaries are at https://dl.bintray.com/sbt/maven-snapshots
The new sbt version is: 1.3.0-bin-20181026T084504
(restart)

eed3si9n

Thanks for all the work @leonardehrenfried, @andreaTP, and @alexarchambault

Originally based on sbt/librarymanagement#270

leonardehrenfried and others added 5 commits October 15, 2018 10:24

Add implementation of coursier

ed25bcb

compilation and tests are ok

e1ce3a0

Starting point for going on

21e2c5f

going on, everything but sbt-plugins looks working

d18c379

fix

1f46e3e

eed3si9n added the ready label Oct 16, 2018

eed3si9n added review and removed ready labels Oct 16, 2018

alexarchambault reviewed Oct 18, 2018

View reviewed changes

andreaTP added 2 commits October 19, 2018 10:50

targeting alex review

7d275cf

removed unused jsoncodec plugin

0499b58

andreaTP and others added 8 commits October 19, 2018 12:05

added hashing function

572f873

better attributes management and plugin resolution test disabled

5ceed97

Add ignore errors flag

a03f0ad

Tweak configuration handling

3b7b2ff

minor cleanup and using missingOk instead of external property

2e6870d

fixing tests

33de222

cleaning up scripted test

eeddaa3

rework again configurations management

1951da2

This was referenced Oct 24, 2018

adding Coursier dependency resolution sbt/sbt#4430

Closed

[DO NOT MERGE] just to produce a SNAP release from Jenkins sbt/io#200

Closed

compatibility with 2.12

abc6d69

andreaTP force-pushed the coursierAgain branch from 789a4cc to abc6d69 Compare October 26, 2018 08:05

adding multi-project test

11fdf8e

eed3si9n approved these changes Oct 29, 2018

View reviewed changes

eed3si9n merged commit 4b179e5 into sbt:develop Oct 29, 2018

eed3si9n mentioned this pull request Oct 29, 2018

Add coursier implementation of dependency resolution API #190

Closed

alexarchambault added a commit to alexarchambault/sbt-coursier that referenced this pull request Nov 20, 2018

Add DependencyResolutionInterface implementation in lm-coursier

8db99f8

Originally based on sbt/librarymanagement#270

alexarchambault mentioned this pull request Nov 20, 2018

Add sbt-lm-coursier plugin coursier/sbt-coursier#7

Merged

alexarchambault added a commit to alexarchambault/sbt-coursier that referenced this pull request Nov 20, 2018

Add DependencyResolutionInterface implementation in lm-coursier

55f457a

Originally based on sbt/librarymanagement#270

alexarchambault mentioned this pull request Nov 28, 2018

Write a coursier-based implem of sbt lib management API coursier/coursier#601

Closed

eed3si9n mentioned this pull request Apr 20, 2019

Coursier dependency resolution integration sbt/sbt#4614

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Coursier as a library management implementation #270

Add Coursier as a library management implementation #270

andreaTP commented Oct 16, 2018 •

edited by eed3si9n

Loading

andreaTP commented Oct 16, 2018

leonardehrenfried commented Oct 16, 2018

andreaTP commented Oct 16, 2018

eed3si9n commented Oct 16, 2018

leonardehrenfried commented Oct 17, 2018 •

edited

Loading

alexarchambault commented Oct 17, 2018 •

edited

Loading

alexarchambault Oct 18, 2018

andreaTP Oct 19, 2018

alexarchambault Oct 18, 2018

andreaTP Oct 19, 2018

alexarchambault Oct 18, 2018

andreaTP Oct 19, 2018

alexarchambault Oct 18, 2018

alexarchambault Oct 18, 2018

andreaTP Oct 19, 2018

alexarchambault Oct 18, 2018

andreaTP Oct 19, 2018

alexarchambault Oct 18, 2018

andreaTP Oct 19, 2018

andreaTP commented Oct 19, 2018

andreaTP commented Oct 24, 2018

typesafe-tools commented Oct 25, 2018

typesafe-tools commented Oct 25, 2018

typesafe-tools commented Oct 25, 2018

typesafe-tools commented Oct 26, 2018

typesafe-tools commented Oct 26, 2018

eed3si9n left a comment

	libraryDependencies ++= Seq(coursier, coursierCache, scalaTest, scalaCheck),
	libraryDependencies ++= Seq(coursier, coursierCache, scalaTest % Test, scalaCheck % Test),


		case class CoursierModuleSettings() extends ModuleSettings

		private[sbt] class CoursierDependencyResolution(resolvers: Vector[Resolver])


		import scala.concurrent.ExecutionContext.Implicits.global

		val fetch = Fetch.from(repositories, Cache.fetch[Task](logger = Some(createLogger())))

Add Coursier as a library management implementation #270

Add Coursier as a library management implementation #270

Conversation

andreaTP commented Oct 16, 2018 • edited by eed3si9n Loading

andreaTP commented Oct 16, 2018

leonardehrenfried commented Oct 16, 2018

andreaTP commented Oct 16, 2018

eed3si9n commented Oct 16, 2018

leonardehrenfried commented Oct 17, 2018 • edited Loading

alexarchambault commented Oct 17, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andreaTP commented Oct 19, 2018

andreaTP commented Oct 24, 2018

typesafe-tools commented Oct 25, 2018

typesafe-tools commented Oct 25, 2018

typesafe-tools commented Oct 25, 2018

typesafe-tools commented Oct 26, 2018

typesafe-tools commented Oct 26, 2018

eed3si9n left a comment

Choose a reason for hiding this comment

andreaTP commented Oct 16, 2018 •

edited by eed3si9n

Loading

leonardehrenfried commented Oct 17, 2018 •

edited

Loading

alexarchambault commented Oct 17, 2018 •

edited

Loading