scalafix-interfaces: higher-level API for class loading #1152

bjaglin · 2020-06-06T23:06:01Z

This ports the module resolution/fetching code currently in sbt-scalafix into scalafix-interfaces. All JVM clients can now benefit from it, and avoid runtime errors due to different Scala binary versions across classloaders.

bjaglin · 2020-06-07T00:04:27Z

scalafix-tests/unit/src/test/scala/scalafix/tests/interfaces/ScalafixSuite.scala

+ * Tests in this suite require scalafix-cli & its dependencies to be published (`sbt +publishLocal`) so that
+ * Coursier can fetch them.


This is annoying, but I really wanted to exercise different Scala binary versions between the client & the implementation to confirm the proper isolation. Initially, I had every Scala version (as set in sbt) run against every scala version (as previously published, fetched via Coursier), but I couldn't find a nice way to automate the publishLocal step across versions, as it cannot be done in a task. https://github.com/sbt/sbt-projectmatrix could help with that, but would require big changes in the build.

I ended up hardcoding a single version as the target no matter what the sbt version is, and picking the default sbt version so when running that default sbt version (2.13.2), https://github.com/scalacenter/scalafix/pull/1152/files#diff-fdc3abdfd754eeb24090dbd90aeec2ceR246 makes the step unecessary. For other cases, I added an explicit publishLocal in the CI steps, and that note for developers running locally all tests in 2.11 or 2.12.

The metals build contains custom sbt logic to publish local across different Scala versions if you want to take a second stab at this

Great, thanks for the pointer, I'll look at it - maybe I'll follow-up in a separate PR though.

@olafurpg I managed to adapt scalameta/metals@3ab0608 (which is just gold!), to do exactly what I wanted (modulo a scala bug I ran into) https://github.com/scalacenter/scalafix/compare/4a382e5f..a082cd3e 🎉

The overhead of cross-publishing all artifacts before running tests is surprisingly acceptable, so I think this is ready to go. The biggest impact of cross-publishing as a prerequisite might be that failure to build one version impacts CI for all of them, but there is not much to do.

bjaglin · 2020-06-07T00:12:15Z

...fix-interfaces/src/main/java/scalafix/internal/interfaces/ScalafixInterfacesClassloader.java

+                // include types in scalafix.interfaces.* signatures
+                || name.startsWith("coursierapi")) {


I initially had the new overloads in ScalafixArguments implemented within scalafix-interfaces as Java8 default methods (leaving scalafix-cli untouched), so this was not required. It did feel weird though, so I ended up pushing them down to scalafix-interfaces, keeping most of the logic in scalafix-interfaces. I am not sure what's best...

This looks good

bjaglin · 2020-06-07T06:52:23Z

scalafix-interfaces/src/main/java/scalafix/interfaces/Scalafix.java

+
+        List<URL> jars = ScalafixCoursier.scalafixCliJars(repositories, scalafixVersion, scalaVersion);
+        ClassLoader parent = new ScalafixInterfacesClassloader(Scalafix.class.getClassLoader());
+        return classloadInstance(new URLClassLoader(jars.stream().toArray(URL[]::new), parent));


regarding bytecode target Java version: even though it's not explicit in javacOptions, I guess we can assume Java 8 even for older Scala versions since CI builds with java8 which defaults to -target 1.8?

Correct. It’s ok to target java 8

olafurpg

Amazing work!

olafurpg · 2020-06-07T13:02:30Z

scalafix-interfaces/src/main/java/scalafix/interfaces/Scalafix.java

+
+        List<URL> jars = ScalafixCoursier.scalafixCliJars(repositories, scalafixVersion, scalaVersion);
+        ClassLoader parent = new ScalafixInterfacesClassloader(Scalafix.class.getClassLoader());
+        return classloadInstance(new URLClassLoader(jars.stream().toArray(URL[]::new), parent));


Correct. It’s ok to target java 8

scalafix-interfaces/src/main/java/scalafix/interfaces/ScalafixArguments.java

olafurpg · 2020-06-07T13:05:59Z

scalafix-interfaces/src/main/java/scalafix/internal/interfaces/ScalafixCoursier.java

+            String scalafixVersion,
+            String scalaVersion
+    ) throws ScalafixException {
+        Dependency scalafixCli = Dependency.parse(


TIL Dependency.parse 🤩

I have implemented this logic too many times, nice to have it built in!

Indeed! The hardest is/was to find a good name for the input (dep is not very clear). I saw coordinates was used in other places of Coursier source code so I went for this below... Any suggestion is welcome!

olafurpg · 2020-06-07T13:06:52Z

...fix-interfaces/src/main/java/scalafix/internal/interfaces/ScalafixInterfacesClassloader.java

+                // include types in scalafix.interfaces.* signatures
+                || name.startsWith("coursierapi")) {


This looks good

olafurpg · 2020-06-07T13:08:48Z

scalafix-tests/unit/src/test/scala/scalafix/tests/interfaces/ScalafixSuite.scala

+ * Tests in this suite require scalafix-cli & its dependencies to be published (`sbt +publishLocal`) so that
+ * Coursier can fetch them.


The metals build contains custom sbt logic to publish local across different Scala versions if you want to take a second stab at this

bjaglin · 2020-06-07T13:36:00Z

@olafurpg sorry for the amend while you were reviewing. I was simply adding an assertion to introspect/confirm the running Scalafix version within the tool classpath: https://github.com/scalacenter/scalafix/compare/17447d9..4a382e5f

bjaglin · 2020-06-07T13:38:53Z

scalafix-tests/unit/src/test/scala/scalafix/tests/interfaces/ScalafixSuite.scala

+      val scalafixMainCallback = new ScalafixMainCallback {
+        override def reportDiagnostic(diagnostic: ScalafixDiagnostic): Unit =
+          maybeDiagnostic = Some(diagnostic)
+      }


SAM is behind -Xexperimental in 2.11 so I decided to go the verbose way

bjaglin · 2020-06-07T13:45:50Z

scalafix-tests/unit/src/test/scala/scalafix/tests/interfaces/ScalafixSuite.scala

+  }
+
+  def fetchAndLoad(scalaVersion: String): Unit = {
+    test(s"fetch & load instance for Scala version $scalaVersion", SkipWindows) {


error: Unknown rule 'file:C:\Users\RUNNER~1\AppData\Local\Temp\CaptureScalaVersion3780416901127757866.scala' on Windows... I saw that source-compilation was tagged SkipWindows in ToolClasspathSuite as well, so I didn't look further.

bjaglin · 2020-06-07T13:53:10Z

@joan38 @evis will this help for your respective plugins?

joan38 · 2020-06-07T14:41:35Z

@bjaglin yes, thanks for tagging me!
I'll take advantage of it once released.

bjaglin · 2020-06-08T00:25:30Z

project/ScalafixBuild.scala

+        version =>
+          val withScalaVersion = Project
+            .extract(currentState)
+            .appendWithoutSession(


with this (as opposed to appendWithSession), we don't observe the effect of ++ (which makes it harder to override scalaVersion as it seems that it's done at the project level)

This ports the module resolution/fetching code currently in sbt-scalafix into scalafix-interfaces. All JVM clients can now benefit from it, and avoid runtime errors due to different Scala binary versions across classloaders.

bjaglin · 2020-06-08T09:00:58Z

@mlachkar I'll cut 0.9.17 with this if that's OK for you

mlachkar · 2020-06-08T14:29:18Z

@bjaglin yes, please do. Thanks

joan38 · 2020-06-15T20:35:12Z

Could you comment on why it's using coursierapi.Repository and not coursier.Repository?
And how can I convert to a coursierapi.Repository from a coursier.Repository

bjaglin · 2020-06-15T20:45:04Z

Could you comment on why it's using coursierapi.Repository and not coursier.Repository?

@joan38 I can't find any coursier.Repository in the classpath of scalafix-interfaces, could you point me at the one you are looking at? AFAIK the coursier-interface Java facade exposes its API under coursierapi.

joan38 · 2020-06-15T22:08:23Z

https://github.com/coursier/coursier/blob/master/modules/core/shared/src/main/scala/coursier/core/Repository.scala

bjaglin · 2020-06-15T22:14:18Z

coursier-interface (the pure Java, non-cross-versioned module we want clients to depend on to prevent scala binary version conflicts) abstracts the regular (scala) Coursier API under https://github.com/coursier/interface/blob/1dd201e88ef95e2f4d48d580d9678e541d8a4b2e/build.sbt#L53, so it cannot be used directly.

bjaglin · 2020-06-15T22:23:16Z

And how can I convert to a coursierapi.Repository from a coursier.Repository

You should be able to copy the logic https://github.com/coursier/interface/blob/v0.0.22/interface/src/main/scala/coursier/internal/api/ApiHelper.scala#L162-L176.

joan38 · 2020-06-15T23:32:45Z

Thanks @bjaglin for the code, it would have be great if coursier was exposing those utils instead of us translating via a copy and past.

bjaglin · 2020-06-15T23:45:29Z

Indeed. From what I understand, those utils would have to be exposed via a third module that would depend on interface and a non-shared core, so it's not trivial. Maybe you can follow up on the coursier channels to confirm that there is no better way?

joan38 · 2020-06-16T02:39:43Z

So I realised this thing is not resolving scalafix-rules and that's ok.
But I'm still hitting this one:

project.fix java.util.ServiceConfigurationError: scalafix.v1.Rule: scalafix.internal.rule.DisableSyntax not a subtype
    java.base/java.util.ServiceLoader.fail(ServiceLoader.java:591)
    java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1238)
    java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1266)
    java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1301)
    java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1386)
    scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:43)
    scala.collection.Iterator.foreach(Iterator.scala:943)
    scala.collection.Iterator.foreach$(Iterator.scala:943)
    scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
    scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
    scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
    scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:184)
    scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:47)
    scala.collection.TraversableOnce.to(TraversableOnce.scala:348)
    scala.collection.TraversableOnce.to$(TraversableOnce.scala:346)
    scala.collection.AbstractIterator.to(Iterator.scala:1431)
    scala.collection.TraversableOnce.toList(TraversableOnce.scala:332)
    scala.collection.TraversableOnce.toList$(TraversableOnce.scala:332)
    scala.collection.AbstractIterator.toList(Iterator.scala:1431)
    scalafix.internal.v1.Rules$.all(Rules.scala:101)
    scalafix.v1.RuleDecoder$$anon$1.<init>(RuleDecoder.scala:99)
    scalafix.v1.RuleDecoder$.decoder(RuleDecoder.scala:98)
    scalafix.internal.interfaces.ScalafixArgumentsImpl.rulesThatWillRun(ScalafixArgumentsImpl.scala:167)
    com.goyeau.mill.scalafix.ScalafixModule$.fixAction(ScalafixModule.scala:87)
    com.goyeau.mill.scalafix.ScalafixModule.$anonfun$fix$4(ScalafixModule.scala:52)
    mill.api.Result$Success.flatMap(Result.scala:29)
    com.goyeau.mill.scalafix.ScalafixModule.$anonfun$fix$1(ScalafixModule.scala:37)
    mill.define.ApplyerGenerated.$anonfun$zipMap$6(ApplicativeGenerated.scala:15)
    mill.define.Task$MappedDest.evaluate(Task.scala:392)

See: joan38/mill-scalafix#3

Anyone can help?

bjaglin · 2020-06-16T07:14:39Z

Sure - I'll check the PR and comment there

github-brice-jaglin added 3 commits June 5, 2020 02:00

remove dead code

46374f6

explain what scalafix-interfaces is for

58c98bb

make it clear that scala-interfaces is a Java module

f6c2ad7

bjaglin force-pushed the interfaces-fetch branch 3 times, most recently from 713859c to a64696a Compare June 6, 2020 23:53