Split "Compile API" (compiler-interface, compiler-bridge) into another repo

[eed3si9n]

Edit:

This comes up several times a year when sbt team meets up with Lightbend Scala team.

The idea is to split up the compiler interface and the compiler bridge to be a third entity with an independent version number, so it can be maintained partially by Scala team.
Having configurable bridge allows us to have multiple bridge sources, but still the current situation is that we utilize compiler internals for API extraction etc.

[jvican]

I think it's better to have the compiler interface and the bridge in the same repo since all the current modules define what Zinc is. Splitting up these modules will not bring us any benefit, in fact, it will make it more difficult to contribute, IMO

[eed3si9n]

Updated the description.

[eed3si9n]

@adriaanm brought this up again recently.

Dotty is basically doing this already using Java here - https://github.com/lampepfl/dotty/tree/0.13.0-RC1/sbt-bridge
If I understand correctly, this allows Dotty to ship binary stable version of Zinc compiler bridge, and thus it does not require current hack Scala 2-Zinc does of downloading source JAR off of the internet and compiling the bridge upon the first use of sbt. For the sake of reliability (both in terms of quality and network), this sounds like a win.

A benefit for Scala at least is that we can move the contract to compiler-interface, as opposed to the current situation where code-used-in-compiler-bridge becomes an informal contract that Scala needs to keep forever.

[jvican]

@eed3si9n I discussed this topic with @adriaanm casually a week ago. I'm for the migration to another repo, but unlike in Dotty I think we should keep the compiler bridge projects and infrastructure as they are. I propose to migrate properly (with git history) the incremental phases to scala/scala only for Scala versions >= 2.13 and keep the existing sources for previous versions. The bridge sources for Scala 2.13 and future versions would then call the scalac compiler endpoints to extract the analysis.

I'm for leaving the compiler bridge infrastructure in place because it enables us to add compiler-specific code to patch old and new versions without the need of making changes in scala/scala or lampepfl/dotty and cutting new versions. Without this capability, for example, I would have never been able to implement pipelining for Scala 2.11.x, Scala 2.12.x and Scala 2.13.x. I'm pretty sure we'll need this in the future, so I find it really important we keep it in place.

[eed3si9n]

I understand it is a bit bittersweet to give up the cross-Scala-version delivery of improvements, but in a way that's basically using Zinc as a compiler plugin distribution mechanism. It does so at the cost of holding scala/scala hostage because it can never remove any of the old methods that the compiler bridge is calling (and downloading bridge off the internet as I've mentioned above).

Especially since Dotty has made this move, it's really a matter of time that compiler bridge change requires new patch versions to the Scala compiler. We can always make this behavior configurable to make it bring-your-own-compilerbridge-jar, which just requires some opt-in at build tool level.

[jvican]

It does so at the cost of holding scala/scala hostage because it can never remove any of the old methods that the compiler bridge is calling

With the model I'm proposing, the bridge would still be there but it would not use compiler internals. It would instead use an official high-level compiler API provided by Scala compiler versions > 2.13, such as the one Dotty has

I'm not satisfied with the solution of removing the compiler bridge and therefore ask to find a compromise where these two technical goals are met: the standard, default bridge does not use private APIs and zinc is still configurable enough to change the default way of compiling things. I believe my solutions meets these two goals.

[eed3si9n]

I guess it would be helpful to list out our interests and solutions, which I think we are in agreement mostly.

interests

• Reliability of the incremental compilation, and being able to change methods that weren't intended to be exposed.
• Extensibility of incremental compilation.

solution space

• Move compiler interface (Java API) to a separate repo, and version them separately.
• Leave the compiler bridge implementation for Scala 2.10, 2.11, and 2.12 as is.
• Move the compiler bridge implementation for Scala 2.13.x to scala/scala.
• Leave the possibility of bringing your own compiler bridge.
  • Pass in compilerBridgeJar argument as File like Dotty (build tool will resolve the JAR).
  • Leave the compiler bridge infrastructure in place.

notes

Here's current sbt code that uses scalaCompilerBridgeBinaryJar setting.

      val scalac =
        scalaCompilerBridgeBinaryJar.value match {
          case Some(jar) =>
            ZincUtil.scalaCompiler(
              scalaInstance = scalaInstance.value,
              classpathOptions = classpathOptions.value,
              compilerBridgeJar = jar
            )
          case _ =>
            ZincUtil.scalaCompiler(
              scalaInstance = scalaInstance.value,
              classpathOptions = classpathOptions.value,
              globalLock = launcher.globalLock,
              componentProvider = app.provider.components,
              secondaryCacheDir = Option(zincDir),
              dependencyResolution = bootDependencyResolution.value,
              compilerBridgeSource = scalaCompilerBridgeSource.value,
              scalaJarsTarget = zincDir,
              log = streams.value.log
            )
        }

For Scala 2.10/2.11/2.12 (and also for the sake of semantic versioning) both the API points would be available for Zinc 1.x.

#653 is a good evidence that everyone would probably benefit from extractor being managed as part of Scala development.

[eed3si9n]

@jvican I had some more discussion on this with the rest of Lightbend Scala team today. They weren't receptive to the idea of tying this effort (bringing in Zinc phases into scala) with TASTY backend effort planned for later, ultimately because you'd need to process that more to get xsbti data structure. I guess you can discuss more with Adriaan when you see him.