New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
provide guidance to project maintainers on including “latest Scala nightly” in their CI matrices #648
Comments
https://stackoverflow.com/questions/40622878/how-do-i-tell-sbt-to-use-a-nightly-build-of-scala-2-11-2-12-or-2-13 shows how to get the latest nightly in sbt, but doing it in |
I had a little look at how to do this. So I looked for the simplest project I could find that use Travis CI and picked https://github.com/lihaoyi/sourcecode However it depends on sbt-scalajs, which depends on a fully cross-built version of scalajs-compiler... See scala-js/scala-js#3140 for my conversation with the Scala.js project about this issue. I expect Scala Native to be similarly difficult (if not worst...). |
In general, for compiler plugins, the only safe(-ish) way to set it up is to clone the source of the relevant compiler plugin and compile/publish it locally, built againt the particular Scala nightly. This is necessary because scala-compiler.jar does not guarantee any kind of binary compatibility. In fact, it does not even provide any kind of source compat', so even this can fail, but in that case you want to know anyway. You could try using using the published version of the compiler plugin that is built against the most recent Scala release, but you are open to binary incompatibilities. |
Using Scala 2.12.4-bin-38628d1 with
|
My code snippet: // When using a nightly build of Scala (i.e when scalaVersion has a "-bin-.." suffix)
// Replace the scalajs-compiler compiler plugin injected by sbt-scalajs
// With the one built for the _previous_ version of Scala
//
// Example:
// When building with Scala 2.12.4-bin-38628d1, which is a nightly build for future Scala 2.12.4
// Replace scalajs-compiler_2.12.4-bin-38628d1 (which doesn't exist)
// with scalajs-compiler_2.12.3
libraryDependencies := {
/** If scalaVersion has a "-bin-.." suffix, return the previous version.
*
* | scalaVersion | crossSv | baseSv | prevSv |
* |--------------------+----------------+----------------+----------------|
* | 2.12.4 | Some("2.12.4") | None | None |
* | 2.12.4-bin-38628d1 | Some("2.12.4") | Some("2.12.4") | Some("2.12.3") |
*/
def prevScalaVersion(scalaVersion: String, scalaBinaryVersion: String): Option[String] = {
val crossVersion = CrossVersion(CrossVersion.patch, scalaVersion, scalaBinaryVersion)
val crossSv = crossVersion map (fn => fn("") stripPrefix "_")
val baseSv = crossSv filter (_ != scalaVersion)
val prevSv = baseSv collect {
case VersionNumber(Seq(x, y, z), Seq(), Seq()) if z > 0 => s"$x.$y.${z - 1}"
}
prevSv
}
val prevSv = prevScalaVersion(scalaVersion.value, scalaBinaryVersion.value)
val libDeps = libraryDependencies.value
prevSv match {
case None => libDeps
case Some(prevSv) =>
val scalaJsCompiler =
compilerPlugin("org.scala-js" % s"scalajs-compiler_$prevSv" % scalaJSVersion)
libDeps map (m => if (m.name == "scalajs-compiler") scalaJsCompiler else m)
}
} |
Well, guess what? You found an actual binary incompatibility between 2.12.3 and 2.12.4-bin-whatever, introduced by scala/scala@5f5d525#diff-96d929f42cdd21128555c2ea328fbf95 Not sure how the community build passes, btw, because it's also not source compatible. Apparently calling that See also:
|
Maybe I misunderstand, but I'm surprised this hasn't come up before: the scala-compiler jar is not binary compatible across minor versions. |
The new thing here is that @dwijnand is putting himself in harm's way from bin incompatibility. The community build always builds from source, so only source incompatibilities are a problem. But @dwijnand is playing with fire by linking a compiler plugin built for a different version of the compiler. I had warned against this in my first comment in this thread. (The reason I pinged you is only mildly related to the main topic of the discussion; I was surprised that the renaming of |
OK, I read the rest of this ticket now, so there's no misunderstanding about binary / source compatibility for scala-compiler :) Indeed, calling |
So the backend has quite a bit of state that is derived from the symbol table. Since types of symbols might change between runs, this has to be re-computed in each compiler run. This used to be done in the Now we have a bit of an abstraction, state is created through If you're running multiple compiler runs, you should probably call |
Hum. So IIUC, now Scala.js with 2.12.4 is vulnerable to crashes or inconsistent behavior in the PC? |
I'm just experimenting the viability of people testing Scala nightly builds in their projects. They would be configured with |
@sjrd Would the Scala.js project be willing to setup nightly releases of Scala.js against Scala nightlies? |
If someone provides the engineering time to set it up and keep it running, sure. But on my own I can't take that on, I'm afraid. |
The PC doesn't run until the backend, so there's no issue there. This affects cases where a Global instance remains resident and is re-used for compilation (REPL, our JUnit tests). As far as I know, sbt and partest create a new Global instance each time. There's no change wrt 2.12.3: the backend needed to be re-initialized in each run as well. |
Bracketing all the other discussion, I think a JVM-only build with the latest Scala nightly would be very useful. I'd be happy to add it to the shapeless build. |
one new possibility here is for projects to scrape a decently current version out of the new |
to make this more concrete, here's what I have in one repo: resolvers += "scala-integration" at "https://scala-ci.typesafe.com/artifactory/scala-integration/"
def nightly(branch: String): String = {
val url = s"https://raw.githubusercontent.com/scala/community-builds/$branch/nightly.properties"
val props = new java.util.Properties
props.load(new java.net.URL(url).openStream)
props.getProperty("nightly").ensuring(_ != null)
}
crossScalaVersions := Seq("2.13.0", "2.12.9", "2.11.12",
nightly("2.13.x"), nightly("2.12.x"), nightly("2.11.x"))
scalaVersion := crossScalaVersions.value.head
publish / skip := scalaVersion.value.containsSlice("-bin-") but I'm not sure how to write a |
Add the 3 nightly options to the yaml and make it call a wrapper command that does the lookup and calls ++. Think |
FWIW, I've been leaning quite heavily on the |
we wanted to do this for Scala.js (context: scala/community-build#1009) but @sjrd didn't want to modify his main CI matrix since the "latest" aspect would make the builds nonreproducible so he set up a separate repo https://github.com/scala-js/scala-js-test-scala-nightly — it has a this approach may be of interest to other maintainers as well |
I've shamelessly pilfered this for shapeless here. |
proposal to add this to sbt: sbt/sbt#5464 cc @martijnhoekstra more relevant to interactive use than to typical CI, but scala-runners offers it, https://github.com/dwijnand/scala-runners |
One thing I dislike about my advice above is that it forces you to use I guess if you really want the matrix thing, you could rig a thing where instead of controlling the Scala version with Or just hope sbt/sbt#5464 lands at some point :-) |
@dwijnand am I imagining it, or did you do a version of this in one of your repos recently? (scala-rewrites?) |
I'm going to close this ticket as I think the way the Scala 2 ecosystem operates isn't worth changing at this point. Of course this is very much still a live issue generally, but for Scala 3. (Some considerations may be different in Scala 3 because unlike Scala 2, Scala 3 minor releases are only backwards compatible, not forwards compatible.) cc @romanowski |
we should figure out how to do this and provide a sample
.travis.yml
that does it. the Scala modules could be a good place to startThe text was updated successfully, but these errors were encountered: