New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add a Java 9 community build #609

Closed
SethTisue opened this Issue Sep 24, 2017 · 64 comments

Comments

Projects
None yet
5 participants
@SethTisue
Copy link
Member

SethTisue commented Sep 24, 2017

starting, I suppose, with just a stripped-down thing that only attempts to build some basic stuff like the Scala standard modules, then expanding from there

Scala 2.12 only is fine for now, I think. the behemoths are already somewhat pressed for space (and the space can't be increased without rebuilding the nodes), and even just adding the 2.12/jdk9 combo will make the problem, one might say, worse enough

plan:

  • get it running locally on my Mac
  • get it running on Jenkins

re: Jenkins, @retronym did some work, prior to JDK 9 going final/gold/GA, on making it available on our Jenkins nodes; see scala/scala-jenkins-infra#174 for details

@SethTisue SethTisue self-assigned this Sep 24, 2017

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Sep 25, 2017

my experience attempting this locally was that all extractions failed with [scalacheck] [error] scala.reflect.internal.MissingRequirementError: object java.lang.Object in compiler mirror not found.

but then I made an empty sbt project and ran it with Java 9, and sbt/sbt-launcher-package#143 kicked in and showed the "Copying runtime jar." message. I thought then running dbuild again might work, but same error

maybe I also need to do something to make sure -Dscala.ext.dirs=... gets set?

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Sep 25, 2017

maybe I also need to do something to make sure -Dscala.ext.dirs=... gets set?

helps, but is not sufficient. adding extra.options: ["-Dscala.ext.dirs=/Users/tisue/.sbt/0.13/java9-rt-ext-oracle_corporation_9"] to every projects allows extraction to proceed farther, but then eventually every projects gets an error like:

[scala-parser-combinators] java.lang.AssertionError: assertion failed: not a type: symbol
[scala-parser-combinators]  loaded from class ScalaModulePlugin$ in package with name ScalaModulePlugin$ and classloader sbt.PluginManagement$PluginClassLoader@9b8d3db
[scala-parser-combinators] 	at scala.reflect.runtime.JavaMirrors$JavaMirror.scala$reflect$runtime$JavaMirrors$JavaMirror$$classToScala1(JavaMirrors.scala:979)
[scala-parser-combinators] 	at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$classToScala$1.apply(JavaMirrors.scala:938)
[scala-parser-combinators] 	at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$classToScala$1.apply(JavaMirrors.scala:938)
@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Sep 25, 2017

an old (2014) ticket on Java 9 support dbuild is lightbend/dbuild#155

@retronym

This comment has been minimized.

Copy link
Member

retronym commented Sep 25, 2017

That might be scala/scala-dev#304. The only workaround I know it to refactor the .scala files in the build to avoid using the empty package.

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Sep 25, 2017

java.lang.AssertionError: assertion failed: not a type: symbol

this is reproducible outside dbuild simply by switching to Java 9 and then firing up sbt in e.g. the scala-xml or scala-parser-combinators repo

two repos that don't have this problem: utest, scalacheck

as an experiment, I went into scala-parser-combinators and ripped out sbt-scala-module, after which scala-parser-combinatorsJVM/test passed. so the problem is with that plugin somehow

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Sep 25, 2017

the problem is with that plugin somehow

okay, so let's work on that over at scala/sbt-scala-module#33, then come back here later

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Sep 25, 2017

a "community build" consisting only of scalacheck (with scala and scala-xml binaries retrieved via aether), at https://github.com/SethTisue/community-builds/blob/wip-2.12.x-jdk9/configs/community.dbuild, is passing locally with the extra.options: ["-Dscala.ext.dirs=... workaround 🎉

so it looks like dbuild works on Java 9, which is good news, and we'll just need to expand the build a step at a time from here (and work around the scala.ext.dirs thing in a more elegant way)

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Sep 25, 2017

we'll just need to expand the build a step at a time from here

questioning, all the while, how ambitious it's appropriate to get with this how soon! I could see this staying pretty minimal for a long time

until at least:

  • community projects have added Java 9 to their own CI matrices
    • which will probably be happening pretty rapidly over the next year or so... but in the meantime we don't need to get out ahead of the community on this

and probably not even until either or both of these stronger conditions are met:

  • we start work on a version of Scala which requires Java 9
  • significant community projects start requiring Java 9
    • this is what prompted the creation of the scala-2.11.x-jdk8 build — significant libraries started dropping support for Java 6 and/or 7
@retronym

This comment has been minimized.

Copy link
Member

retronym commented Sep 25, 2017

Are we able to set the Java version individually per-project? That would allow us more flexibility: we could build problematic projects on Java 8 and run compliant ones on 9.

@cunei

This comment has been minimized.

Copy link
Contributor

cunei commented Sep 25, 2017

Not at the moment; dbuild uses whatever version of Java is in the environment, and makes no attempt to control it.

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Oct 2, 2017

Akka has announced Java 9 support (as of Akka 2.5.6). I'll make Akka a priority next time I do some work on this

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Oct 17, 2017

the new build has grown from one to five projects.

they are: scala-xml, plus

[info] Project scala-js----------------: SUCCESS (project rebuilt ok)
[info] Project scala-parser-combinators: SUCCESS (unchanged, not rebuilt)
[info] Project scalacheck--------------: SUCCESS (unchanged, not rebuilt)
[info] Project utest-------------------: SUCCESS (project rebuilt ok)

how I got there:

I rebased Jason's backport PR onto 2.10.x (which has other JDK9 stuff merged on it that Jason's PR doesn't include yet)

after much trial and error, my efforts to get dbuild+sbt to use a Scala version with "SNAPSHOT" and/or a SHA in the version number failed, so I publish a 2.10.7 snapshot as 2.10.7 to ~/.m2 locally (no -SNAPSHOT or SHA in the version):

brew install ant@1.9
PATH=/usr/local/Cellar/ant@1.9/1.9.9/bin:$PATH
export JAVA_HOME=$(/usr/libexec/java_home -v 1.6)
ANT_OPTS="-Xmx2048M -Xss4M -XX:MaxPermSize=256M" ant -Dbuild.release=true fastdist distpack
cd dists/maven/2.10.7
ant deploy.local

and then in my repositories file put:

  maven-local-alt: file:///Users/tisue/.m2/repository/

since as @dwijnand discovered, sbt is hardwired to ignore a normal maven-local entry when resolving Scala snapshots specifically (?!)

the config file changes necessary are in the wip branch in my fork

I'm also not going to bother trying to run this on Jenkins until Scala 2.10.7 and sbt 0.13.17 are published — I think it would be wasted work, it's enough if I just run it locally for now.

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Oct 20, 2017

@retronym might pick this up and run with it for a while. thus, brain dump:

I have pushed my wip fumblings on this to https://github.com/SethTisue/community-builds/commits/wip-2.12.x-jdk9

to get it up on Jenkins, some work on the scala-jenkins-infra repository will be needed:

  • the new build should go on behemoth 1; behemoth 2 already runs out of space a lot, behemoth 1 is much less prone to that. (we have so far avoided just cranking up the disk space on the boxes because doing so requires re-creating them, which Chef is only some help with)
  • an appropriate JDK needs to be available. Jason already did some work to make JDK 9 available there, I think, but it might be out of date? ideally we go through Chef to get stuff installed on our Jenkins nodes, but if doing so is just too much hassle, we're not above just manually installing something; I think we have done that there in the past with JDKs. @adriaanm might be more aware than I am of where we currently stand on that.

adding more Scala modules to the build would require doing something about scala/sbt-scala-module#35, iirc

besides the remaining Scala modules, obvious next additions could include ScalaTest and akka-actor

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Jan 17, 2018

sbt 0.13.17-RC2 and 1.1.0 exist now and should make this a bit easier

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Mar 2, 2018

what are those great crashing footfalls I hear? it's our THREE mighty behemoths, LARGER than the old ones. what a thrilling and awe-inspiring sight they are! and apparently they have JDK 9 on them already 🤖🤖🤖 💪💪💪 ⓽⓽⓽ 💥💥💥

I created https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk9-integrate-community-build via the Jenkins UI by copying the regular 2.12.x job, adding -jdk9- to the name, and changing the JVM version to 9, and a run is in progress here: https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk9-integrate-community-build/1/consoleFull

note that's a run of the regular 2.12.x branch of this repo. thought I'd see how that does first. we can also try the wip-2.12.x-jdk9 branch — some of the changes there shouldn't be needed anymore, maybe all of them.

@ashawley

This comment has been minimized.

Copy link
Member

ashawley commented Mar 22, 2018

It failed. Didn't look like there were any more builds kicked off. Did this end up getting hung up?

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Mar 22, 2018

work on 2.12.5 (and the usual thousand other things) interceded. ironically, since this would have caught the big regression in 2.12.5. sigh

yeah so in the first run a bunch of projects failed to even extract, I looked at a sampling and it was usually (always?) because something in the build choked on the unexpected Java version number

so I'll need to start just marching down the list and making pull requests or issues in those projects, starting with the ones with the most downstream dependents

I kicked off a new run just now at https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk9-integrate-community-build/2/console , perhaps some of the included projects have already been fixed

@ashawley

This comment has been minimized.

Copy link
Member

ashawley commented Mar 22, 2018

it was usually (always?) because something in the build choked on the unexpected Java version number

Oh, ok. Yeah, that's brutal.

@retronym

This comment has been minimized.

Copy link
Member

retronym commented Mar 22, 2018

[argonaut-shapeless] [error] (coursierResolutions) java.lang.IncompatibleClassChangeError: Inconsistent constant pool data in classfile for class coursier/shaded/fastparse/core/LowestPriSequencer. Method $anonfun$Sequencer1$1(Ljava/lang/Object;Ljava/lang/Object;)Lscala/Tuple2; at index 39 is CONSTANT_MethodRef and should be CONSTANT_InterfaceMethodRef
[argonaut-shapeless] [error] (coursierSbtClassifiersResolution) java.lang.IncompatibleClassChangeError: Inconsistent constant pool data in classfile for class coursier/shaded/fastparse/core/LowestPriSequencer. Method $anonfun$Sequencer1$1(Ljava/lang/Object;Ljava/lang/Object;)Lscala/Tuple2; at index 39 is CONSTANT_MethodRef and should be CONSTANT_InterfaceMethodRef
[argonaut-shapeless] Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? 

I know this problem, it requires an updated shade plugin, as in playframework/play-ws#173. I'll take this up with coursier and argonaut-shapeless.

Issue: coursier/coursier#822

Workaround: #700

@retronym

This comment has been minimized.

Copy link
Member

retronym commented Mar 22, 2018

[paradox] [info] 'compiler-interface' not yet compiled for Scala 2.10.6. Compiling...
[paradox:error] error: error while loading package, Missing dependency 'object java.lang.Object in compiler mirror', required by /home/jenkins/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.10.6.jar(scala/package.class)
[paradox:error] error: error while loading package, Missing dependency 'object java.lang.Object in compiler mirror', required by /home/jenkins/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.10.6.jar(scala/runtime/package.class)
[paradox:error] error: scala.reflect.internal.MissingRequirementError: object java.lang.Object in compiler mirror not found.
[paradox:error] 	at scala.reflect.internal.MissingRequirementError$.signal(MissingRequirementError.scala:16)
[paradox:error] 	at scala.reflect.internal.MissingRequirementError$.notFound(MissingRequirementError.scala:17)
[paradox:error] 	at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
[paradox:error] 	at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40)

This probably just means that project is building with SBT 0.13.16 or lower, which is yet to update to Scala 2.10.7 to get support of for Java 9.

#701

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Mar 23, 2018

This probably just means that project is building with SBT 0.13.16 or lower

that's odd — we override the sbt versions of all included projects (to 0.13.17 and 1.1.1, currently)

@ashawley

This comment has been minimized.

Copy link
Member

ashawley commented May 4, 2018

I've opened a PR for ScalaTest:

scalatest/scalatest#1340

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented May 6, 2018

this is still a back-burner task, but, to at least make sure we're not slipping backwards, I merged current 2.12.x onto https://github.com/SethTisue/community-builds/commits/wip-2.12.x-jdk9 and made sure it still works:

[info] Project cloc-plugin-----------------: SUCCESS (unchanged, not rebuilt)
[info] Project scala-----------------------: SUCCESS (unchanged, not rebuilt)
[info] Project scala-js--------------------: SUCCESS (project rebuilt ok)
[info] Project scala-js-stubs--------------: SUCCESS (project rebuilt ok)
[info] Project scala-parser-combinators----: SUCCESS (unchanged, not rebuilt)
[info] Project scalacheck------------------: SUCCESS (unchanged, not rebuilt)
[info] Project utest-----------------------: SUCCESS (project rebuilt ok)
[info] >>> The dbuild result is----------------: SUCCESS (project rebuilt ok)

as for the full build, I've set a personal to-do item to make gradual progress on at least getting all of the projects extracting. today's increment of progress on that is here: scala/scala-swing#78

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented May 17, 2018

it didn't, and I couldn't reproduce it outside dbuild, so maybe it's choking on a dbuild-mangled version number

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented May 18, 2018

gah, of course, it's scala/sbt-scala-module#35 again, which has been pending so long I forgot about it

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented May 18, 2018

scala/scala-xml#220 should do it

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented May 18, 2018

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented May 19, 2018

sigh, for some reason slick/slick#1908 brings the entire build to a halt

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented May 19, 2018

do the best you can, run 117!

@hvesalai

This comment has been minimized.

Copy link

hvesalai commented May 19, 2018

You need to add --add-modules java.xml.bind -XX:+IgnoreUnrecognizedVMOptions to JVM parameters

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented May 19, 2018

thanks, that does it (--add-modules=java.xml.bind). next up: run 118

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented May 19, 2018

latest is run 121

in my local testing, we're now very close to being able to declare victory (as in, enough green projects, not all of them), but there is one more keystone project that is failing and taking out a big swath of the build, namely specs2. I've filed etorreborre/specs2#659 on it

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented May 20, 2018

oh and there's one other thing I need to fix, which is that I pushed a "fix" for Scala.js that hardcodes /Users/tisue which obviously doesn't work on Jenkins. I think we shouldn't need to be setting scala.ext.dirs explicitly like this, it should be set for us by the sbt launch script, except I guess on Jenkins we have an outdated one.

locally, with Homebrew-installed sbt, if I run sbt on Java 9 and do eval System.getProperty("scala.ext.dirs") I get a path, but if I grab /usr/local/bin/sbt off of Jenkins (it's the sbt-extras one, not sure what version) and run that locally, scala.ext.dirs is null

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Jun 11, 2018

specs2 needs scala-js-stubs, which I haven't fixed yet (still the highest priority needed fix)

scala-collection-compat and sourcecode were both failing because of a too-old sbt-osgi version; I've pushed fixes for both

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Jun 12, 2018

I pushed a scala-js-stubs fix. https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk9-integrate-community-build/182/consoleFull has scala-js-stubs green. the rest is still running, I'll swing back by tomorrow and see how it did.

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Jun 13, 2018

119 projects are green now! declaring victory.

@SethTisue SethTisue closed this Jun 13, 2018

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Jun 13, 2018

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Jun 17, 2018

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Jun 26, 2018

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Jun 26, 2018

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Jul 13, 2018

status at https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk9-integrate-community-build/274/ : [info] >>> The dbuild result is----------------: FAILED (failed: paradox, akka-actor, slick, jackson-module-scala, http4s-websocket, fs2, kxbmap-configs, zinc)

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Jul 31, 2018

141 green projects now! https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk9-integrate-community-build/310/console has failed: paradox, slick, scalikejdbc, kxbmap-configs, specs2-more, fs2, github4s, pureconfig, http4s-websocket, scala-debugger, twitter-util, zinc, akka-stream

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Jul 31, 2018

someone want to look into the specs2-more and zinc failures...? or any of the others?

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Aug 6, 2018

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Aug 8, 2018

the expected-to-fail list is now kept in this repo itself, here: https://github.com/scala/community-builds/blob/2.12.x/report/Report.scala#L38

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Aug 8, 2018

PR to fix blaze: http4s/blaze#213

we are using a frozen blaze so we won't immediately get the fix once this is merged. but perhaps http4s has moved on and could handle blaze master now, I should try it and see

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Aug 9, 2018

ticket on zinc failure: sbt/zinc#567

milessabin added a commit to milessabin/shapeless that referenced this issue Aug 27, 2018

upgrade sbt-api-mappings so build works on JDK 9
this came up in the context of the Scala community build,
at scala/community-builds#609
@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Sep 19, 2018

ticket on boopickle failure:
suzaku-io/boopickle#105

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Sep 19, 2018

perhaps http4s has moved on and could handle blaze master now, I should try it and see

nope, we had to keep blaze frozen for now. I could re-fork it and cherry-pick
http4s/blaze#213

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment