New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

flesh out JDK 11 and 12 community builds #796

Open
SethTisue opened this Issue Sep 20, 2018 · 38 comments

Comments

Projects
None yet
6 participants
@SethTisue
Copy link
Member

SethTisue commented Sep 20, 2018

in #742 we got the build up and running, all projects extracting, and enough green projects (18 green) to be sure that basics are working

next steps:

  • fix any new 11-only failures that are making the 11 build worse than the old 9 & 10 builds, the big glaring one is the scalacheck failure (because .lines)
  • then resume making gradual progress towards an all-green build, as we've been doing for much of 2018
@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Sep 21, 2018

I fixed these blockers: scalacheck, scalatest, utest, scala-java8-compat.

a new run is now in progress at https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk11-integrate-community-build/18/consoleFull and will hopefully have substantially more than 18 green projects

@smarter joked that "every Scala codebase ever" is going to hit scala/bug#11125, and it seems that's not far from the truth.

modifying projects one at a time to replace foo.lines with Predef.augmentString(foo).lines feels like the kind of work Judge Wills sentenced Oscar Wilde to. perhaps the problem can be solved community-build-wide with appendScalacOptions -release 8. I'll try it.

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Sep 21, 2018

with -release 8 multiple projects fail to compile because during Scaladoc generation we get the same "java.lang.AssertionError: assertion failed: type AnyRef in java.lang" errors @retronym and the Akka folks already saw back in March at akka/akka#24711 (comment)

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Sep 21, 2018

"As a workaround, try removing -release 8 from the scalacOptions in doc", suggests @retronym. our current addScalacOptions mechanism isn't customizable that way, but probably I could do it project-by-project (which isn't so painful when it's only our own config file I'm modifying, not having to push commits to lots of forked repos)

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Sep 21, 2018

60 projects are green now. top ​blockers: ​87 ​specs2, ​81 ​macro-paradise, ​22 ​fastparse, ​21 ​akka

I'll need to go through the failures one by one and see if any them represent regressions in 2.12.7.

@retronym

This comment has been minimized.

Copy link
Member

retronym commented Sep 21, 2018

Can we disable scaladoc generation globally and conveniently?

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Sep 21, 2018

@retronym good thought, I'll come back to it later.

the 14 failures at https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk11-integrate-community-build/18/consoleFull:

  • specs2, expecty, fastparse, macro-paradise, pprint, scala-partest, scala-refactoring, scalamock: scala/bug#11125
  • scodec-bits: newly ambiguous methods on ByteBuffer because Oracle added new overloads
  • akka: [akka] akka.actor.dungeon.SerializationCheckFailedException: Failed to serialize and deserialize message of type java.util.BitSet for testing. To avoid this error, either disable 'akka.actor.serialize-messages', mark the message with 'akka.actor.NoSerializationVerificationNeeded', or configure serialization to support this message
  • pcplod: scala.reflect.internal.MissingRequirementError: object scala.annotation.Annotation in compiler mirror not found

the only one of that looks maybe a bit worrisome is the pcplod one, but pcplod is an unusual project, I don't think we need to worry about it for 2.12.7

TL;DR looks good for 2.12.7

@adriaanm

This comment has been minimized.

Copy link
Member

adriaanm commented Sep 21, 2018

Thanks Seth! I know it’s late over there.

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Oct 1, 2018

etorreborre/specs2#710 should fix specs2

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Oct 1, 2018

scalacommunitybuild/paradise@e4b6a5f should fix macro paradise

@xuwei-k

This comment has been minimized.

Copy link
Contributor

xuwei-k commented Oct 1, 2018

scodec-bits: newly ambiguous methods on ByteBuffer because Oracle added new overloads

scodec/scodec-bits#98

@xuwei-k

This comment has been minimized.

Copy link
Contributor

xuwei-k commented Oct 2, 2018

@xuwei-k

This comment has been minimized.

Copy link
Contributor

xuwei-k commented Oct 3, 2018

@xuwei-k

This comment has been minimized.

Copy link
Contributor

xuwei-k commented Oct 3, 2018

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Oct 3, 2018

thank you, Yoshida-san, for continuing to chip away at this. I'll keep at it as well :-)

https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk11-integrate-community-build/57/ shows the current top blockers as:

BLOCKERS:
22 fastparse
21 akka
20 ssl-config
14 pprint
14 scalatags
13 expecty
4 fs2
4 case-app
3 slick
3 twitter-util
3 scalamock
1 blaze
@xuwei-k

This comment has been minimized.

Copy link
Contributor

xuwei-k commented Oct 4, 2018

@xuwei-k

This comment has been minimized.

Copy link
Contributor

xuwei-k commented Oct 5, 2018

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Oct 9, 2018

lightbend/ssl-config#128 (just an issue, not a PR)

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Oct 9, 2018

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Oct 11, 2018

coursier/coursier#935 (just an issue, not a PR)

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Oct 11, 2018

scalameta: f4dfa83

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Oct 11, 2018

@exoego exoego referenced this issue Oct 12, 2018

Merged

Add JDK11 support #67

@exoego

This comment has been minimized.

Copy link

exoego commented Oct 12, 2018

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Oct 18, 2018

139 green, up from 120 two weeks ago!

in some cases JDK 11 support PRs have been merged, in other cases I've (impatiently) changed our config to point to the PR branch

status as of https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk11-integrate-community-build/105/ is:

SUCCESS 139 FAILED 13 DID NOT RUN 37 TOTAL 189
UNEXPECTED 0
BLOCKERS:
20 ssl-config
13 metaconfig
4 fs2
3 coursier
3 slick
3 twitter-util
3 scalamock
1 blaze

from report/Report.scala:

  val expectedToFail: Set[String] =
    System.getProperty("java.specification.version") match {
      case "1.8" =>
        Set()
      case _ =>
        Set(
          "blaze",  // test failure (org.http4s.blaze.pipeline.stages.SSLStageSpec)
          "breeze",  // "object istack is not a member of package com.sun"
          "case-app",  // needs scala/bug#11125 workaround
          "coursier",  // needs scala/bug#11125 workaround
          "fs2",  // new overloads on JDK 11 need disambiguation
          "kxbmap-configs",  // "sjsonnew.DeserializationException: Field not found: $type"
          "metaconfig",  // needs scala/bug#11125 workaround
          "scala-refactoring",  // needs scala/bug#11125 workaround?
          "scalamock",  // needs scala/bug#11125 workaround
          "scalatex",  // scripted-plugin not found?
          "slick",  // "object bind is not a member of package javax.xml"
          "ssl-config", // https://github.com/lightbend/ssl-config/issues/98
          "twitter-util",  // "javadoc: error - invalid flag: -d"
        )
    }
@xuwei-k

This comment has been minimized.

Copy link
Contributor

xuwei-k commented Oct 19, 2018

"case-app", // needs scala/bug#11125 workaround

fixed in master alexarchambault/case-app@5023eaa

case-app-uri: "https://github.com/scalacommunitybuild/case-app.git#community-build-2.12" # was alexarchambault, master

"fs2", // new overloads on JDK 11 need disambiguation

functional-streams-for-scala/fs2@226ff68#diff-736c8aaa74103281c32f371842cf994bR41

fs2-uri: "https://github.com/functional-streams-for-scala/fs2.git#v1.0.0-M5"

"scalamock", // needs scala/bug#11125 workaround

paulbutcher/ScalaMock@ff52907

scalamock-uri: "https://github.com/paulbutcher/ScalaMock.git#1e84ffe" # was master

"twitter-util", // "javadoc: error - invalid flag: -d"

sbt/sbt#4350

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Oct 19, 2018

@xuwei-k thanks for all this info. I've handled them as follows:

  • case-app: cherry-picked the change onto our fork
  • fs2: rather than mess with forking, let's move to 1.0.0 and get the fix that way. new ticket: #804
  • scalamock: forked it and cherry-picked the change (f8d3fe7)
  • twitter-util: for now, just added a comment about it (f8d3fe7). let Eugene know on the sbt ticket that a fix would be nice to have

Dale hopes to fix ssl-config soon, lightbend/ssl-config#98

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Oct 19, 2018

@exoego

This comment has been minimized.

Copy link

exoego commented Oct 20, 2018

If my investigation and understanding is correct, JDK 9+ support on slick requires

  • Oracle XE 18c that will be released on Octoberer 2018 or later.
  • Public Docker image for Oracle XE 18c
@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Oct 24, 2018

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Oct 24, 2018

scalafmt is green at https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk11-integrate-community-build/126/, we're up to 148 green now

BLOCKERS:
20 ssl-config
4 fs2
3 coursier
3 slick
3 twitter-util
3 sbt-util
1 blaze

I'm working on ssl-config at #808 and fs2 at #805

sbt-util is an easy fix if someone wants to grab that one

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Oct 27, 2018

ssl-config, fs2, blaze went green

I'll ask the Akka folks about the akka-streams compile error

SUCCESS 153 FAILED 11 DID NOT RUN 24 TOTAL 188
BLOCKERS:
15 akka-stream
3 coursier
3 slick
3 twitter-util
3 sbt-util
@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Nov 27, 2018

I haven't had time to work on this in the past month. The blockers list remains as above.

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Dec 8, 2018

https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk11-integrate-community-build/231/

akka-stream is green now (I finally figured out a -release 9 needed to be -release 11 on some machines including our Jenkins, even though it worked on some other setups)

that brings us up to 165 green projects total, which is very satisfying. no major blockers remain

SUCCESS 165 FAILED 12 DID NOT RUN 14 TOTAL 191
UNEXPECTED 5
BLOCKERS:
3 coursier
3 slick
3 twitter-util
3 playframework
3 sbt-util
1 akka-http-cors

I tried unfreezing coursier (#823) but ran into trouble with downstream projects, I'll have to come back to that

the next failure to look at is playframework, @dwijnand want to take a look at the errors?

@dwijnand

This comment has been minimized.

Copy link
Member

dwijnand commented Dec 18, 2018

@dwijnand

This comment has been minimized.

Copy link
Member

dwijnand commented Jan 2, 2019

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Jan 31, 2019

lightbend/ssl-config#142 / playframework/play-meta#4 makes the Play situation messy, so I'm just going to sit tight and wait for that to get sorted out

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Jan 31, 2019

if anyone else is watching this ticket and would like to help out, the current failures list is: scala-refactoring, compiler-bridge-1-0, twitter-util, slick, kxbmap-configs, coursier, sconfig, scala-debugger, breeze, doobie, playframework, sbt-util

and what's blocking downstream projects is:

3 coursier
3 slick
3 twitter-util
3 playframework
3 sbt-util
@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Feb 13, 2019

it's time to start testing on JDK 12, too: #862

@SethTisue SethTisue changed the title flesh out JDK 11 community build flesh out JDK 11 and 12 community builds Feb 16, 2019

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Feb 16, 2019

#862 is complete, all projects extract. I've broadened the scope of this ticket to cover both 11 and 12

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment