2.13: move to new collections

[SethTisue]

774c22c flips the switch. now to deal with the failures

[SethTisue]

after 1338c90, in local testing,

• these are green (10): scala-xml, cloc-plugin, machinist, sbt-testng, scala-collection-compat, scala-js-stubs, scala-partest, scala-partest-interface, scalaj-http, sourcecode
• these key projects fail but I suggest we try hard to get them green before M4 (4): macro-paradise, scala-parser-combinators, scalacheck, kind-projector
• projects currently w/ upstream failures, but also probably essential for M4 (4): scalatest, scala-java8-compat, scala-parallel-collections, mima
• stretch goals for M4 (4): akka-actor, scalaz, specs2, scala-js
• I had to work around a scala-xml issue that will go away once scala-xml is out of the bootstrap; I will prioritize that PR

[lrytz]

for scalacheck there's a pr / branch. on my phone right now

[lrytz]

rickynils/scalacheck#391 / https://github.com/lrytz/scalacheck/tree/newCollections?files=1

[SethTisue]

@lrytz thanks, using that now. and I fixed kind-projector.

nscala-time and sbinary are now giving:

[nscala-time] [error] **** Missing dependency: the library org.scalacheck#scalacheck is not provided (in space "scala") by any project in this configuration file.

which puzzles me. perhaps set publishArtifact in packageDoc in jvm := false is somehow not as selective as it looks

[lrytz]

Oh I forgot to mention: if you build scalacheck from my branch (https://github.com/lrytz/scalacheck/tree/newCollections), the groupId changed to org.scala-lang.modules (lrytz/scalacheck@2013eca). So you'd have to revert that.

[SethTisue]

the groupId changed to org.scala-lang.modules

ahhhhhhhhh :-)

[xuwei-k]

scalaz/scalaz#1730

[julienrf]

scala/scala-parser-combinators#148

[xuwei-k]

fixed scalaprops scalaprops/scalaprops@b98d71c

[lrytz]

For the record, to not duplicate work: I'm working on scala-java8-compat (almost done), i have a local path for genjavadoc, and @ennru created a branch for akka-actor (https://github.com/ennru/akka/tree/newCollections).

[julienrf]

@SethTisue kind-projector compiles without warning and the tests pass with scala-2.13.0-pre-1c0be29. Can you point me to the error you faced?

[SethTisue]

@julienrf it was non/kind-projector#68

[adriaanm]

I'll take macro-paradise

[milessabin]

@adriaanm ping me when you're done with that and I'll get shapeless into shape.

[SethTisue]

status report:

• green key projects: kind-projector, scala-collection-compat, scala-js-stubs, scala-parser-combinators, scala-xml, scalacheck, scalaz
• also green: cloc-plugin, machinist, nscala-time, sbt-testng, scala-partest, scalaj-http, sourcecode
• key projects failing, high priority for M4: scalatest, macro-paradise
• stretch goals for M4: scala-java8-compat, scala-parallel-collections, genjavadoc, akka-actor, specs2, scala-js, shapeless
• more stretch goals for M4: sbinary, fastparse, scalaprops

we would like to get M4 out in time for Scala Days Berlin (May 14)

[SethTisue]

macro-paradise progress can be followed at scala/scala#6606

[SethTisue]

these are the currently failing projects. as in all their upstream dependencies are green, but the project itself fails:

• scala-refactoring, scalaprops, tut, sbinary, scala-js, sbt-io, scala-async, json4s, utest, scalatest, scala-collections-laws, scala-continuations

volunteers wanted, to tackle particular projects. or at least to investigate and report back on the nature of the failure and the expected difficulty of fixing it

https://scala-ci.typesafe.com/job/scala-2.13.x-integrate-community-build/1091/consoleFull has the failure logs.

[SethTisue]

@xuwei-k there is now a further error in sbinary:

[sbinary] [error] /Users/tisue/community.213/target-0.9.12/project-builds/sbinary-a0f53c0c59caf8e1feefc3b1a5e88ad36a208588/core/target/scala-2.13.0-pre-ac5c8fa/src_managed/main/standardtypes.scala:1055: not enough arguments for method apply: (from: Nothing)scala.collection.mutable.Builder[T,CC[T]] in trait BuildFrom.
[sbinary] [error] Unspecified value parameter from.
[sbinary] [error]         val builder = cbf.apply()
[sbinary] [error]                                ^
[sbinary] [error] /Users/tisue/community.213/target-0.9.12/project-builds/sbinary-a0f53c0c59caf8e1feefc3b1a5e88ad36a208588/core/target/scala-2.13.0-pre-ac5c8fa/src_managed/main/standardtypes.scala:1055: not enough arguments for method apply: (from: Nothing)scala.collection.mutable.Builder[T,CC[T]] in trait BuildFrom.
[sbinary] [error] Unspecified value parameter from.
[sbinary] [error]         val builder = cbf.apply()
[sbinary] [error]                                ^

[MasseGuillaume]

moved to: https://github.com/scala/community-builds/wiki/scala-2.13.x-migration

[xuwei-k]

• eed3si9n/sjson-new#95
• lightbend/scala-logging#124
• unfiltered/unfiltered@6150fc4

[xuwei-k]

scalapb/protobuf-scala-runtime#21

[dwijnand]

Should we create a wiki page, in either this repo or scala/scala, to track this?

(ala https://github.com/sbt/sbt/wiki/sbt-1.x-plugin-migration / https://github.com/sbt/sbt/wiki/library-sbt-1.x-migration)

[MasseGuillaume]

@dwijnand https://github.com/scala/community-builds/wiki/scala-2.13.x-migration

[SethTisue]

currently (here's a recent run) 41 projects are green

the highest priority failures (the ones blocking the most projects downstream are):

• akka-actor (the Akka team is working on it, with Guillaume M)
• utest (needs volunteer)
• minitest (Alexandru is on vacation)
• cats (Typelevel folks are working on publishing for M4, but could perhaps use volunteer assistance)
• shapeless (@milessabin?)

the full failures list (not including projects blocked by failing dependencies) currently is: magnolia, scala-js, scala-refactoring, scala-async, minitest, shapeless, scala-continuations, utest, scala-collections-laws, linter, spray-json, twirl, scallop, sksamuel-exts, scalamock, paradox, mima, scalajson, grizzled, slick, scoverage, jackson-module-scala, scala-stm, parboiled, akka-actor, scapegoat, pcplod, scalikejdbc, lift-json, cachecontrol, scalastyle, scalaz8, log4s, http4s-parboiled2, scalatest-tests, scala-swing, cats, tut

misc notes:

• scala-js and spray-json are blocked by scala-parallel-collections being missing
• I just pushed a commit that will hopefully make jackson-module-scala green

[dwijnand]

In addition to https://github.com/scala/community-builds/wiki/scala-2.13.x-migration
see also https://github.com/scala/make-release-notes/blob/2.13.x/projects-2.13.md

[sjrd]

For Scala.js, while waiting for the parallel collections, you could try pruning this line to only keep "testSuite", removing io, logging and linker. The testSuite is supposed to pass.

[milessabin]

@SethTisue scala/scala PR addressing the shapeless failure here: scala/scala#6971.

[xuwei-k]

Create new wiki page for 2.13.0-M5? or reuse https://github.com/scala/community-builds/wiki/scala-2.13.x-migration?

[dwijnand]

I'd say let's keep that URL from M5, and move the M4 content to another page.

[sjrd]

utest should be back on track lihaoyi/utest#180

[SethTisue]

Haven't posted an update here in a while, since I have been concentrating instead on encouraging library authors to publish for 2.13.0-M5, since usually that results in the project going green in the community build as well.

but anyway, the current status in a recent run such as https://scala-ci.typesafe.com/job/scala-2.13.x-integrate-community-build/1529/ is:

unexpected FAILED: ssl-config
SUCCESS 64 FAILED 41 DID NOT RUN 83 TOTAL 188
UNEXPECTED 1
BLOCKERS:
42 cats
22 scala-java8-compat
22 fastparse
20 ssl-config
15 geny
14 pprint
14 scalatags
10 jackson-module-scala
9 jawn-0-10
6 twirl
5 scala-async
5 cachecontrol
3 slick
3 scalamock
3 tut
2 log4s
2 pcplod
2 grizzled
1 http4s-parboiled2
1 base64
1 boopickle
1 perfolation
1 lift-json 

once cats is published for M5, it should go green here as well. scala-java8-compat is on my plate, but it would be awesome if someone volunteered to tackle it. ssl-config can wait until #808 is merged. almost everything else on the list is probably fair game if anyone wants to try to help the maintainers move to 2.13.

[SethTisue]

(note that the logs on Jenkins now end with these stats and the sorted list, I added that in August, 1b5e0e6)

[SethTisue]

a sample recent run is https://scala-ci.typesafe.com/job/scala-2.13.x-integrate-community-build/1593/console

the big news is that cats is now green... except it's missing cats-kernel-laws, which several downstream projects need (algebra, circe, kittens, paiges)

the current top-blockers list is:

21 scala-java8-compat   (@lrytz is looking into it)
19 fastparse
12 geny
12 scalatags
11 cats-effect
11 pprint
11 paiges
10 jackson-module-scala
9 jawn-0-10
8 circe
5 scala-async
5 cachecontrol
3 slick
3 scalamock
3 algebra

the numbers are how many downstream projects are blocked

[SethTisue]

• geny: 4e4715e
• fastparse: 830c7c2
• cats-kernel-laws: 7d05815
• cats-effect: 7249547

let's see how https://scala-ci.typesafe.com/job/scala-2.13.x-integrate-community-build/1625/ does

[SethTisue]

at https://scala-ci.typesafe.com/job/scala-2.13.x-integrate-community-build/1630/ we're up to 78 green projects

unexpected FAILED: fastparse
SUCCESS 78 FAILED 47 DID NOT RUN 63 TOTAL 188
UNEXPECTED 1
BLOCKERS:
21 scala-java8-compat
19 fastparse
11 pprint
10 jackson-module-scala
9 jawn-0-10
5 scala-async
5 cachecontrol
3 slick
3 scalamock
2 specs2-more
2 log4s
2 jsoniter-scala
2 monix
2 fs2
2 spire
2 pcplod
2 tut
2 grizzled

[SethTisue]

[fastparse] [error] /home/jenkins/workspace/scala-2.13.x-integrate-community-build/target-0.9.16/project-builds/fastparse-e5427c7be70c56d55f818bd736a6cce79443c8bc/scalaparse/shared/src/main/scala/scalaparse/Core.scala:83:17: not found: value _
[fastparse] [error]   val `_*` = P( `_` ~ `*` )
[fastparse] [error]                 ^

looks like fastparse ran afoul of scala/scala#6218, @som-snytt does this look as expected?

[som-snytt]

I'm taking a look, but building fastparse against my local scalac snapshot is non-trivial because of dependencies. Is there a quick way for me to compile fastparse against some magic repo at lightbend?

I was about to just use scalac, which I don't mind for a one-off test, but SequencerGen is generated in the build, so everything is quickly out of scope.

The linked PR just specifies how to import a member named underscore. It should just work, unless a macro is fumbling an import.

[SethTisue]

building fastparse against my local scalac snapshot is non-trivial because of dependencies. Is there a quick way for me to compile fastparse against some magic repo at lightbend

no (#611)

the easiest thing is to use dbuild with a reduced config file so it doesn't extract dependencies from 180 projects you don't care about. I made this branch of the community build that has only fastparse and its dependencies: https://github.com/SethTisue/community-builds/tree/fastparse-only . just grab the branch and ./run.sh

[soronpo]

I made this branch of the community build that has only fastparse and its dependencies

Perhaps a nice feature will be to able to specify the required library as an (optional) argument and the community build script will only build that specific library and its dependencies.

[SethTisue]

Perhaps a nice feature will be to able to specify the required library as an (optional) argument and the community build script will only build that specific library and its dependencies.

note that it does something like that already, in the sense that ./run.sh fastparse will build only fastparse and its dependencies. but in order to discover what those dependencies even are, every project must be extracted, which takes a while. the changes @cunei is working on at lightbend/dbuild#210 will speed up followup runs, but not first runs. so the "delete all the projects you don't care about" trick will remain useful even after Toni's changes land

so, yes: it would be nice if there was a way to say "don't even extract any projects I didn't mention on the command line". but deleting the entries you don't care about from the config file is generally not that hard, so it's not absolutely clear it would be worth Toni's time to automate it more, given that dbuild only has a few users

[soronpo]

Is it possible to somehow query scaladex to achieve this easier?

[SethTisue]

@soronpo scaladex only has runtime dependencies. in this context we need compile-time dependencies, including test-only dependencies.

[som-snytt]

Another great feature would be if Seth could stop by my cubicle. I ran the special branch but got errors, I'll spend time with it later.

Edit: I experimented again today, near the end of January, having forgotten I'd already tried to get back on the dbuild horse in December. Probably I shouldn't try that right before Pearl Harbor Day.

[SethTisue]

I've deleted
https://github.com/scala/community-builds/wiki/scala-2.13.x-migration as it had fallen completely out of date. sorry, I know this was well-intentioned, but I didn't find it useful in my own (informal) process.

the best way for anyone to check the current status is to look at the log of a recent run at https://scala-ci.typesafe.com/job/scala-2.13.x-integrate-community-build/ and the expected-to-fail project list at https://github.com/scala/community-builds/blob/2.13.x/report/Report.scala (which I do actively maintain)

[SethTisue]

overall the current status is: we've been hovering at around 80 green projects. I've been tackling any significant regressions myself, enlisting project maintainers' help as needed

it's now been so long since M5, and so many little breaking changes have accumulated, that it's not really productive at present to do any better than ~80

after 2.13.0-RC1 is out, we can make a fresh attempt to get more projects green.

I have been concentrating instead on encouraging library authors to publish for 2.13.0-M5, since usually that results in the project going green in the community build as well.

(meanwhile, I have continued this nagging campaign)

[dwijnand]

I think https://github.com/scala/community-builds/wiki/scala-2.13.x-migration served the initial purpose of bootstrapping the ecosystem for the new breaking pre-release of 2.13. When 2.13.0-RC1 comes out it'll be useful to swarm over those usual suspect libraries that'll need updating. And when 2.13.0 is out we'll port even more than usual.