New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2.13: move to new collections #710

Open
SethTisue opened this Issue Apr 12, 2018 · 68 comments

Comments

Projects
None yet
@SethTisue
Copy link
Member

SethTisue commented Apr 12, 2018

774c22c flips the switch. now to deal with the failures

@SethTisue SethTisue self-assigned this Apr 12, 2018

SethTisue added a commit that referenced this issue Apr 12, 2018

2.13: fix some projects, remove others
trying to get down to a minimal core of green projects plus
projects that are very high priority to get green

references #710
@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Apr 12, 2018

after 1338c90, in local testing,

  • these are green (10): scala-xml, cloc-plugin, machinist, sbt-testng, scala-collection-compat, scala-js-stubs, scala-partest, scala-partest-interface, scalaj-http, sourcecode
  • these key projects fail but I suggest we try hard to get them green before M4 (4): macro-paradise, scala-parser-combinators, scalacheck, kind-projector
  • projects currently w/ upstream failures, but also probably essential for M4 (4): scalatest, scala-java8-compat, scala-parallel-collections, mima
  • stretch goals for M4 (4): akka-actor, scalaz, specs2, scala-js
  • I had to work around a scala-xml issue that will go away once scala-xml is out of the bootstrap; I will prioritize that PR
@lrytz

This comment has been minimized.

Copy link
Member

lrytz commented Apr 12, 2018

for scalacheck there's a pr / branch. on my phone right now

@lrytz

This comment has been minimized.

SethTisue added a commit that referenced this issue Apr 12, 2018

SethTisue added a commit that referenced this issue Apr 12, 2018

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Apr 13, 2018

@lrytz thanks, using that now. and I fixed kind-projector.

nscala-time and sbinary are now giving:

[nscala-time] [error] **** Missing dependency: the library org.scalacheck#scalacheck is not provided (in space "scala") by any project in this configuration file.

which puzzles me. perhaps set publishArtifact in packageDoc in jvm := false is somehow not as selective as it looks

@lrytz

This comment has been minimized.

Copy link
Member

lrytz commented Apr 13, 2018

Oh I forgot to mention: if you build scalacheck from my branch (https://github.com/lrytz/scalacheck/tree/newCollections), the groupId changed to org.scala-lang.modules (lrytz/scalacheck@2013eca). So you'd have to revert that.

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Apr 13, 2018

the groupId changed to org.scala-lang.modules

ahhhhhhhhh :-)

@xuwei-k

This comment has been minimized.

Copy link
Contributor

xuwei-k commented Apr 19, 2018

@julienrf

This comment has been minimized.

Copy link

julienrf commented Apr 20, 2018

@xuwei-k

This comment has been minimized.

Copy link
Contributor

xuwei-k commented Apr 21, 2018

@lrytz

This comment has been minimized.

Copy link
Member

lrytz commented Apr 23, 2018

For the record, to not duplicate work: I'm working on scala-java8-compat (almost done), i have a local path for genjavadoc, and @ennru created a branch for akka-actor (https://github.com/ennru/akka/tree/newCollections).

@julienrf

This comment has been minimized.

Copy link

julienrf commented Apr 24, 2018

@SethTisue kind-projector compiles without warning and the tests pass with scala-2.13.0-pre-1c0be29. Can you point me to the error you faced?

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Apr 24, 2018

@adriaanm

This comment has been minimized.

Copy link
Member

adriaanm commented Apr 26, 2018

I'll take macro-paradise

@milessabin

This comment has been minimized.

Copy link

milessabin commented Apr 26, 2018

@adriaanm ping me when you're done with that and I'll get shapeless into shape.

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented May 5, 2018

status report:

  • green key projects: kind-projector, scala-collection-compat, scala-js-stubs, scala-parser-combinators, scala-xml, scalacheck, scalaz
  • also green: cloc-plugin, machinist, nscala-time, sbt-testng, scala-partest, scalaj-http, sourcecode
  • key projects failing, high priority for M4: scalatest, macro-paradise
  • stretch goals for M4: scala-java8-compat, scala-parallel-collections, genjavadoc, akka-actor, specs2, scala-js, shapeless
  • more stretch goals for M4: sbinary, fastparse, scalaprops

we would like to get M4 out in time for Scala Days Berlin (May 14)

@SethTisue SethTisue added the tracking label May 7, 2018

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented May 7, 2018

macro-paradise progress can be followed at scala/scala#6606

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented May 7, 2018

these are the currently failing projects. as in all their upstream dependencies are green, but the project itself fails:

  • scala-refactoring, scalaprops, tut, sbinary, scala-js, sbt-io, scala-async, json4s, utest, scalatest, scala-collections-laws, scala-continuations

volunteers wanted, to tackle particular projects. or at least to investigate and report back on the nature of the failure and the expected difficulty of fixing it

https://scala-ci.typesafe.com/job/scala-2.13.x-integrate-community-build/1091/consoleFull has the failure logs.

xuwei-k added a commit to xuwei-k/json4s that referenced this issue May 8, 2018

xuwei-k added a commit to xuwei-k/io that referenced this issue May 8, 2018

fix compile error in Scala 2.13.0-M4
- scala/community-builds#710
- https://scala-ci.typesafe.com/job/scala-2.13.x-integrate-community-build/1092/consoleFull

```
[sbt-io] [error] /home/jenkins/workspace/scala-2.13.x-integrate-community-build/target-0.9.12/project-builds/sbt-io-1aeb11a3d751ba13cb0d1d22d082306f3c24a209/io/src/main/scala/sbt/io/MacOSXWatchService.scala:61:32: overloaded method value flatMap with alternatives:
[sbt-io] [error]   [K2, V2](f: ((java.nio.file.Path, sbt.io.MacOSXWatchKey)) => scala.collection.IterableOnce[(K2, V2)])scala.collection.mutable.Map[K2,V2] <and>
[sbt-io] [error]   [B](f: ((java.nio.file.Path, sbt.io.MacOSXWatchKey)) => scala.collection.IterableOnce[B])scala.collection.mutable.Iterable[B]
[sbt-io] [error]  cannot be applied to (((java.nio.file.Path, sbt.io.MacOSXWatchKey)) => Option[(sbt.io.MacOSXWatchKey, scala.collection.mutable.Buffer[java.nio.file.WatchEvent[java.nio.file.Path]])])
[sbt-io] [error]       .synchronized(registered.flatMap {
[sbt-io] [error]                                ^
[sbt-io] [error] /home/jenkins/workspace/scala-2.13.x-integrate-community-build/target-0.9.12/project-builds/sbt-io-1aeb11a3d751ba13cb0d1d22d082306f3c24a209/io/src/main/scala/sbt/io/MacOSXWatchService.scala:61:32: overloaded method value flatMap with alternatives:
[sbt-io] [error]   [K2, V2](f: ((java.nio.file.Path, sbt.io.MacOSXWatchKey)) => scala.collection.IterableOnce[(K2, V2)])scala.collection.mutable.Map[K2,V2] <and>
[sbt-io] [error]   [B](f: ((java.nio.file.Path, sbt.io.MacOSXWatchKey)) => scala.collection.IterableOnce[B])scala.collection.mutable.Iterable[B]
[sbt-io] [error]  cannot be applied to (((java.nio.file.Path, sbt.io.MacOSXWatchKey)) => Option[(sbt.io.MacOSXWatchKey, scala.collection.mutable.Buffer[java.nio.file.WatchEvent[java.nio.file.Path]])])
[sbt-io] [error]       .synchronized(registered.flatMap {
[sbt-io] [error]                                ^
[sbt-io] [warn] one warning found
[sbt-io] [error] one error found
[sbt-io] [info] No documentation generated with unsuccessful compiler run
[sbt-io] [warn] two warnings found
[sbt-io] [error] one error found
[sbt-io] [error] (io / Compile / compileIncremental) Compilation failed
[sbt-io] [error] (io / Compile / doc) Scaladoc generation failed
```
@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented May 8, 2018

@xuwei-k there is now a further error in sbinary:

[sbinary] [error] /Users/tisue/community.213/target-0.9.12/project-builds/sbinary-a0f53c0c59caf8e1feefc3b1a5e88ad36a208588/core/target/scala-2.13.0-pre-ac5c8fa/src_managed/main/standardtypes.scala:1055: not enough arguments for method apply: (from: Nothing)scala.collection.mutable.Builder[T,CC[T]] in trait BuildFrom.
[sbinary] [error] Unspecified value parameter from.
[sbinary] [error]         val builder = cbf.apply()
[sbinary] [error]                                ^
[sbinary] [error] /Users/tisue/community.213/target-0.9.12/project-builds/sbinary-a0f53c0c59caf8e1feefc3b1a5e88ad36a208588/core/target/scala-2.13.0-pre-ac5c8fa/src_managed/main/standardtypes.scala:1055: not enough arguments for method apply: (from: Nothing)scala.collection.mutable.Builder[T,CC[T]] in trait BuildFrom.
[sbinary] [error] Unspecified value parameter from.
[sbinary] [error]         val builder = cbf.apply()
[sbinary] [error]                                ^
@MasseGuillaume

This comment has been minimized.

Copy link
Member

MasseGuillaume commented Jun 19, 2018

@xuwei-k

This comment has been minimized.

@xuwei-k

This comment has been minimized.

Copy link
Contributor

xuwei-k commented Jun 19, 2018

@dwijnand

This comment has been minimized.

Copy link
Member

dwijnand commented Jun 19, 2018

Should we create a wiki page, in either this repo or scala/scala, to track this?

(ala https://github.com/sbt/sbt/wiki/sbt-1.x-plugin-migration / https://github.com/sbt/sbt/wiki/library-sbt-1.x-migration)

@MasseGuillaume

This comment has been minimized.

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Jul 18, 2018

currently (here's a recent run) 41 projects are green

the highest priority failures (the ones blocking the most projects downstream are):

  • akka-actor (the Akka team is working on it, with Guillaume M)
  • utest (needs volunteer)
  • minitest (Alexandru is on vacation)
  • cats (Typelevel folks are working on publishing for M4, but could perhaps use volunteer assistance)
  • shapeless (@milessabin?)

the full failures list (not including projects blocked by failing dependencies) currently is: magnolia, scala-js, scala-refactoring, scala-async, minitest, shapeless, scala-continuations, utest, scala-collections-laws, linter, spray-json, twirl, scallop, sksamuel-exts, scalamock, paradox, mima, scalajson, grizzled, slick, scoverage, jackson-module-scala, scala-stm, parboiled, akka-actor, scapegoat, pcplod, scalikejdbc, lift-json, cachecontrol, scalastyle, scalaz8, log4s, http4s-parboiled2, scalatest-tests, scala-swing, cats, tut

misc notes:

  • scala-js and spray-json are blocked by scala-parallel-collections being missing
  • I just pushed a commit that will hopefully make jackson-module-scala green
@sjrd

This comment has been minimized.

Copy link
Member

sjrd commented Jul 19, 2018

For Scala.js, while waiting for the parallel collections, you could try pruning this line to only keep "testSuite", removing io, logging and linker. The testSuite is supposed to pass.

@milessabin

This comment has been minimized.

Copy link

milessabin commented Jul 25, 2018

@SethTisue scala/scala PR addressing the shapeless failure here: scala/scala#6971.

@xuwei-k

This comment has been minimized.

Copy link
Contributor

xuwei-k commented Aug 31, 2018

Create new wiki page for 2.13.0-M5? or reuse https://github.com/scala/community-builds/wiki/scala-2.13.x-migration?

@dwijnand

This comment has been minimized.

Copy link
Member

dwijnand commented Aug 31, 2018

I'd say let's keep that URL from M5, and move the M4 content to another page.

@sjrd

This comment has been minimized.

Copy link
Member

sjrd commented Sep 15, 2018

utest should be back on track lihaoyi/utest#180

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Oct 24, 2018

Haven't posted an update here in a while, since I have been concentrating instead on encouraging library authors to publish for 2.13.0-M5, since usually that results in the project going green in the community build as well.

but anyway, the current status in a recent run such as https://scala-ci.typesafe.com/job/scala-2.13.x-integrate-community-build/1529/ is:

unexpected FAILED: ssl-config
SUCCESS 64 FAILED 41 DID NOT RUN 83 TOTAL 188
UNEXPECTED 1
BLOCKERS:
42 cats
22 scala-java8-compat
22 fastparse
20 ssl-config
15 geny
14 pprint
14 scalatags
10 jackson-module-scala
9 jawn-0-10
6 twirl
5 scala-async
5 cachecontrol
3 slick
3 scalamock
3 tut
2 log4s
2 pcplod
2 grizzled
1 http4s-parboiled2
1 base64
1 boopickle
1 perfolation
1 lift-json 

once cats is published for M5, it should go green here as well. scala-java8-compat is on my plate, but it would be awesome if someone volunteered to tackle it. ssl-config can wait until #808 is merged. almost everything else on the list is probably fair game if anyone wants to try to help the maintainers move to 2.13.

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Oct 24, 2018

(note that the logs on Jenkins now end with these stats and the sorted list, I added that in August, 1b5e0e6)

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Nov 20, 2018

a sample recent run is https://scala-ci.typesafe.com/job/scala-2.13.x-integrate-community-build/1593/console

the big news is that cats is now green... except it's missing cats-kernel-laws, which several downstream projects need (algebra, circe, kittens, paiges)

the current top-blockers list is:

21 scala-java8-compat   (@lrytz is looking into it)
19 fastparse
12 geny
12 scalatags
11 cats-effect
11 pprint
11 paiges
10 jackson-module-scala
9 jawn-0-10
8 circe
5 scala-async
5 cachecontrol
3 slick
3 scalamock
3 algebra

the numbers are how many downstream projects are blocked

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Dec 3, 2018

let's see how https://scala-ci.typesafe.com/job/scala-2.13.x-integrate-community-build/1625/ does

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Dec 4, 2018

at https://scala-ci.typesafe.com/job/scala-2.13.x-integrate-community-build/1630/ we're up to 78 green projects

unexpected FAILED: fastparse
SUCCESS 78 FAILED 47 DID NOT RUN 63 TOTAL 188
UNEXPECTED 1
BLOCKERS:
21 scala-java8-compat
19 fastparse
11 pprint
10 jackson-module-scala
9 jawn-0-10
5 scala-async
5 cachecontrol
3 slick
3 scalamock
2 specs2-more
2 log4s
2 jsoniter-scala
2 monix
2 fs2
2 spire
2 pcplod
2 tut
2 grizzled
@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Dec 4, 2018

[fastparse] [error] /home/jenkins/workspace/scala-2.13.x-integrate-community-build/target-0.9.16/project-builds/fastparse-e5427c7be70c56d55f818bd736a6cce79443c8bc/scalaparse/shared/src/main/scala/scalaparse/Core.scala:83:17: not found: value _
[fastparse] [error]   val `_*` = P( `_` ~ `*` )
[fastparse] [error]                 ^

looks like fastparse ran afoul of scala/scala#6218, @som-snytt does this look as expected?

@som-snytt

This comment has been minimized.

Copy link

som-snytt commented Dec 5, 2018

I'm taking a look, but building fastparse against my local scalac snapshot is non-trivial because of dependencies. Is there a quick way for me to compile fastparse against some magic repo at lightbend?

I was about to just use scalac, which I don't mind for a one-off test, but SequencerGen is generated in the build, so everything is quickly out of scope.

The linked PR just specifies how to import a member named underscore. It should just work, unless a macro is fumbling an import.

plokhotnyuk added a commit to plokhotnyuk/jsoniter-scala that referenced this issue Dec 5, 2018

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Dec 5, 2018

building fastparse against my local scalac snapshot is non-trivial because of dependencies. Is there a quick way for me to compile fastparse against some magic repo at lightbend

no (#611)

the easiest thing is to use dbuild with a reduced config file so it doesn't extract dependencies from 180 projects you don't care about. I made this branch of the community build that has only fastparse and its dependencies: https://github.com/SethTisue/community-builds/tree/fastparse-only . just grab the branch and ./run.sh

@soronpo

This comment has been minimized.

Copy link

soronpo commented Dec 6, 2018

I made this branch of the community build that has only fastparse and its dependencies

Perhaps a nice feature will be to able to specify the required library as an (optional) argument and the community build script will only build that specific library and its dependencies.

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Dec 6, 2018

Perhaps a nice feature will be to able to specify the required library as an (optional) argument and the community build script will only build that specific library and its dependencies.

note that it does something like that already, in the sense that ./run.sh fastparse will build only fastparse and its dependencies. but in order to discover what those dependencies even are, every project must be extracted, which takes a while. the changes @cunei is working on at lightbend/dbuild#210 will speed up followup runs, but not first runs. so the "delete all the projects you don't care about" trick will remain useful even after Toni's changes land

so, yes: it would be nice if there was a way to say "don't even extract any projects I didn't mention on the command line". but deleting the entries you don't care about from the config file is generally not that hard, so it's not absolutely clear it would be worth Toni's time to automate it more, given that dbuild only has a few users

@soronpo

This comment has been minimized.

Copy link

soronpo commented Dec 6, 2018

Is it possible to somehow query scaladex to achieve this easier?

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Dec 6, 2018

@soronpo scaladex only has runtime dependencies. in this context we need compile-time dependencies, including test-only dependencies.

@som-snytt

This comment has been minimized.

Copy link

som-snytt commented Dec 7, 2018

Another great feature would be if Seth could stop by my cubicle. I ran the special branch but got errors, I'll spend time with it later.

Edit: I experimented again today, near the end of January, having forgotten I'd already tried to get back on the dbuild horse in December. Probably I shouldn't try that right before Pearl Harbor Day.

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Feb 13, 2019

I've deleted
https://github.com/scala/community-builds/wiki/scala-2.13.x-migration as it had fallen completely out of date. sorry, I know this was well-intentioned, but I didn't find it useful in my own (informal) process.

the best way for anyone to check the current status is to look at the log of a recent run at https://scala-ci.typesafe.com/job/scala-2.13.x-integrate-community-build/ and the expected-to-fail project list at https://github.com/scala/community-builds/blob/2.13.x/report/Report.scala (which I do actively maintain)

@SethTisue

This comment has been minimized.

Copy link
Member Author

SethTisue commented Feb 13, 2019

overall the current status is: we've been hovering at around 80 green projects. I've been tackling any significant regressions myself, enlisting project maintainers' help as needed

it's now been so long since M5, and so many little breaking changes have accumulated, that it's not really productive at present to do any better than ~80

after 2.13.0-RC1 is out, we can make a fresh attempt to get more projects green.

I have been concentrating instead on encouraging library authors to publish for 2.13.0-M5, since usually that results in the project going green in the community build as well.

(meanwhile, I have continued this nagging campaign)

@dwijnand

This comment has been minimized.

Copy link
Member

dwijnand commented Feb 14, 2019

I think https://github.com/scala/community-builds/wiki/scala-2.13.x-migration served the initial purpose of bootstrapping the ecosystem for the new breaking pre-release of 2.13. When 2.13.0-RC1 comes out it'll be useful to swarm over those usual suspect libraries that'll need updating. And when 2.13.0 is out we'll port even more than usual.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment