unfreezing scala-parallel-collections caused failures downstream

[SethTisue]

something to do with sbt-scala-module 2.1

there are some notes at #980 (comment)

splitting that out as a separate issue so I can get #980 merged. #980 freezes scala-parallel-collections at an older version for now

[lrytz]

sbt/sbt#5043 maybe?

I hit this bug here: https://github.com/scala/scala-collection-contrib/blob/master/project/build.properties

[SethTisue]

the problem is reproducible outside of dbuild by cloning scala-parallel-collections, doing core/publishLocal, and then cloning twitter-util (develop branch), replacing e.g.

-        Seq("org.scala-lang.modules" %% "scala-parallel-collections" % "0.2.0" % "test")
+        Seq("org.scala-lang.modules" %% "scala-parallel-collections" % "0.2.0+18-693c5fb1-SNAPSHOT" % "test")

and then doing ++2.13.1 and util-stats/Test/compile. (regardless of what sbt version twitter-util uses, 1.2.8 or 1.3.2)

[SethTisue]

and you can cut twitter-util out of the picture with:

% scala213 -classpath ~/.ivy2/local/org.scala-lang.modules/scala-parallel-collections_2.13/0.2.0+18-693c5fb1-SNAPSHOT/bundles/scala-parallel-collections_2.13.jar
Welcome to Scala 2.13.1 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_202).
Type in expressions for evaluation. Or try :help.

scala 2.13.1> new collection.parallel.immutable.ParRange(1 to 50)
                  ^
              error: Symbol 'type scala.collection.generic.GenericParTemplate' is missing from the classpath.
              This symbol is required by 'trait scala.collection.parallel.ParIterable'.
              Make sure that type GenericParTemplate is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
              A full rebuild may help if 'ParIterable.class' was compiled against an incompatible version of scala.collection.generic.

[SethTisue]

the classes in question are in fact missing from the published JAR:

% jar tf ~/.ivy2/local/org.scala-lang.modules/scala-parallel-collections_2.13/0.2.0+18-693c5fb1-SNAPSHOT/bundles/scala-parallel-collections_2.13.jar | grep GenericPar
% jar tf ~/Library/Caches/Coursier/v1/http/127.0.0.1%3A8081/artifactory/scala-ci-virtual/org/scala-lang/modules/scala-parallel-collections_2.13/0.2.0/scala-parallel-collections_2.13-0.2.0.jar | grep GenericPar
scala/collection/generic/GenericParCompanion.class
scala/collection/generic/GenericParCompanion$.class
scala/collection/generic/GenericParMapCompanion$.class
scala/collection/generic/GenericParMapCompanion$ToFactory.class
scala/collection/generic/GenericParMapCompanion.class
scala/collection/generic/GenericParCompanion$ToFactory.class
scala/collection/generic/GenericParMapTemplate.class
scala/collection/generic/GenericParTemplate.class

[SethTisue]

in the publishLocal output, this seems possibly relevant:

[warn] bnd: Unused Private-Package instructions, no such package(s) on the class path: [org.scala-lang.modules.scala-parallel-collections.*]
[warn] bnd: Version for package scala.collection is set to different values in the source (0.2.0+18-693c5fb1-SNAPSHOT) and in the manifest (2.13.0.v20190604-151517-VFINAL-43e040f). The version in the manifest is not picked up by an other sibling bundles in this project or projects that directly depend on this project
[warn] bnd: Version for package scala.collection.generic is set to different values in the source (0.2.0+18-693c5fb1-SNAPSHOT) and in the manifest (2.13.0.v20190604-151517-VFINAL-43e040f). The version in the manifest is not picked up by an other sibling bundles in this project or projects that directly depend on this project

@lrytz note that I charge extra for OSGi work

[SethTisue]

GenericParTemplate.class is in the core/package JAR but not the core/publishLocal JAR

some of the scala.collection.generic classes make it through, but not all of them

[SethTisue]

surely the problem is here somewhere?

  OsgiKeys.exportPackage := Seq(
    s"scala.collection.parallel.*;version=${version.value}",
    // The first entry on the classpath is the project's target classes dir but sbt-osgi also passes all
    // dependencies to bnd. Any "merge" strategy for split packages would include the classes from scala-library.
    s"scala.collection;version=${version.value};-split-package:=first",
    s"scala.collection.generic;version=${version.value};-split-package:=first"
  ),
  // Use correct version for scala package imports
  OsgiKeys.importPackage := Seq(osgiImport("scala*", scalaVersion.value), "*"),

[SethTisue]

so it seems we want -split-package:=merge-first here, instead of :=first, but the comment seems to indicate that was already tried

but something is different now or this wouldn't be breaking, so it's worth another try I guess

the newer sbt-scala-modules includes an sbt-osgi version bump from 0.9.4 to 0.9.5. not clear to me at a glance if any of https://github.com/sbt/sbt-osgi/milestone/6?closed=1 are relevant, like maybe sbt/sbt-osgi#60 ? not obvious

[SethTisue]

hey @szeiger that's your code and your comment, how much do you charge for OSGi work? (and how much when you're on vacation?)

[SethTisue]

some of the scala.collection.generic classes make it through, but not all of them

oops, no, those are being copied from scala-library, rather than coming from within scala-parallel-collections

[SethTisue]

do we need to make scala-parallel-collections a "bundle fragment"? https://www.ibm.com/support/knowledgecenter/SSRTLW_9.5.0/com.ibm.aries.osgi.doc/topics/cbundlefragment.html

[SethTisue]

what's the old saying, a child can split a package and ten wise men can't put it back together, something like that?

[SethTisue]

the ticket on the unfortunateness of having split packages at all is scala/scala-parallel-collections#5

[SethTisue]

if you do:

-    s"scala.collection;version=${version.value};-split-package:=first",
-    s"scala.collection.generic;version=${version.value};-split-package:=first"
+    s"scala.collection;version=${version.value};-split-package:=merge-first",
+    s"scala.collection.generic;version=${version.value};-split-package:=merge-first",
+    s"scala.collection.mutable;version=${version.value};-split-package:=merge-first",
+    s"scala.collection.immutable;version=${version.value};-split-package:=merge-first",

then you get all of the classes you want, but you also get a bunch of copies of stuff from Scala stdlib

in the community build context, this should be sufficient? let's try it

[SethTisue]

yup that did it