What kinds of changes/additions are possible?

[NthPortal]

1. For changes that require access to classes inside the scala package and outside of the scala.next package, what procedure should be followed? Do we pursue such changes at all? (If not, the scope of this library is drastically reduced.)
2. For changes that require access to the internals of a class in the standard library, how should such changes be made? Do we pursue such changes at all? (If not, the scope of this library is reduced.)
3. To what degree are we willing to duplicate/copy code from the standard library? (This question relates to possible answers to both 1 and 2.)

[NthPortal]

A possible solution to 1 is to add classes that are private[scala] and outside of the scala.next package, and then alias them or otherwise publicly expose them via shims within the scala.next. If that is done, we must be extremely careful not overlap any names between the standard library and this library. A possible solution to that is to prefix the classes to indicate they are part of this library (e.g. NV1_HashMap).

[SethTisue]

Overlapping question: what will the user experience be? Will individual extension methods need to be imported individually, or will we provide some sort of blanket import?

[SethTisue]

Also: do we care about split packages? Either for OSGi or (probably more importantly) the Java module system? Suppose we go ahead and allow ourselves to use split packages, will module systems be able to opt-out by exempting our JARs, or is it viral/transitive somehow?

[SethTisue]

When is the next full post-3.0 ecosystem reboot, when everything will need to rebuilt anyway so breaking bincompat everywhere is fair game? @sjrd says the next time might be 3.1 or 2.2 but in any case it will be the last reboot (since after that TASTy will take over)

[SethTisue]

Rough consensus from the design meeting today (subject to revision if someone makes new arguments):

• No wholesale duplication of code from the standard library. e.g. no making a copy all of ListBuffer
  • If new code (e.g. extension methods) needs access to a class's private stuff, that class will need to widen access to those internals first (e.g. by changing private to private[collection] or private[scala]. That means we won't be able to release the extension methods until the modified scala-library has been released; that's annoying but not fatal.
• New classes will go in their normal scala.* location (and not under scala.next or what have you)
• Extension methods will go in scala.next or some such package
• When 3.1 comes out, it will include all of the code from the old scala-library-next, and we will bump the major version of scala-library-next for the next round. (If there is a next round, which depends on the timing of the next full ecosystem reboot.)
• As with scala-collection-compat, we must be absolutely strict about bincompat, as this will be a common transitive dependency.

[SethTisue]

At some point the 3.x series, it will become permissible to add new classes and methods without requiring an ecosystem reboot. But those points where things get added will be e.g. 3.2, 3.3, ... which will be fairly far apart. So there will still be a role for scala-library-next, to provide additions in the shorter term.

(Or?)

[sjrd]

If necessary, we can use calls on cast structural types to get access to stuff that we're not supposed to have access to:

val x: Any = ???
def foo(x: Int): Int = ...
type HasFoo = { def foo(x: Int): Int }
x.asInstanceOf[HasFoo].foo(5)

[NthPortal]

If new code (e.g. extension methods) needs access to a class's private stuff, that class will need to widen access to those internals first (e.g. by changing private to private[collection] or private[scala]. That means we won't be able to release the extension methods until the modified scala-library has been released; that's annoying but not fatal.

that doesn't work if it's fully private - the JVM will throw a VerifyError (I believe—might be a different exception) if you run against the older library version

[NthPortal]

regarding scala/bug#12132:

we agreed that it should be possible to add an addOne(K, V):this.type extension method to Map builders, though an efficient implementation was non-obvious (how expensive are type tests for the private Builder types for Maps?).

however, subsequent to the meeting, I tried out the following in a REPL and discovered that it doesn't work due to auto-tupling.

scala> import scala.collection.mutable.Builder
import scala.collection.mutable.Builder

scala> implicit final class MapBuilderOps[K, V, M <: collection.Map[K, V]](val self: Builder[(K, V), M]) {
     |   def addOne(key: K, value: V): self.type = self.addOne(key -> value)
     | }
class MapBuilderOps

scala> Map.newBuilder[Int, Int].addOne(1, 2).result() //print
scala.Predef.Map.newBuilder[scala.Int, scala.Int].addOne(scala.Tuple2.apply[Int, Int](1, 2)).result() // : scala.collection.immutable.Map[Int,Int]

edit: suggested alternative name: addEntry

[sjrd]

That is unfortunate. We'd have to give it a different name, then, I guess.

[dwijnand]

• If new code (e.g. extension methods) needs access to a class's private stuff, that class will need to widen access to those internals first (e.g. by changing private to private[collection] or private[scala]. That means we won't be able to release the extension methods until the modified scala-library has been released; that's annoying but not fatal.

Not fatal, but also makes it (even) harder to evaluate if a mima exclusion in scala/scala is safe or not. We already have parallel collections... which you can take both ways 😄 : the horse has already bolted, we need to check; or: let's not add to the problem..

[lrytz]

* access to a class's `private` stuff

During a meeting later in the day, @dwijnand made the important point that accessing private[x] components of the standard library creates a tight coupling between specific versions of the standard library and scala-library next.

This can prevent improvements in the internals of the standard library, in case they are not binary compatible with existing releases of scala-library-next. Introducing additional restrictions on what we can change in minor releases of the standard library is the opposite of what scala-library-next is supposed to achieve. So we have to be very careful about what internals are actually accessed.

[lrytz]

do we care about split packages? Either for OSGi or (probably more importantly) the Java module system?

Random googling suggests that there are ways for people that use the Java module system to depend on jars with split packages. From https://www.reddit.com/r/java/comments/9yexog/jdk_11_migration_how_to_deal_with_split_packages/

• leave everything on the classpath
• leave just the problematic JARs on the classpath
• command line options like --patch-module (https://nipafx.dev/five-command-line-options-hack-java-module-system)

Using jlink (which seems to be one of the benefits of using JPMS) with non-modular jars looks more tricky, but not impossible (here's a post).

About OSGi: we gave up on including OSGi metadata in our other modules that have split packages and I'm not aware of any complaints (scala/scala-collection-compat#321, scala/scala-parallel-collections#99, scala/scala-collection-contrib#72). I think it's fine to do the same here. If someone ever contributes a reliable way of doing OSGi for these modules that would be accepted of course.

My conclusion: using the same package names makes things easier for the large majority, and I doubt this library would be the only one causing problems for people using JPMS or OSGi. So I'm for using the same packages.

[lrytz]

(I only saw some of the comments above mine just now, sorry for missing that context)

[SethTisue]

some notes from a team discussion a few weeks ago:

• @dwijnand was a bit skeptical (only initially? still?) of the whole thing, along the lines of: why not just have everyone publish their additions separately?
  • My own take is:
    • 1. publishing stuff to Maven Central is a very high hurdle to clear the first time you to do it. It's easier for veterans, but never all that easy.
    • 2. Some prospective additions might be large enough to make sense as independent libraries, but often additions will be smaller, e.g., perhaps just one extension method. These small additions will not just be inconvenient for authors to publish independently, but also inconvenient for users to discover and depend on.
• @retronym wondered if we could handle additions in the scala/scala repo itself via some magic, perhaps an opt-in compiler flag without which the new stuff is ignored.
  • It seems I didn't write down what the team reaction was, but I assume I'd remember if there was team agreement/enthusiasm for such an approach.
• Dale asked about the Dotty standard library additions, what is the relation of that to this?
  • Well for one thing, we want something is published for Scala 2.

[NthPortal]

why not just have everyone publish their additions separately?

• Some of these are tweaks/changes to existing parts of the core library, and having lots of people publish them separately seems to me like a recipe for conflicts.
• I believe we're intending these to be in the scala package, which I don't think we want to endorse others using.
• Additions to this library are Guaranteed™ to be in a subsequent version of the standard library (unless we find a major problem with them).
• Having them in a publicised, "blessed" library increases adoption hopefully (as we've seen with scala-collection-compat).
• Given that others were not in favour of duplicating collections as we discussed, I may publish a personal library that duplicates collections to add things that we can't otherwise (such as mutating iterators), but I'm not sure that's a super promising path in general.

Well for one thing, we want something is published for Scala 2.

Shouldn't that be moot at this point (or soon), due to the Scala 2 compiler being able to handle Tasty?

[SethTisue]

As time passes, we should try to migrate conclusions and guidelines from this ticket to README.md. For now, the readme just links here.

[sinanspd]

What is the correct way of implementing changes for abstract classes within the scope of this repo? I would like to make a PR that adds to Try. Following the structure Seth laid out, I can make a single implicit class where the method matches on the concrete instances but that would need to be separated into the respective functions when merging. Would that impose a major difficulty in the future?