Port “The Architecture of Scala Collections” to the new architecture

[julienrf]

Do not merge unless 2.13 is released!

I’ve started adapting the documentation page The architecture of Scala collections to the new design.

I’m not sure we should continue with the same examples. They were very interesting to show how to use CanBuildFrom to drive the return type of transformation operations, but this is not a feature which is supported by the new design (so… there is nothing really fancy to show here). On the other hand I don’t think the examples here demonstrate all the features of the new framework (like how to get the return type of map & cie work as expected without having to add overloads, which is only possible for polymorphic collection types).

[julienrf]

I was also wondering if we should provide helpers to implement such operations in a strict way. For the view based version it is as easy as writing:

def map(f: Base => Base): RNA = fromSpecificIterable(View.Map(this, f))

We should probably have an equivalent strict version:

object Builder {

  def map[CC[_], A, B](source: IterableOnce[A], f: A => B, builder: Builder[B, CC[B]]): CC[B] = {
    for (a <- source) {
      b += f(base)
    }
    builder.result()
  }
}

So that users could just write:

def map(f: Base => Base): RNA = Builder.map(this, f, newSpecificBuilder())

And we could use it also in StrictOptimizedIterableOps.

The only part that I don’t like is that the builder is created in one place (user code) and used in a different place (framework code). This can be error prone because we really want the builder to be freshly created.

[julienrf]

The PR is ready to review!

In addition to explaining the new architecture, I’ve made the following changes to the examples:

• I start with a new example of a collection with a capped size, which illustrates the basic case (how to get right the return type of transformation operations such as filter and map, which traits to inherit from in the case of a strict collection),
• I’ve kept RNA and PrefixMap, which show how to use overloading to apply the “same-result-type” principle.

[julienrf]

I’d like to get this PR reviewed so that we have time to improve things if needed. @SethTisue, would you have time to do that soon?

[SethTisue]

I think the document would be more helpful and easier to follow if you led with a summary of all of the challenges that the design seeks to solve, before proceeding to discuss each challenge in depth.

The challenges include:

• Some collections have a single element type (e.g. List[A]), others have two (e.g. Map[K, V]).
• Some transformations return the same concrete collection type, e.g. filter never changes the type.
• Some transformations return the same type constructor with a different element type (e.g. map from List[A] to List[B]).
• Operations on some collections require additional implicit parameters, e.g. map on a SortedSet takes an implicit Ordering
• Some collections are strict (e.g. List), others are non-strict (e.g. LazyList)

I made that list by looking through the opening pages of the document, but as I read it over, it strikes me as curiously incomplete. Missing items that come to mind:

• Some "collections" are actually existing types the JVM provides, e.g. Array and String, that must be transparently integrated somehow.
• Some collections are mutable, others are immutable.
• Some operations on collections return a different collection type constructors depending on the element type. For example String.map[A, B] returns a String if B is Char but must otherwise return some other kind of collection.
• Views. One of the main goals of the redesign is to correctly support views.
• Builders. The builder design is different than the old collections, right? What challenge prompted the change?

Having given a list of challenges, I think it would also be helpful to the reader to briefly say what the solution to each one is, in a sentence or two, before going into the detailed, multi-paragraph discussions of each one.

(Meta-question: is this the kind of feedback you were looking for...? I assume it's too soon for a detailed proofread/copyedit.)

[SethTisue]

this table syntax does not render on GitHub

[SethTisue]

is there material from these blog posts that should be integrated here? (I'm suggesting that there is.)

• http://scala-lang.org/blog/2017/02/28/collections-rework.html
• http://scala-lang.org/blog/2017/05/30/tribulations-canbuildfrom.html
• http://scala-lang.org/blog/2017/07/02/collections-0.2.0.html
• http://scala-lang.org/blog/2017/09/12/collections-status.html
• http://scala-lang.org/blog/2017/11/28/view-based-collections.html

[julienrf]

Thanks @SethTisue for the feedback! (yes that’s exactly what I was looking for)

[lrytz]

Very nice work, thanks! Great suggestions by Seth as well, fully agree.

I wonder if it's a good approach to hold this updated document back until 2.13 is released and then replace the old version. I think it would be valuable to have the 2.13 version available before the release (during milestones / RCs), and probably also valuable to keep the 2.12 one around for a while.

There should be instructions how to cross-build custom collections against 2.12 and 2.13 - could be in this or a separate document.

Does the "Collections" guide need a rework as well (http://docs.scala-lang.org/overviews/collections/introduction.html)? Is someone planning to work on this?

[lrytz]

explain why operatinos are not implemented in Iterable directly

[julienrf]

Maybe that wasn’t very clear but that’s what I’ve tried to explain before that section: to implement the operations you need to abstract over their return type, hence the need for the template traits.

[lrytz]

i just wonder if it's worth saying that having the ops trait (with 3 type params) allows having a clean Iterable[A] type with a single type param

[lrytz]

this makes readers curious without giving a link or hint where to look further :-)

[julienrf]

Yes. We could add a link to StringOps for instance.

[lrytz]

show fromIterable as well?

[lrytz]

give an example (groupMap)?

[lrytz]

maybe provide a link to https://docs.scala-lang.org/overviews/collections/seqs.html ?

[julienrf]

I wonder if it's a good approach to hold this updated document back until 2.13 is released and then replace the old version. I think it would be valuable to have the 2.13 version available before the release (during milestones / RCs), and probably also valuable to keep the 2.12 one around for a while.

You’re right, we shouldn’t replace that document yet but have both documents coexist.

There should be instructions how to cross-build custom collections against 2.12 and 2.13 - could be in this or a separate document.

I’d rather put that information in a separate document. I find this one already quite long.

Does the "Collections" guide need a rework as well (http://docs.scala-lang.org/overviews/collections/introduction.html)? Is someone planning to work on this?

Yes! We have that meta-ticket to keep track on what needs to be done: https://github.com/scala/collection-strawman/issues/311. And the part that you are referring to corresponds to #995. Nobody is working on it.

[julienrf]

Superseded by #1078