Unified extension methods

[odersky]

Dotty currently uses three syntactically different schemes for extension methods

1. Single methods with prefix parameters such as def (x: T).f(y: U) or def (x: T) op (y: U).
   These methods can be abstract, can override each other, etc, which is crucial for infix ops
   in type classes.
2. Given instances defining (only) extension methods with syntax extension ops { ... }
3. Collective extension methods with syntax extension on (x: T) { ... }

This PR proposes another scheme that unifies (1) and (3) and all but eliminates the need for (2).

The simplification is achieved by redefining many fundamentals of extension methods. In a sense,
the price we pay for user-facing simplifications is a more complex implementation scheme. But that's
probably a price worth paying.

The first commit is docs only. I updated the doc pages for extension methods, type classes, and
the grammar.

[odersky]

I know it's late in the game for language changes. I started thinking about this while revising the material for the ProgFun MOOC to use Scala 3. I wanted to introduce extension methods early, but found that the overall syntax was too complex to admit a simple introduction. Each of the three parts are OK, but together they are confusing. Why three different ways to do things? There's no good explanation since there is no good reason, after all.

[odersky]

Three particular complications for the implementation are:

• An extension method m is mapped to a normal method with name extension_m instead of just m. This is probably better anyway, since it avoids accidental calling of methods in the wrong mode.

• Extension methods are now also searched directly in the implicit scope of a type, not just in given instances in that scope.

• There's a general scheme to allow expansion of simple identifiers to extension methods.

[smarter]

At first glance, I love it! This addresses the two reservations I had about extension methods:

• Putting infix defs directly inside a companion feels so natural, but before this PR this meant they weren't
  available as extensions outside the object, which I feared would be a big source of confusion.
• The relationship between extension methods and typeclasses is clearer in the code. I find this very interesting pedagogically: one can start by teaching extension methods (which are fairly intuitive), and once someone has seen:

object List:
  extension [T](xs: List[List[T]]) def flatten: List[T] = ...
object Future:
  extension [T](xs: Future[Future[T]]) def flatten: Future[T] = ...

It's natural to want to abstract over them, and like with regular methods the answer is to use a trait with an abstract method inside:

trait Flattenable[F[_]]:
  extension [T](xs: F[F[T]]) def flatten: F[T]

At this point, the given mechanism needs to be introduced, but it's not such a big conceptual leap anymore: in the code, one only has to wrap the existing definitions of extension methods without changing them in any way:

object List:
  given Flattenable[List]:
    extension [T](xs: List[List[T]]) def flatten: List[T] = ...
...

(previously, this would have required switching from an extension block to a given block, each using slightly different syntax).

[soronpo]

In the old design there was a limitation that the extended and extendee couldn't have different type arguments.
Are the following now possible (the documentation doesn't show these kind of examples):

 extension [T](foo: Foo[T])
   def +[R](rightFoo : Foo[R]) = ???

 extension [T](foo: Foo[T]):
   def +[R](rightFoo : Foo[R]) = ???
   def *[R](rightBar : Bar[R]) = ???

[nicolasstucki]

I have been looking into syntax highlighting for def (x: T) f (y: U): V = ... and it turns out that the regexp needed needs to be overly complicated because we need to identify the def and the f as part of the extension method, but we have a parameter in between that could be arbitrarily complex.

The solution proposed in this PR would make syntax highlighting trivial for extension.

[nicolasstucki]

We used to have names for the collective extension as in extension Ops on (x: X) .... We seem to be losing some expressivity with this change. These were useful for:

extension Ops1 on [T](x: List[T]):
  def h: T = x.head

extension Ops2 on (x: List[Int]):
  def sm: Int = x.sum

We also use those names to disambiguate extensions in the reflection interface to https://github.com/lampepfl/dotty/blob/master/library/src/scala/tasty/Reflection.scala#L541. Though for that case, we could work around it if ClassDefOps could be defined inside ClassDef.

[nicolasstucki]

We currently have somthing like

trait R:
  type X
  type Y
  extension Xops on (x: X):
    def i: Int = 1
  extension Yops on (y: Y):
    def i: Int = 2

Which would become

trait R:
  type X
  type Y
  extension (x: X) def i: Int = 1 // def extension_i(x: X): Int = 1
  extension (y: Y) def i: Int = 2 // def extension_i(y: Y): Int = 2

But after erasure, both extension_i have the same signature.

This could be OK if we allow extension methods to be found without given imports for

trait R:
  type X
  type Y
  object X:
    extension (x: X) def i: Int = 1 // def extension_i(x: X): Int = 1
  object Y:
    extension  (y: Y) def i: Int = 2 // def extension_i(y: Y): Int = 2

val r: R = ...
val x: r.X = ...
x.i

[odersky]

trait R:
  type X
  type Y
  extension Xops on (x: X):
    def i: Int = 1
  extension Yops on (y: Y):
    def i: Int = 2

can be expressed like this:

trait R:
  type X
  type Y
  given Xops as AnyRef:
    extension (x: X) def i: Int = 1
  given Yops as AnyRef:
    extension (y: Y) def i: Int = 2

We could allow to drop the initial type AnyRef to make this use case nicer. Then it would be:

trait R:
  type X
  type Y
  given Xops as:
    extension (x: X) def i: Int = 1
  given Yops as:
    extension (y: Y) def i: Int = 2

[odersky]

@soronpo The restriction that there may be only one type parameter section is still present (found in the section "Collective Extension Methods"). it probably should be moved to "Generic Extensions".

[nicolasstucki]

Do we also support collective extension with { ... } as in extension (x: Int) { def ... }?

[nicolasstucki]

Should we?

[odersky]

Yes, we do support that.

[nicolasstucki]

We should mention that the following are equivalent

extension (x: Int) {
  def f(y: Int) = x + y
}

extension (x: Int):
  def f(y: Int) = x + y

This would help users understand the scope of the extension.

[sjrd]

This seems to make sense, overall. It seems to be a nice simplification from the user point of view.

@nicolasstucki For the erasure issues, can they be circumvented by adding a (using DummyImplicit) clause to one of the extension groups?

[nicolasstucki]

Dummy implicits could work, but it does not scale well and there are the performance implications.

[soronpo]

Dummy implicits could work, but it does not scale well and there are the performance implications.

Maybe @alpha will be usable for this purpose in the future: #7942

[japgolly]

Looks awesome!

Where would override go wrt extensions? Like this?

given Monoid[String]:
  extension (x: String):
    override def combine (y: String): String = x.concat(y)

[odersky]

@japgolly Yes. Modifiers precede the def.

[japgolly]

Ah sorry, I see I should've been a bit clearer. So to confirm these are ok:

given Monoid[String]:
  extension (x: String):
    override def combine (y: String): String = x.concat(y)

// and 

given Monoid[String]:
  extension (x: String) override def combine (y: String): String = x.concat(y)

and these are not, right?

given Monoid[String]:
  override extension (x: String):
    def combine (y: String): String = x.concat(y)

// and 

given Monoid[String]:
  override extension (x: String) def combine (y: String): String = x.concat(y)

[japgolly]

Also would it not be better to translate extension methods to extension$abc instead of extension_abc to highlight that it's a generated name that shouldn't overlap with anything in userspace?

[odersky]

@japgolly

Also would it not be better to translate extension methods to extension$abc instead of extension_abc to highlight that it's a generated name that shouldn't overlap with anything in userspace?

Users should be able to refer to extension names, for instance in order to disambiguate calls, or debug type inference. That's why we wanted a user-accessible name for them.

So to confirm these are ok:

They are all illegal since the givens would already produce a double definition error.

[soronpo]

Also would it not be better to translate extension methods to extension$abc instead of extension_abc to highlight that it's a generated name that shouldn't overlap with anything in userspace?

I would opt for $extension_abc

[neko-kai]

The loss of the very short syntax for

def [A, B](fa: F[A]).map(f: A => B): F[B]

is regrettable...
Also there's an effect on typeclasses that use extension methods - the previous scheme without extension_ prefix afforded an ability to write typeclass methods as extensions-only and automatically get both the infix and prefix forms:

trait Functor[F[_]] {
  def [A, B](fa: F[A]).map(f: A => B): F[B]
}

def plus[F[_]](f: F[Int])(using F: Functor[F]) = 
  // prefix form
  F.map(f)(_ + 1)
  // infix form
  f.map(_ + 1)

The proposed change eliminates this convenience and forces one to write copies of extension methods to enable prefix form without extension_:

trait Functor[F[_]] {
  final def map(fa: F[A])(f: A => B): F[B] = fa.map(f)
  extension[A, B](fa: F[A]) def map(f: A => B): F[B]
}

def plus[F[_]](f: F[Int])(using F: Functor[F]) = 
  // prefix form doesn't look very good...
  F.extension_map(f)(_ + 1)
  // prefix form of boilerplate:
  F.map(f)(_ + 1)
  // infix form
  f.map(_ + 1)

[odersky]

@neko-kai It's true that the previous short extension method syntax was more concise, and was quite elegant in some cases. But in the use cases I have seen so far, collective extension methods turned out to be just as important as individual ones, and having two different syntaxes is problematic for learning. While being more verbose, the new syntax has also some advantages:

  extension [A, B](fa: F[A])
     def map(f: A => B): F[B]

Here, we keep the invariant that method names always follow a def, so it's easy to grep for them, say. Furthermore, the method name appears towards the left margin, so it's easy to find visually.

About extension_map vs map. Again, it's more verbose, but easier to disambiguate. I would argue that

  F.extension_map(f)(_ + 1)

is actually clearer than

  F.map(f)(_ + 1)

Adding the extension_ allows us to also expand simple identifiers as extension methods, and there are quite a few use cases for that. With the previous syntax, we could do something like that only for collective extension methods, but not for individual ones. Here's an example:

  extension (s: String)
    def position(ch: Char, n: Int): Int =
      if n < s.length && s(n) != ch then position(ch, n + 1)
      else n

Here, the recursive call position(ch, n + 1) expands to s.position(ch, n + 1). This is important, since otherwise
we'd get lots of errors in particular for collective extension methods. We can do that since there is no way to
confuse with a non-extension call, which would be written extension_position(s)(ch, n + 1). Without the extension_
we'd not know in general whether the call we see was an extension call with implied prefix, or a non-extension call.

[soronpo]

In classes and objects : creates a new scope, but not for extensions

If so, what does this refer to in extension methods?

[odersky]

If so, what does this refer to in extension methods?

It's the outer this of the scope where extension is defined.

[odersky]

Maybe it's best to allow an optional : after the extension parameter. There's a
tendency to write it in analogy to given, object, etc. Or is there a need
to be picky about this?

[sideeffffect]

My 2 cents:

1. It would be good, if there were just one way of writing this, since otherwise people may spend unnecessary time discussing this in code review or rewriting it back and forth,
2. It would be good, if the : syntax were consistent across all of given, object, catch, extension, ...
3. This PR is about the new mechanism and syntax for extension methods. Does the syntax of : deserve a discussion in a dedicated ticket?

(Btw, I really like this new way of declaring extension methods, thank you.)

[neko-kai]

That does not work for local extension methods since there is no prefix we could write for them.

@odersky
True, but that's no different from the status quo for all locals, e.g. there's no way to refer to outer localfun inside the inner localfun in this example:

def outerlocal = {
  
  def localfun: Int = {
    def localfun(i: Int): String = {
      if (i == 0) "x" else (1 + localfun).toString
    }
    localfun(util.Random.nextInt()).toInt
  }
  
  localfun
}