[Proposal] Typeclass Traits

[odersky]

This is a proposal to support extension methods and typeclasses in a more direct and convenient way. It is based on #4114.

Status

This is a first draft proposal (consider it Pre-SIP stage). None of the features that go beyond #4114 are implemented yet.

Rationale

There are two dominant styles of structuring Scala programs: Standard (object-oriented) class hierarchies or typeclass hierarchies encoded with implicit parameters. Standard class hierarchies lead to simpler code and allow to dispatch on the runtime type, which enables some optimizations that are hard to emulate with implicits. Typeclass hierarchies are more flexible: instances can be given independently of the implementing types and the implemented interfaces, and instances can be made conditional on other typeclass instances.

Unfortunately, typeclass-oriented programming has a high upfront cost. It starts with the definitions of "typeclasses" themselves. Say you want to implement a typeclass for a map method (often called a Functor). The usage you want to support is:

xs.map(f)

However, you can't define map as a unary method like this, not if it is part of a typeclass. Instead you need to define a binary version of map, along the following lines:

trait Functor[F[_]] {
  def map[A, B](f: A => B, xs: F[A]): F[B] = ...
}

You need another implicit class, to get back map as an infix operator. Something like this:

implicit class InfixMap[F[_]: Functor, A](private val x: F[A]) extends AnyVal {
  def map(f: A => B): F[B] = implicitly[F].map(f, x)
}

It does the job, but at a cost of lots of boilerplate! The required boilerplate is very technical, advanced, and, I believe, frightening for a newcomer. The complexity of typeclasses does not end with their definition, either. It continues to the use sites, which typically need one extra type parameter per typeclass argument. Scala is a language set out to eliminate boilerplate and promote the simplest possible style of expression. But it seems in this area it has utterly failed to do so.

https://github.com/mpilquist/simulacrum is macro library that removes a lot of the boilerplate. But it relies on "macro paradise" which has never been officially supported and has lost its maintainer https://contributors.scala-lang.org/t/stepping-down-as-the-maintainer-of-scalamacros-paradise/1703. The kind of annotation macros required by simulacrum will almost certainly only be supported in Dotty as code generation tools, requiring a separate build step.

The aim of this proposal is to

• make typeclasses easy and pleasant to use.
• remove the rift between standard class hierarchies and typeclass hierarchies. Both should be written in the same way and it should be possible to mix both styles and move fluidly between them.
• in conjunction with opaque types, replace the lower-level constructs of value classes and implicit classes.

Details

The proposal is written up as a list of reference pages:

1. Extension Methods
2. Instance Declarations
3. Typeclass Traits
4. Common Declarations
5. Context Bounds
6. Parameterised Typeclass Traits
7. Factored Instance Declarations
8. Local Coherence
9. Syntax and Type Checking
10. Translation Scheme

The proposal was heavily inspired by Rust's traits and implementations. The main differences to Rust are

• extensions are named instead of being anonymous
• a tight integration with standard classes and traits
• a transparent mapping to the underlying system of implicits
• implied self parameters
• direct support for higher kinds
• local instead of global coherence

[smarter]

I like the idea but I'm not a fan of the surface syntax: common def is weird because it only works in traits and because it has unintuitive scoping rules (the enclosing class type and term parameters are not available), I think we could generalize the concept of a companion object instead:

trait Text {
  //...
}
abstract object Text {
  def fromString(str: String): Instance
  def fromStrings(strs: String*): Instance =
    ("" :: strs).map(fromString).reduceLeft(_.concat)
}

enum ConcText extends Text {
  //...
}
object ConcText { // Implicitly extends the abstract object Text
  def fromString(str: String): ConcText = ConcText.Str(str)
}

[odersky]

@smarter That's an interesting idea! There's one complication though, that we sometimes need both an abstract and a concrete companion object for a trait. We'd have to establish the intuition that abstract objects are a third thing, different from objects. Another variant would be

trait Text {
  //...
}
common {
  def fromString(str: String): Instance
  def fromStrings(strs: String*): Instance =
    ("" :: strs).map(fromString).reduceLeft(_.concat)
}

That avoids the scoping confusion but does not cause a name clash.

[odersky]

[UPDATE: Instance is not exposed anymore in the latest proposal]

Thinking over it, I believe we should not expose Instance as in the proposal. Exposing both Instance and This is confusing, and Instance is really only needed for the typeclass encoding. Without Instance, the common part of the proposal is no longer related to kinding.

The downside of dropping Instance is that using common alone is less useful, so it's harder to find good examples where it makes sense. But we have already established that the main use of common would be in connection with typeclasses.

[smarter]

Exposing both Instance and This is confusing

What's the difference between Instance and This?

[smarter]

we sometimes need both an abstract and a concrete companion object for a trait.

Yes, this does make it more difficult to find a good design, but is this really a hard requirement?

[odersky]

What's the difference between Instance and This?

In the typeclass encoding Instance is the type that is produced, This the type for which the instance is produced.

[odersky]

Yes, this does make it more difficult to find a good design, but is this really a hard requirement?

For the typeclass encoding, yes. Also, if the implementation is parameterized, the common definitions won't be in an object anymore, but in a class. This is necessary to properly support This. So in general we will have to let common definitions access type parameters of the enclosing class. I have to update the proposal with this. Example:

class C[T] {
   common elem: T
}
C[T].elem

[OlivierBlanvillain]

It should be xs: Array[T] here

[OlivierBlanvillain]

First argument list missing? (the (x: T))

[OlivierBlanvillain]

Here is my attempt to desugar the first part of typeclass-encoding3.scala to the regular encoding of type classes: https://gist.github.com/OlivierBlanvillain/d314ddbcb640e2ce5604d860b5073008.

Note that there are no implicit conversions involved, only implicit parameters.

[odersky]

I think the problem with this is that you need to define every normal trait twice - once as written and then again with a type parameter. I believe that's way to burdensome - we should not have to pay any overhead at all for normal traits.

[Blaisorblade]

I think the problem with this is that you need to define every normal trait twice - once as written and then again with a type parameter.

Shouldn't that be just for typeclass traits? Not every trait makes sense as a type class, at least in Haskell, for a few different reasons (can elaborate if needed).

[odersky]

[UPDATE: This is now solved. See the section on Factored Instance Declarations.]

Right now I am stumped to come up with a scheme to re-use extensions. Example:

    extension IntSemiGroup for Int : SemiGroup {
      def add(that: Int) = this + that
    }

    extension IntMonoid for Int : Monoid {
      common def unit = 0
    }

We'd like to be able to write extensions like the above and transparently re-use the SemiGroup implementation for Int for the Monoid implementation. I tried to accommodate that using inheritance, but it got very messy and looks inefficient without further deep optimizations.

One technique we should look at it to generate for every trait method a static method in the trait with this as an explicit parameter. The instance method in the trait is then a forwarder to the static method. That's hopefully low overhead (essentially it means we go back to some degree to the 2.11 way of generating code). As far as I recall that was actually slightly faster than the 2.12 way to use default methods, but it generated sometimes significantly more code. On the other hand, using inheritance we could do the forwarders only for the methods implemented by a trait directly, but not for inherited trait methods, which should cut down on code size. Example:

 trait A {
    def a() = "a"
 }
 trait B extends A {
   def b() = "b"
 }

becomes

trait A {
  def a() = a$(this)
  static def a$($this: A) = "a"
}
trait B extends A {
  def b() = b$(this)
  static def b$($this: B) = "b"
}

If we have that, we might be able to replace inheritance by delegation, which would give us more flexibility.

[odersky]

Shouldn't that be just for typeclass traits? Not every trait makes sense as a type class, at least in Haskell, for a few different reasons (can elaborate if needed).

Please elaborate, yes! I mean, discussing at lunch, we decided that only no-init traits qualify as type classes (i.e. no vars or vals allowed). Are there other restrictions?

[Blaisorblade]

Please elaborate, yes! I mean, discussing at lunch, we decided that only no-init traits qualify as type classes (i.e. no vars or vals allowed). Are there other restrictions?

I expected the translation should be opt-in. Here's a conceptual reason about what seems to work in practice in Haskell, but that's a bit fuzzy. Then there's a few examples where I'm not sure what a translation should do.

0. Even in Haskell, most of the time, creating a typeclass is a pretty advanced operation. You're discouraged most of the time from inventing a new typeclass (because of the magic action-at-a-distance, so to say), unless there's reason to expect your typeclass to be well-behaved — Haskellers typically require it to have (algebraic) laws.

1. I'd have expected that the translation should be opt-in — Olivier turns HasLength into HasLength_TC, but that does not seem useful for a trait that wasn't a typeclass trait already.

2. Take the following interface for sets:

trait Set[A] {
  def append(that: Set[A]): Set[A]
  // Beware: this isn't
  //def append(that: This): This
}

which allows appending different implementations of sets, do you expect to translate that to a typeclass implementation? How? With Olivier's translation I can maybe use:

trait Set_TC[T, A] {
  def append($this: T, that: Set[A]): Set[A]
}

but in Olivier's translation, adding a Set_TC[T, A] instance doesn't allow turning T into Set[A].

• The translation could produce something like:

trait Set_TC[T, A] {
  def append[U]($this: T, that: U)(implicit $ev: Set_TC[U, A]): Set[A]
}

• Alternatively, allow converting T into Set[A] given a Set_TC[T, A]. In Olivier's example,

implicit class HasLengthOps[T](self: T)(implicit ev: HasLength_TC[T]) {
  def length: Int = ev.length(self)
}

by

implicit class HasLengthOps[T](self: T)(implicit ev: HasLength_TC[T]) extends HasLength {
  def length: Int = ev.length(self)
}

3. Take the interface of java.util.Iterator (or the abstract methods of the Scala Iterator). Those methods expect to mutate the instance, yet that's not visible. EDIT: do you want to still allow using a type class for that? I've never seen such a thing and it's not obvious it'd work. Wouldn't the typeclass implementation need access to the private parts of an iterator implementation? Usual typeclass encodings don't allow doing that.

[EECOLOR]

For general discussions (i.e. what the right approach to typeclasses should be) I believe scala contributors is the best forum.

I can't find the correct thread, could you provide a link?

[odersky]

@EECOLOR: Somebody has to start a thread. There is none yet.

[odersky]

@oxbowlakes I agree about @deriving. We are looking how we can accommodate it in the language, since the macro systems we will support in the future look too weak for this. @OlivierBlanvillain is working on this.

Otherwise I am not sure in what sense users of a typeclass would be affected. There might be small differences, i.e. it's currently Monoid.impl[T].unit instead of Monoid[T].unit but that seems to be largely cosmetic, and can be changed. As stated this is a first proposal and I am looking forward to evolve it and simplify it where possible.

[rkrzewski]

@odersky, you have asked for ApplicativeError instance declarations in #4153 (comment) but I cant's see anyone providing them above.

ApplicativeError for cats is defined here: https://github.com/typelevel/cats/blob/1.1.x/core/src/main/scala/cats/ApplicativeError.scala#L12
MonadError here: https://github.com/typelevel/cats/blob/1.1.x/core/src/main/scala/cats/MonadError.scala#L8
and MonadError instance for Either is defined here: https://github.com/typelevel/cats/blob/1.1.x/core/src/main/scala/cats/instances/either.scala#L31

For IO data types, the libraries the typically provide MonadError[IO, Throwable] instance, for example https://github.com/typelevel/cats/blob/1.1.x/core/src/main/scala/cats/MonadError.scala#L8
(ConcurrentEffect[F[_]] extends Sync[F[_]] which in turn extends MonadError[F[_], Throwable])

When EitherT monad transformer is used together with an IO/Task type, there are two Applicative/MonadError instances available: EitherT[IO, MyErrrorType, ?] has both MonadError[?, Throwable] derived from MonadError[IO, Throwable] and MonadError[?, MyErrorType] derived from MonadError for embedded Either

As for the functional dependencies - I skimmed Mark P. Jones "Type classes with functional dependencies" paper, and while this matter is way over my head, my understanding is that CanBuildFrom is an example of such typeclass because it's From and To parameters are collection types of element type Elem. No such relationship exists between arguments of ApplicativeError.

[Sciss]

I wonder if it could (aliased to) apply, so it becomes Monoid[T].unit which looks nicer and less bulky. Some of the alternative proposals had that short form.

[ivanopagano]

is this a typo?

(implicit $ev: Monoid.common[T]) shouldn't that be Monoid.Common[T]? a type and not a term

[Jasper-M]

@rkrzewski I think the instances that @odersky was looking for are the ones for EitherT.

However to me it still seems that the second type parameter of MonadError is completely dependent on the first type parameter. In other words you could perfectly encode this with the usual dependent types pattern:

trait MonadError[F[_]] {
  type ErrorType
}
object MonadError extends LowPriority {
  type Aux[F[_], E] = MonadError[F] { type ErrorType = E }  

  implicit def either[A] = new MonadError[Either[A, ?]] { type ErrorType = A }
  implicit def io = new MonadError[IO] { type ErrorType = Throwable }
  implicit def eithert1[F[_], L](implicit F0: Monad[F]) = new MonadError[EitherT[F, L, ?]] { type ErrorType = L }
}
trait LowPriority {
  implicit def eithert2[F[_], E, L](implicit FE0: MonadError.Aux[F, E]) = new MonadError[EitherT[F, L, ?]] { type ErrorType = E }
}

// here I care about the error type
def foo[F[_]](f: F[String])(implicit m: MonadError.Aux[F, Throwable]) = ???

// and here I don't
def bar[F[_]: MonadError](f: F[String]) = ???

[odersky]

@rkrzewski @Jasper-M Thanks for the answers. Would be good to get to the bottom of this!

[SystemFw]

What about things with a bidirectional functional dependency?
So given any one of two parameters, the other is always determined?
This is still not unconstrained, so it doesn't suffer from the problems of general multi parameter type classes, but it doesn't lend itself well to the suggested strategy, which is to make the uniquely determined type a type member.
Bidirectionality means that according to use, one parameter would have to be This and the other would have to be a type member, which is I think a barrier.

[rkrzewski]

@SystemFw is FunctionK[F[_], G[_]] aka NaturalTransformation an example of such typeclass? Or you have something different on your mind?

[SystemFw]

@rkrzewski No, FunctionK is not a typeclass at all :)

I've actually just realised that there's a better example though. Typeclasses with three type arguments where there's a dependency between the first two and the third.

For example typeclasses for type safe dimensionality: multiplying a vector with a matrix, a scalar with a matrix and so on

class Mult a b c | a b -> c where
  (*) :: a -> b -> c
 
instance Mult Matrix Matrix Matrix where
  {- ... -}
 
instance Mult Matrix Vector Vector where

[odersky]

Mult seems to be indeed not encodable. But I am not sure it would fit anyway in a typeclass proposal. The core of any proposal is really what context bounds mean. For Mult there does not seem to be a natural way to write a context bound, precisely because none of the parameter types can be seen as "the implementation type". So instead of context bounds, you'd still use a constraint such as

def genMult[A, B, C](x: A, y: B)(implicit ev: Mult[A, B, C]): C = ...

Once you do that, it seems actually quite clear to write

implicit object MultMMM extends Mult[Matrix, Matrix, Matrix] { ... }
implicit object MultMMV extends Mult[Matrix, Matrix, Matrix] { ... }

Or where do you see room here for further improvement? I would say: maybe we omit the names of the objects MultMMM and MultMMV. That was indeed my first design for extend clauses, but there was very well-founded opposition against that, mostly having to do with binary compatibility.

[SystemFw]

@oderky Honestly at this point I'm still in the exploration phase, trying to understand benefits, drawbacks and limitations of the current proposal :)

But I am not sure it would fit anyway in a typeclass proposal. The core of any proposal is really what context bounds mean

Because of the above, the examples are really just meant to highlight possible limitations, not to argue that such limitations are a showstopper one way or another. The point about context bounds is interesting though, I'll keep that in mind.

More generally, I personally prefer a typeclass solution that doesn't rely on This (for various reasons), but I'll keep those thoughts for a different issue, so that we can keep this one on topic :)

[odersky]

Because of the above, the examples are really just meant to highlight possible limitations, not to argue that such limitations are a showstopper one way or another.

Understood :-) That's what we both are trying to do here.

[etorreborre]

I'm having a hard time keeping up with all the comments but here is one comment (by @odersky) that I would like to discuss:

trait HasArea {
def area: Double
}
Should that be a normal trait or a type class?

I want to be able to migrate fluidly from a normal trait to a typeclass without having to refactor code.

I think that using a trait and using a typeclass are 2 fundamentally different design decisions so having to refactor is a normal consequence because they are not the same thing:

1. with a trait you tie functionality very close to the type / class implementing the trait. So close that you need to take care about name clashes when mixing in different traits for example.

2. with a typeclass you explicitly extract that functionality into another object (the instance) and make it apart from the typeclass target type (or "types" this is even more obvious when we involve several types). Typeclasses make it easy to add behaviour "after the fact" which you cannot do with traits at the expense of some indirection.

In my mind this justifies having a different constructs in the language like typeclass/instance and find how to make syntax/coherence/implementation practical. This is not an "OO approach" but I think for a good reason: OO is not a fit for everything. And maybe making Scala an OO/FP language is about providing first class OO constructs as well as first class FP ones?

I am sorry if this rambling comment is more a vague intuition rather than a formal proposal but maybe this can shed some light on the discussion?

[odersky]

@etorreborre

I think that using a trait and using a typeclass are 2 fundamentally different design decisions so having to refactor is a normal consequence because they are not the same thing:

That sentiment is shared my many people. But I don't think it needs to be true. Two counter-examples:

• With implicit classes in Scala2 / extensions in Dotty we already have a way to disentangle a class from a trait and make it implement the trait optionally. If you assume Add named extensions #4114 everything is already provided, except that the traits that can be implemented are limited because they cannot talk about the implementation type (i.e. This is missing).

• Rust has demonstrated that it is perfectly possible to have traits serve as both typeclasses and trait objects. Rust's model seems to be universally liked. We should not discount it out of hand!

[aloiscochard]

Rust has demonstrated that it is perfectly possible to have traits serve as both typeclasses and trait objects. Rust's model seems to be universally liked. We should not discount it out of hand!

@odersky well, that's right. But did you use it heavily in practice? I mean in non trivial projects.

I'm practicing Rust heavily for more than a year now (wrote several dozen of thousands of line of code), and even if I totally agree it must be a good source of inspiration for your current work. I'm really not enthusiast about the trait objects features.

I did use it quite heavily at first, literally thinking "oh maybe that's finally a nice fusion of OOP/FP here!", so I totally understand you enthusiasm here... but I got very disappointed in the long run.

I had to spend hours of refactoring to "fix" code that was using trait object and could not compile anymore after adding new function in my trait because I had to box it everywhere (while avoiding boxing... is my main reason to use Rust). OTOH, by making my code parametric I do avoid the issue.

As JVM memory model is arguably pretty different, do you think it would be possible to avoid such limitations? I'm happy to provide concrete example if there is interest, the reason of the boxing is tied to how methods are dispatched and I'm pretty sure that would have an impact on Scala design.

Also, an other important point, please consider that even if Rust allows trait to be treated as object, there is no sub-typing on Struct... In a Scala context, couldn't that introduce unsoundness/ambiguity issues?

I hope this put some emphasis on the motivation to look at those both concepts separately as @etorreborre suggest, in my opinion if you want to fuse them, great advantages must be demonstrated... My goal here was to demonstrate that trait object is not one of them.

---

Finally, on a more personal note, just a word to say how happy I am to see the Scala team looking to support natively type-classes after all this years of intense (and interesting!) debates. It is also pretty awesome for me to hear the discussion about coherence and this concept being discussed in a Scala context.

Even though my interested have shift after all this time (and as you say Martin, if people want a pure encoding on the JVM they look into Eta-lang, and not hijack your design), I'm very glad to see that there is a true and serious attempt at providing a native encoding for typeclasses in Scala.

[etorreborre]

@odersky I think your approach is definitely worth exploring and it is interesting to contrast what you are proposing with Rust's self. When you write:

except that the traits that can be implemented are limited because they cannot talk about the implementation type (i.e. This is missing).

Indeed This is missing and I wonder what is the overall price of adding it (plus other consequences like the change with context bounds?), in terms of implementation, cognitive load, potential bugs, compared to introducing typeclass/instance. But talk is cheap and people convinced with a typeclass/instance approach should prepare a SIP so we can do side by side comparisons. I can offer to review such a SIP but I unfortunately don't have to time to write it down :-( (nor the skills I'm afraid). My hunch is that the second design will end up being easier to implement, teach and use.

[odersky]

@aloiscochard Thanks for your input! [I was busy with other things for a while so saw it only recently]. Very interesting to compare with your Rust experience. I would think that boxing on the JVM is purely a performance issue, so you should not see any compilation errors when switching between context bounds and value parameters. That would be a design goal, for sure. As to performance, it will depends a lot what a compiler and the JVM runtime would do in each case.

[Ichoran]

@aloiscochard

But did you use it heavily in practice? I mean in non trivial projects.

I'm practicing Rust heavily for more than a year now (wrote several dozen of thousands of line of code), and even if I totally agree it must be a good source of inspiration for your current work. I'm really not enthusiast about the trait objects features.

I use Rust--not as heavily as you have; I'm still mostly writing Scala or C++--but I have had a better experience with trait objects than you. I tended to use them lightly to begin with (preferring static dispatch for performance), and I was wary of the restrictions needed to make a trait object-safe, so I haven't run into refactoring gotchas much. Being able to have an object at all is way better, in some cases, than not, so it's still a win from my perspective.

But, anyway, all the annoyances in the Rust are due to being explicit about memory and avoiding allocations. The JVM doesn't have to worry about them (aside from the standard friction around boxing primitives). A Rust-on-the-JVM would have trait objects just work, always. (Everything on the JVM can be thought of as Sized because it's all just pointers to heap allocated stuff.)

So I think as inspiration for Scala, the upsides for Rust are relevant and the drawbacks not.

[Blaisorblade]

This rule seems a bit fragile for ADT. What if you want you have an implementation which is a trait? Motivation: consider

trait Filter[A] extends TypeClass { def filter(f: A => Boolean): This[A] }
trait Option[A] extends Filter[A]
class Some[A] extends Option[A] { def filter(...): ??? }

the return type would seem to have to be Some[A], not Option[A]. Clearly, what we actually want is to have type This = Option. And yes, I realize this can be achieved by making Option an abstract class (but this isn't very orthogonal — what if I need to mix other classes in Option's children) or by using extensions.

Incidentally, this puzzler would be avoided if typeclass traits were marked otherwise.

EDIT: credit for the question to @tpolecat on Gitter.

[Blaisorblade]

Back to the example above with Option[A] extending typeclass Filter[A], if we want Option to implement typeclasses of different kinds, like Filter (or Monad) and Semigroup, we can't use the OOP typeclass-implementation style. Rust avoids this problem because it doesn't have higher kinds.

[SystemFw]

If you are looking for a concrete example, from cats:

trait SemigroupK[F[_]] {
   def combineK[A](fa: F[A], fb: F[A]): F[A]
}

vs

trait Semigroup[A] {
   def combine(a: A, b: A): A
}

Option has instances for both, SemigroupK encodes prioritised choice (orElse basically), and Semigroup combines the A: Semigroup in Option[A]

[aloiscochard]

@odersky thanks, it's very good to hear this should be purely a performance issue on the JVM (and that this will be enforced during design)!
All the best for the implementation, I'm looking forward to play around with what you will come up :)

@Ichoran yeah indeed, interesting to hear about your experience. I wish I was wary of this before writing so much code, but I guess it's a fair price to pay for me not RTFM ;) ... anyway, now I'm still using them in few places, not sure how long they will survive and I just try to avoid them on new design.

[LoranceChen]

Support it! Different semantic should use different keywords, implicit has many semantic for now, hope we can split implicit as different keywords for different semantic.
For now, I think implicit has those semantics:

1. typeclass,as this, for pure functional from Haskell
2. extends class method, such as RichInt
3. as a context, such as thread pool for Future.

I think they should have themselves keywords.It's also good for tools.

Besides, for unify concept of Scala, they could be implements by one origin thought which depend on how does DOT defines(I not really know how does Scala compiler works).