Add opaque types #4028

odersky · 2018-02-21T14:59:18Z

Implements SIP 35

odersky · 2018-02-21T17:41:51Z

Still TODO: Make implicit scope include companion objects of opaque types. [EDIT: done]

odersky · 2018-02-22T16:59:46Z

Overall the implementation adds just 50 lines of new code (partly because the scheme to find companions has gotten simpler). Anyway, that's a lot cheaper than value classes!

EDIT: As of latest count it's more like 170 lines added. Still, not terribly bad.

odersky · 2018-02-22T17:13:56Z

test performance please

dottybot · 2018-02-22T17:14:14Z

performance test scheduled: 1 job(s) in queue, 0 running.

odersky · 2018-02-22T17:39:37Z

I think this one is best reviewed commit by commit.

odersky · 2018-02-22T17:50:45Z

If somemone has more tests it would be great to add them. I only picked those from the SIP and added some small tests that test various invalid code.

dottybot · 2018-02-22T19:02:04Z

Performance test finished successfully:

Visit http://dotty-bench.epfl.ch/4028/ to see the changes.

Benchmarks is based on merging with master (8d07271)

odersky · 2018-02-22T20:17:05Z

test performance please

dottybot · 2018-02-22T20:17:26Z

performance test scheduled: 1 job(s) in queue, 0 running.

dottybot · 2018-02-22T22:04:08Z

Performance test finished successfully:

Visit http://dotty-bench.epfl.ch/4028/ to see the changes.

Benchmarks is based on merging with master (8d07271)

jvican · 2018-02-23T19:48:07Z

I’ll have a look at this on Tuesday, it’s great to see progress on this area.

japgolly · 2018-02-23T21:10:37Z

Just gave this a try:

opaque type Fix[F[_]] = F[Fix[F]]

4 |  opaque type Fix[F[_]] = F[Fix[F]]
  |              ^
  |illegal cyclic reference: alias [F <: [_$1] => Any] => F[opaquetypes.Fix[F]] of type Fix refers back to the type itself

Any plans to support recursive opaque types?

odersky · 2018-02-24T09:48:40Z

Any plans to support recursive opaque types?

No. That would demand a different approach. In that case the only sane way is to demand explicit conversions between the left and right hand sides of the type.

odersky · 2018-02-24T09:48:58Z

test performance please

dottybot · 2018-02-24T09:49:01Z

performance test scheduled: 1 job(s) in queue, 0 running.

dottybot · 2018-02-24T11:32:57Z

Performance test finished successfully:

Visit http://dotty-bench.epfl.ch/4028/ to see the changes.

Benchmarks is based on merging with master (02725c6)

jvican

LGTM. I left some minor comments, but the implementation looks great and simple 👍

Could you elaborate on why recursive opaque types are not supported in this implementation and how that affects existing language semantics? Our goal would be to support this in the scalac implementation.

jvican · 2018-03-05T10:17:17Z

compiler/src/dotty/tools/dotc/core/tasty/TastyFormat.scala

@@ -299,6 +300,7 @@ object TastyFormat {
  final val DEFAULTparameterized = 30
  final val STABLE = 31
  final val MACRO = 32
+  final val OPAQUE = 33


Does this deserve a version bump in the tasty format?

You are right, it should be a minor version bump.

jvican · 2018-03-05T10:21:06Z

compiler/src/dotty/tools/dotc/sbt/ExtractAPI.scala

@@ -212,8 +212,7 @@ private class ExtractAPICollector(implicit val ctx: Context) extends ThunkHolder

    // Synthetic methods that are always present do not affect the API
    // and can therefore be ignored.
-    def alwaysPresent(s: Symbol) =
-      s.isCompanionMethod || (csym.is(ModuleClass) && s.isConstructor)
+    def alwaysPresent(s: Symbol) = csym.is(ModuleClass) && s.isConstructor


Why don't declarations in the API extractor include companion methods? What invariant has opaque types introduced to make the previous code obsolete?

Companion methods don't exist anymore. (They were the synthetic methods that pointed to the companion class or object to the other. We now use an explicit field in ClassDenotation for this).

jvican · 2018-03-05T10:22:34Z

compiler/src/dotty/tools/dotc/transform/FirstTransform.scala

-      mcCompanion.enteredAfter(thisPhase)
-      classCompanion.enteredAfter(thisPhase)
+      val modcls = modul.moduleClass
+      modcls.registerCompanion(forClass)


Should modcls be installed after this phase too?

odersky · 2018-03-07T22:38:59Z

Could you elaborate on why recursive opaque types are not supported in this implementation and how that affects existing language semantics? Our goal would be to support this in the scalac implementation.

Recursion is one of the toughest things to handle, generally. For instance, F-bounds seem useful, but have turned out the single biggest snake-pit for Scala's implementation. I can't count the number of times I wished Scala did not have them! In DOT we do have recursion via this, i.e. rec(x: T). That's necessary to express a lot of what Scala does, including all objects and classes (and F-bounds as well), but it is also very tricky to formalize and prove sound. Adding another kind of recursion over type variables for some (IMO rather marginal) use cases is completely the wrong tradeoff of complexity for convenience. This is a fundamental choice, not an implementation-dependent one.

Blaisorblade · 2018-03-08T11:13:31Z

Could you elaborate on why recursive opaque types are not supported in this implementation and how that affects existing language semantics? Our goal would be to support this in the scalac implementation.

@jvican To elaborate from the theoretical point of view: a recursive opaque type T = F[T] would amount to an equirecursive type as soon as the equality between T and F[T] is visible. And typechecking for equirecursive types are hard to implement for simple types (see Pierce's TaPL). If you add just type constructors you still get soundness, but typechecking lands between "barely decidable", "decidable with a complexity class involving multiple exponentials" and "undecidable" (see POPL paper at http://www-ps.informatik.uni-tuebingen.de/research/functors/, which doesn't even include subtyping or any other Scala feature or inference).
And that's if you make the recursion explicit and make sure to have a relevant node in your types

The only hope is that you want to typecheck programs before the equality is visible, but type-directed algorithms where the equality is visible would still risk needing to deal with recursion somehow.
But from the source program you can easily tell where you convert between T and F[T]: you can then insert a special coercion node that is erased by the backend. But even a sketch would take longer than a comment here.

jvican · 2018-03-09T10:04:50Z

Good arguments, thanks for chiming in the discussion.

odersky · 2018-03-12T10:40:00Z

I integrated the remaining test cases from SIP 35, except for the fixpoint one, which is not supported. The tests brought up some problems with how we treat GADT bounds, which are now addressed. Also, some of the tests did not compile at first. @jvican, it would be good to backport the fixed versions here to the SIP.

odersky · 2018-10-13T16:51:19Z

Opaque types have very tricky interactions with the compiler's sophisticated caching strategy. The danger is that some cached information leaks between companion objects of opaque types and the outside world. 41791f7 tries to improve the situation.

We also have to keep in mind that the same problems arise for all GADT constrained type variables, not just for opaque types. That means changing the opaque type spec, so that instead of GADT equalities we define (say) conversions between an opaque type and its alias gains us nothing, and in fact reduces test coverage.

odersky · 2018-10-14T11:58:02Z

Reverting to wip status until #5254 is resolved.

odersky · 2018-10-14T16:04:35Z

If we can't rely on GADTs to represent the equality between opaque type and its right hand side in some contexts, but not in others, we need a fallback scheme. The most straightforward one would be to make conversions available. I.e the compiler could provide in the companion object of an opaque type a pair of private conversion methods reveal from opaque type to its rhs and inject in the other direction. I.e. as a first approximation, in the Logarithm example, the compiler would
generate in object Logarithm the two methods

private def reveal(x: Logarithm): Double
private def inject(x: Double): Logarithm

This is still too cumbersome, as one could not easily map a List[Double] to a List[Logarithm], nor an Array[Logarithm] to an Array[Double].

A more convenient solution treats reveal and inject calls as compiler magic. Each

requires its argument to be fully typed
obtains its result type by applying the map between opaque type and its rhs to all parts of its argument type (using a TypeMap).

That would be straighforward to use and straightforward to implement. The only downside is that it is still a bit verbose.

johnynek · 2018-10-14T16:48:51Z

You could have an =:=[Opaque, Original] private val in the object. It is a bit of a pain to use but it has substitution so it can deal with the List example you mention.

odersky · 2018-10-14T17:07:29Z

You could have an =:=[Opaque, Original] private val in the object. It is a bit of a pain to use but it has substitution so it can deal with the List example you mention.

Can you explain that in more detail? I am not sure how to use =:= for the purpose you have in mind.

johnynek · 2018-10-14T17:27:58Z

so, in your example, you have:

opaque type ImmutableArray[+A] = Array[A]
object ImmutableArray {
  def sum[A: Numeric](a: ImmutableArray[A]): A = ...
}

opaque type Log = Double

object Log {
  private implicit val reveal: Log =:= Double = ... // compiler generated, but could also be a cast of `refl`

  // a product in the original space, is sum in log space, so this is a key operation
  def sum(l: ImmutableArray[Log]): Log = { 
    val p = ImmutableArray.sum(reveal.substituteCo(l))
    reveal.flip(p)
  }
}

johnynek · 2018-10-14T17:29:36Z

https://github.com/scala/scala/blob/e57997a9cd1d7e73792cfa1a49eff6e4d7848932/src/library/scala/Predef.scala#L668

johnynek · 2018-10-14T17:30:24Z

PS: immutable array is one I'm really excited about. We can have an ImmutableArray[Log] but get the same unboxed performance as Array[Double].

odersky · 2018-10-14T18:31:16Z

@johnynek Ah I see what you mean now. That works to some degree but I also think it can be a pain to use.

I agree that immutable array is one the most exciting parts of this effort.

LPTK · 2018-10-15T10:09:52Z

@odersky probably worth mentioning that there is a less painful alternative to the substitution methods of =:= and <:<.

@Blaisorblade, @sstucki, and I discussed it in a previous issue: #3844 (comment)

It already works in Dotty (last I tried), and solves the problem neatly. The basic idea is to have:

abstract class <:< [-A,+B] { type Ev >: A <: B }

Then, if you have a List[S] and want a List[T] when you have an S <:< T evidence, you can just use a type ascription:

def foo[S,T](xs: List[S])(implicit ev: S <:< T): List[T] = (xs: List[ev.Ev])

The same scheme can probably be used for opaque type. What'd be interesting is to see whether Dotty could leverage these evidence values automatically when they are in scope, to avoid the need for explicit ascriptions to some extent (though the general problem seems undecidable, as was discussed in a Scala’17 paper).

PS: the more general/potent definition for <:< is actually the following, as discussed on the issue mentioned above:

abstract class <:<[A, B] {
  type ConstrainedB >: A <: A & B // = B with constraints
  type ConstrainedA >: A | B <: B // = A with constraints
}

LPTK · 2018-10-15T10:58:40Z

PPS: the above is for covariant and contravariant substitution. For substitution in invariant places, one just needs type Ev >: A | B <: A & B as a member of =:=.

Here is a complete example (Scastie) of an opaque type implemented using my strategy:

abstract class OpaqueAPI {
  type S  // abstract type
  protected implicit val ev: S =:= String  // not visible from outside
  
  def in (xs: Set[String]): Set[S] = xs:Set[ev.Ev]
  def out(xs: Set[S]): Set[String] = xs:Set[ev.Ev]
  
  val arr0: Array[String] = Array("a")
  val arr1: Array[S] = Main.foo(arr0)
}
object Main {
  def foo[A,B](arr: Array[A])(implicit sub: A =:= B): Array[B] = arr:Array[sub.Ev]
  
  val Opaque: OpaqueAPI = new OpaqueAPI {
    type S = String  // not visible from outside
    val ev: S =:= String = implicitly
  }
  import Opaque._
  
  def main(args: Array[String]): Unit = {
  	
    val s0 = Set("a")
    val s1: Set[S] = in(s0)
    println(s1)
    val s2: Set[String] = out(s1)
    println(s2)
    
    println(s"Done.")
  }
}

...where we have:

abstract class <:< [-A,+B] { type Ev >: A <: B }
object <:< {
  implicit def ev[A<:B,B]: A <:< B = new { type Ev = A }
}
abstract class =:= [A,B] { type Ev >: A | B <: A & B }
object =:= {
  implicit def ev[A>:B<:B,B]: A =:= B = new { type Ev = A }
  implicit def flip[A,B](implicit ev: A =:= B): B =:= A = new { type Ev = ev.Ev }
}

Open opaque types as gadts in opaque companion modules.

In the new implementation, a companion object of an opaque type opaque type T = A only knows that T <: A and that A <: T. By itself that does not propagate some informations from `A` to `T`. For instance the members of A are now not the members of T.

odersky · 2018-10-15T15:10:00Z

I revived the original approach since #5254 is now resolved. The main difference is that, in light of the closing comment of #5254, we no longer assume all the properties of a full type alias in the opaque type companion. So some tests have to be changed, as is shown in 207c380.

odersky · 2018-10-19T22:02:39Z

We found a much better way to get opaque types without restrictions and without too many contortions in the compiler: #5300

odersky force-pushed the add-opaque branch from 3482c27 to d058973 Compare February 21, 2018 17:40

odersky changed the title ~~[WIP] Add opaque types~~ Add opaque types Feb 22, 2018

odersky requested a review from biboudis February 22, 2018 17:39

odersky assigned biboudis Feb 22, 2018

Blaisorblade added the area:typer label Feb 26, 2018

odersky mentioned this pull request Feb 27, 2018

Add augment clauses #4043

Closed

odersky added the itype:language enhancement label Mar 1, 2018

jvican reviewed Mar 5, 2018

View reviewed changes

odersky force-pushed the add-opaque branch 2 times, most recently from 157c5fa to 5de563b Compare March 12, 2018 10:37

odersky assigned abgruszecki and unassigned odersky Oct 12, 2018

odersky requested a review from abgruszecki October 12, 2018 15:20

odersky mentioned this pull request Oct 14, 2018

GADTs might need a different implementation scheme #5254

Closed

odersky added stat:wip and removed stat:needs review labels Oct 14, 2018

odersky added 4 commits October 15, 2018 13:40

Establish opaque companions

53d1fca

Open opaque types as gadts in opaque companion modules.

Fix typo

c6c5bd1

Docs and tests

716b3a0

odersky force-pushed the add-opaque branch from 41791f7 to 9dcd977 Compare October 15, 2018 13:25

Make elim opaque transform declare that it changes base types

a6edd60

odersky force-pushed the add-opaque branch from 9dcd977 to a6edd60 Compare October 15, 2018 16:35

odersky added stat:needs review and removed stat:wip labels Oct 15, 2018

odersky mentioned this pull request Oct 19, 2018

Opaque types - selftype encoding #5300

Merged

odersky closed this Oct 19, 2018

Add opaque types #4028

Add opaque types #4028

Conversation

odersky commented Feb 21, 2018

odersky commented Feb 21, 2018 • edited

odersky commented Feb 22, 2018 • edited

odersky commented Feb 22, 2018

dottybot commented Feb 22, 2018

odersky commented Feb 22, 2018

odersky commented Feb 22, 2018

dottybot commented Feb 22, 2018

odersky commented Feb 22, 2018

dottybot commented Feb 22, 2018

dottybot commented Feb 22, 2018

jvican commented Feb 23, 2018

japgolly commented Feb 23, 2018

odersky commented Feb 24, 2018

odersky commented Feb 24, 2018

dottybot commented Feb 24, 2018

dottybot commented Feb 24, 2018

jvican left a comment

Choose a reason for hiding this comment

jvican Mar 5, 2018

Choose a reason for hiding this comment

odersky Mar 9, 2018

Choose a reason for hiding this comment

jvican Mar 5, 2018

Choose a reason for hiding this comment

odersky Mar 9, 2018

Choose a reason for hiding this comment

jvican Mar 5, 2018

Choose a reason for hiding this comment

odersky commented Mar 7, 2018

Blaisorblade commented Mar 8, 2018

jvican commented Mar 9, 2018

odersky commented Mar 12, 2018 • edited

odersky commented Oct 13, 2018

odersky commented Oct 14, 2018

odersky commented Oct 14, 2018 • edited

johnynek commented Oct 14, 2018

odersky commented Oct 14, 2018

johnynek commented Oct 14, 2018 • edited

johnynek commented Oct 14, 2018

johnynek commented Oct 14, 2018

odersky commented Oct 14, 2018

LPTK commented Oct 15, 2018

LPTK commented Oct 15, 2018

odersky commented Oct 15, 2018

odersky commented Oct 19, 2018

odersky commented Feb 21, 2018 •

edited

odersky commented Feb 22, 2018 •

edited

odersky commented Mar 12, 2018 •

edited

odersky commented Oct 14, 2018 •

edited

johnynek commented Oct 14, 2018 •

edited