Compilation with -Ypartial-unification flag fails

[aSapien]

tl;dr

Compilation fails with boopickle-shapeless and -Ypartial-unification. Test case repo here

---

Full story

Recently I needed to enable -Ypartial-unification in a codebase I'm working on. To my surprise, at a specific part of the codebase - compilation failed, which forced me to revert the change.

I invested an effort to investigate and resolve the issue, but to no avail. However, I did manage to isolate the issue for the purpose of seeking help online.

I created a repo with a small test case which fails to compile with -Ypartial-unification. Link here
This is the code for the test case:

package example

import java.nio.ByteBuffer

object TestCase {
  import boopickle.shapeless.Default._

  case class Parametrized[T]()
  case class Container(include: Map[Parametrized[_], Seq[String]])

  def encode(c: Container): ByteBuffer = {
    Pickle.intoBytes(c)
  }
}

Compilation fails with the following error

Error:(12, 21) could not find implicit value for parameter p: boopickle.Pickler[example.TestCase.Container]
    Pickle.intoBytes(c)

I decided to add compiler debug flags, one of the compiler flags (namely -Ymacro-debug-lite) revealed that a missing implicit was expected to be materialized by a macro, but was not. Finding the reason currently appears to be beyond my skills and expertise.

Comparing the logs of the failing and succeeding compilations output (with -Ymacro-debug-lite flag) I can see that the failing version is missing the following macro expansion:

Warning:scalac: performing macro expansion shapeless.this.Generic.materialize[example.TestCase.Parametrized[_], R] at source-/Users/dimaryskin/projects/scala-debug/src/main/scala/example/TestCase.scala,line-12,offset=265
Warning:scalac: {
  final class anon$macro$5 extends _root_.shapeless.Generic[example.TestCase.Parametrized[_]] {
    def <init>() = {
      super.<init>();
      ()
    };
    type Repr = shapeless.this.HNil;
    def to(p: example.TestCase.Parametrized[_]): Repr = p match {
  case TestCase.this.Parametrized() => _root_.shapeless.HNil
}.asInstanceOf[Repr];
    def from(p: Repr): example.TestCase.Parametrized[_] = p match {
      case _root_.shapeless.HNil => TestCase.this.Parametrized()
    }
  };
  (new anon$macro$5(): _root_.shapeless.Generic.Aux[example.TestCase.Parametrized[_], shapeless.this.HNil])
}

It's important to note that making minor changes to the code results in a successful compilation. Some examples are:

- case class Container(include: Map[Parametrized[_], Seq[String]])
+ case class Container(include: Map[Seq[String], Parametrized[_]])
# OR
+ case class Container(include: Map[Parametrized[_], String])
# OR
+ case class Container(include: Parametrized[_], exclude: Seq[String])

Any of the above changes somehow facilitates a successful compilation.

I appreciate any help that can help me understand the problem and cause since it blocks me from introducing -Ypartial-unification because I fear other issues may arise, which I'll be unable to understand and resolve.

Thanks,
Dima

[ochrons]

Ping @cornerman

[cornerman]

Interesting report! Spontaneously I have no idea why this happens and I am not sure whether we can do anything about this. The shapeless module pretty much only provides Pickler instances for shapeless's HList, Generic.Aux and CoProduct. From there, it was supposed to just do its magic :)

It's important to note that making minor changes to the code results in a successful compilation. Some examples are:

I am not an expert here either, but the examples might really show what partial-unification does. To quote from http://eed3si9n.com/herding-cats/partial-unification.html:

... it’s important to understand that the compiler will now assume that the type constructors can be partially applied from left to right. In short, this will reward right-biased datatypes like Either, but you could end up with wrong answer if the datatype is left-biased.

[aSapien]

@cornerman thank you for the attention. I looked into the implementation of Shapeless' Generic.Aux which seems to be used to derive the missing implicit in the case above, but I'm still far from being able to grasp what's going on there. Having also tried to understand the StackOverflow answer on the topic, I'm still having difficulty to connect the dots.

Perhaps, you can help me convert this test case into one that is using vanilla Shapeless, stripping down BooPickle? I would then be able to post an issue at Shapeless repo, the creator of which, is also the author of -Ypartial-unification.

[cornerman]

@aSapien Maybe let us just start with what we want to achieve in the ShapelessPicklers. In simple words, in order to pickle a type T, boopickle needs to have a function for encoding (T => ByteBuffer) and a function for decoding (ByteBuffer => T). Now in the normal boopickle project, we have our own macro for deriving picklers for your types. With shapeless you do not need your own macros but can abstract over a data type generically.

Let us say, we have a case class Foo(i: Int, s: String). For pickling type Foo, we need to be able to pickle all its members - in this case Int and String. So, we want to abstract from the concrete case class and just regard its data members. In shapeless, this abstraction exists in the form of Generic.Aux[Foo, Int :: String :: HNil]. The part Int :: String :: HNil is an HList which is an heterogeneous list, so it can contain different types opposed to a List[Int] which can only contain Int:

case class Foo(i: Int, s: String)
val aux = Generic[Foo] // shapeless.Generic[Foo]{type Repr = Int :: String :: shapeless.HNil}

aux.to(Foo(1, "hallo")) == 1 :: "hallo" :: HNil
aux.from(2 :: "nope" :: HNil) == Foo(2, "nope")

So with this Generic we can now create picklers for case classes. We can pickle a case class if we can pickle all its data members. So, if we can pickle a type T, then we can also pickle an HList ... :: T :: ... :: HNil.

Now, we further want to be able to pickle trait hierarchies, which is why we need to support Coproduct from shapeless. We had HList which was a conjunction of data types. But with inheritance, we want to model a disjunction of types (a Vehicle can either be Motorcycle or a Car). That is exactly what a Coproduct or discriminated union is type VehicleUnion = Motorcycle :+: Car :+: CNil:

sealed trait Vehicle
case class Car() extends Vehicle
case class Motorcycle() extends Vehicle
val aux = Generic[Vehicle] // shapeless.Generic[Vehicle]{type Repr = Car :+: Motorcycle :+: shapeless.CNil}

aux.from(Inl(Car())) == Car()
aux.to(Car()) == Inl(Car())

I hope this helps a bit in understanding. I am testing right now in a project of mine and now see similar errors when using boopickle-shapeless and partial unification in combination. I sadly do not have a good idea how to isolate the issue right now :/ Maybe it has something to do with implicit resolution.

[aSapien]

@cornerman that's a great explanation. Thanks!
I'll try to play around with Shapeless, get more familiar and then attempt to reproduce this.

Hope you have some progress as well.

[milessabin]

Enabling -Ypartial-unification can have the effect of making previously inapplicable instances applicable. If these newly applicable instances now compete with previously applicable instances then you could end up with new ambiguities knocking out both the newly and the previously applicable instances. If these are buried deep in an implicit resolution the nested ambiguity can be hidden and the top level error report won't be particularly informative.

The trick here is to juggle with implicit priorities until it works. If you don't have too many instances then you should be able to figure this out by trial and error.

[aSapien]

The trick here is to juggle with implicit priorities until it works.

@milessabin is there a way to effectively inspect the implicit scope? Define priorities?

I think the problem I'm facing here is that the expected implicit should be generated by a Shapeless macro, but isn't because -Ypartial-unification affects how the macro "sees" the type.

[aSapien]

Inspecting the logs output for macros materialization, with and without -Ypartial-unification this is where they diverge:

The succeeding version (green) is materializing the following macro, while the failing (red) is attempting (and failing) to materialize a different one:

+Warning:scalac: performing macro expansion shapeless.this.Generic.materialize[example.TestCase.Parametrized[_], R] at source-.../TestCase.scala,line-12,offset=265
-Warning:scalac: performing macro expansion shapeless.this.Generic.materialize[Seq[String], R] at source-.../TestCase.scala,line-12,offset=265

Notice the same line and offset: line-12,offset=265