New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SI-7046 partial fix to knownDirectSubclasses for reflection users and macro authors #5284

Merged
merged 1 commit into from Dec 1, 2016

Conversation

Projects
None yet
@milessabin
Contributor

milessabin commented Jul 15, 2016

This is a 95% fix of SI-7046 for comment and feedback.

This appears to do the right thing in the most typical scenarios in which knownDirectSubclasses would be used. The missing 5% is that subclasses defined in local scopes might not be seen by knownDirectSubclasses (see Local and Riddle in the test below). In mitigation, though, it is almost certain that a local subclass would represent an error in any scenario where knownDirectSubclasses might be used.

Errors for such situations are reported by recording (via a symbol attachment) that knownDirectSubclasses has been called and reporting an error if any additional children are added subsequently.

Despite these limitations and caveats, I believe that this represents a huge improvement over the status quo, and would eliminate 100% of the failures that I've seen in practice with people using shapeless for type class derivation.

Aside from that, all (par)tests pass.

@@ -1175,6 +1172,9 @@ trait ContextErrors {
def MissingParameterOrValTypeError(vparam: Tree) =
issueNormalTypeError(vparam, "missing parameter type")
def ParentSealedInheritanceError(parent: Tree, psym: Symbol) =
NormalTypeError(parent, "illegal inheritance from sealed " + psym )

This comment has been minimized.

@xeno-by

xeno-by Jul 15, 2016

Member

Why do we need to reorder this error message?

@xeno-by

xeno-by Jul 15, 2016

Member

Why do we need to reorder this error message?

This comment has been minimized.

@milessabin

milessabin Jul 15, 2016

Contributor

It's moved from ContextErrors to NamerContextErrors so that it's accessible in Namers.

@milessabin

milessabin Jul 15, 2016

Contributor

It's moved from ContextErrors to NamerContextErrors so that it's accessible in Namers.

@@ -113,6 +113,13 @@ trait Symbols extends api.Symbols { self: SymbolTable =>
def knownDirectSubclasses = {
// See `getFlag` to learn more about the `isThreadsafe` call in the body of this method.
if (!isCompilerUniverse && !isThreadsafe(purpose = AllOps)) initialize
enclosingPackage.info.decls.foreach { sym =>
if(sourceFile == sym.sourceFile) {

This comment has been minimized.

@xeno-by

xeno-by Jul 15, 2016

Member

What does this check amount to?

@xeno-by

xeno-by Jul 15, 2016

Member

What does this check amount to?

This comment has been minimized.

@milessabin

milessabin Jul 15, 2016

Contributor

The sealedness constraint means that we only need to search within the same compilation unit.

@milessabin

milessabin Jul 15, 2016

Contributor

The sealedness constraint means that we only need to search within the same compilation unit.

@xeno-by

This comment has been minimized.

Show comment
Hide comment
@xeno-by

xeno-by Jul 15, 2016

Member

I think that it may still be possible to address an important case of subclasses nested in a top-level object. Do you think you'll need an involved context for that?

Member

xeno-by commented Jul 15, 2016

I think that it may still be possible to address an important case of subclasses nested in a top-level object. Do you think you'll need an involved context for that?

@milessabin

This comment has been minimized.

Show comment
Hide comment
@milessabin

milessabin Jul 15, 2016

Contributor

@xeno-by the scenario you described is handled ... see the test.

Contributor

milessabin commented Jul 15, 2016

@xeno-by the scenario you described is handled ... see the test.

@milessabin

This comment has been minimized.

Show comment
Hide comment
@milessabin

milessabin Jul 15, 2016

Contributor

Specifically Wibble and Wobble nested in object Foo.

Contributor

milessabin commented Jul 15, 2016

Specifically Wibble and Wobble nested in object Foo.

@dwijnand

This comment has been minimized.

Show comment
Hide comment
@dwijnand

dwijnand Jul 15, 2016

Member

Maybe I'm just fixating too much on the name, but it's knownDirectSubclasses, not allDirectSubclasses - ie. known at the level in which you ask them, anything declared in some local scope isn't known.

So I would expect that if you called Test.subs in those local scopes it would see those subclass too. Perhaps it would be good to add those cases to the test.

Member

dwijnand commented Jul 15, 2016

Maybe I'm just fixating too much on the name, but it's knownDirectSubclasses, not allDirectSubclasses - ie. known at the level in which you ask them, anything declared in some local scope isn't known.

So I would expect that if you called Test.subs in those local scopes it would see those subclass too. Perhaps it would be good to add those cases to the test.

@milessabin

This comment has been minimized.

Show comment
Hide comment
@milessabin

milessabin Jul 15, 2016

Contributor

@dwijnand yes, that's true, but I would prefer to see an error reported in that case ... it would be reintroducing a compilation order dependency which this PR is intended to eliminate.

Contributor

milessabin commented Jul 15, 2016

@dwijnand yes, that's true, but I would prefer to see an error reported in that case ... it would be reintroducing a compilation order dependency which this PR is intended to eliminate.

@Blaisorblade

This comment has been minimized.

Show comment
Hide comment
@Blaisorblade

Blaisorblade Jul 15, 2016

Contributor

IIUC, the biggest potential danger is the added call to the new method forceDirectSuperclasses in knowDirectSubclasses, because it forces more types than before and that's a well-known source of bugs. The danger is a forcing too much too early; to me, earlier discussions on SI-7046 assumed (implicitly) this couldn't be done. And while tests pass, that might of course be insufficient.

I'd guess the patch should spell out (in code) an informal argument why that's safe, to allow inspecting it for corner cases and writing appropriate testcases.

In particular, this should be fine if classes are typechecked in inheritance order, but is that really the case in Scalac? Can we loop Scalac with this patch by calling knowDirectSubclasses on an inheritance cycle?

Contributor

Blaisorblade commented Jul 15, 2016

IIUC, the biggest potential danger is the added call to the new method forceDirectSuperclasses in knowDirectSubclasses, because it forces more types than before and that's a well-known source of bugs. The danger is a forcing too much too early; to me, earlier discussions on SI-7046 assumed (implicitly) this couldn't be done. And while tests pass, that might of course be insufficient.

I'd guess the patch should spell out (in code) an informal argument why that's safe, to allow inspecting it for corner cases and writing appropriate testcases.

In particular, this should be fine if classes are typechecked in inheritance order, but is that really the case in Scalac? Can we loop Scalac with this patch by calling knowDirectSubclasses on an inheritance cycle?

@milessabin

This comment has been minimized.

Show comment
Hide comment
@milessabin

milessabin Jul 15, 2016

Contributor

@Blaisorblade the use of maybeInitialize means that cycles are harmless. The Unrelated class explicitly tests the case where there is a cycle (via the reference to the macro-defined val which is triggering the force) ... this example was suggested by @retronym.

Contributor

milessabin commented Jul 15, 2016

@Blaisorblade the use of maybeInitialize means that cycles are harmless. The Unrelated class explicitly tests the case where there is a cycle (via the reference to the macro-defined val which is triggering the force) ... this example was suggested by @retronym.

@xeno-by

This comment has been minimized.

Show comment
Hide comment
@xeno-by

xeno-by Jul 15, 2016

Member

@milessabin Very interesting! Why?

Member

xeno-by commented Jul 15, 2016

@milessabin Very interesting! Why?

@milessabin

This comment has been minimized.

Show comment
Hide comment
@milessabin

milessabin Jul 15, 2016

Contributor

@xeno-by we only care about initialising things sufficiently to ensure that children add themselves to their sealed parents. These types won't participate in a cycle (if they did it would be an error which we would expect to be reported), so if we initialise as far as we can we'll get the parent-child relationships that we need. If there are no objectionable cycles we'll get the correct (modulo local classes) result; if there are objectionable cycles they will be an error and be reported as such.

Contributor

milessabin commented Jul 15, 2016

@xeno-by we only care about initialising things sufficiently to ensure that children add themselves to their sealed parents. These types won't participate in a cycle (if they did it would be an error which we would expect to be reported), so if we initialise as far as we can we'll get the parent-child relationships that we need. If there are no objectionable cycles we'll get the correct (modulo local classes) result; if there are objectionable cycles they will be an error and be reported as such.

@notxcain

This comment has been minimized.

Show comment
Hide comment
@notxcain

notxcain Jul 17, 2016

Any chance of getting this in 2.12.0?

Any chance of getting this in 2.12.0?

Show outdated Hide outdated src/compiler/scala/tools/nsc/typechecker/Namers.scala
@@ -1675,6 +1696,9 @@ trait Namers extends MethodSynthesis {
abstract class TypeCompleter extends LazyType {
val tree: Tree
override def forceDirectSuperclasses: Unit = {
tree.foreach(t => Option(t.symbol).map(_.maybeInitialize))

This comment has been minimized.

@retronym

retronym Jul 20, 2016

Member

I believe this should be restricted to DefTrees:

tree.foreach {
  case dt: DefTree if dt.symbol != null => dt.symbol.maybeInitialize
  case _ =>
}
@retronym

retronym Jul 20, 2016

Member

I believe this should be restricted to DefTrees:

tree.foreach {
  case dt: DefTree if dt.symbol != null => dt.symbol.maybeInitialize
  case _ =>
}

This comment has been minimized.

@retronym

retronym Jul 20, 2016

Member

I think that maybeInitialize is a weak link in this chain. It is defined as:

    def maybeInitialize = {
      try   { initialize ; true }
      catch { case _: CyclicReference => debuglog("Hit cycle in maybeInitialize of $this") ; false }
    }

The problem is that this doesn't undo any side effects performed by the type completer before it ran into the cycle. The type completer will be run again, and we might observe problems if those side effects aren't idempotent.

maybeInitialize is currently only used in -Ybreak-cycles, and was described as "sketchy" when it landed: 432f936

@retronym

retronym Jul 20, 2016

Member

I think that maybeInitialize is a weak link in this chain. It is defined as:

    def maybeInitialize = {
      try   { initialize ; true }
      catch { case _: CyclicReference => debuglog("Hit cycle in maybeInitialize of $this") ; false }
    }

The problem is that this doesn't undo any side effects performed by the type completer before it ran into the cycle. The type completer will be run again, and we might observe problems if those side effects aren't idempotent.

maybeInitialize is currently only used in -Ybreak-cycles, and was described as "sketchy" when it landed: 432f936

This comment has been minimized.

@milessabin

milessabin Jul 20, 2016

Contributor

Agreed on the DefTree restriction.

On the sketchiness of maybeInitialize ... yes, side-effects won't be undone. However any side effects that are triggered before a cycle is observed should be stable, ie. we're merely moving them to an earlier point rather than changing the outcome.

Also, this path will only be followed if knownDirectSubclasses is called, and the result can't be any sketchier than the status quo.

Could we run a community build with this change and see how it plays out?

@milessabin

milessabin Jul 20, 2016

Contributor

Agreed on the DefTree restriction.

On the sketchiness of maybeInitialize ... yes, side-effects won't be undone. However any side effects that are triggered before a cycle is observed should be stable, ie. we're merely moving them to an earlier point rather than changing the outcome.

Also, this path will only be followed if knownDirectSubclasses is called, and the result can't be any sketchier than the status quo.

Could we run a community build with this change and see how it plays out?

This comment has been minimized.

@milessabin

milessabin Jul 20, 2016

Contributor

@retronym any thoughts on the idea of using a flag to record that knownDirectSubclasses has been observed so that we can error if any additional subclasses add themselves later?

@milessabin

milessabin Jul 20, 2016

Contributor

@retronym any thoughts on the idea of using a flag to record that knownDirectSubclasses has been observed so that we can error if any additional subclasses add themselves later?

@milessabin

This comment has been minimized.

Show comment
Hide comment
@milessabin

milessabin Jul 20, 2016

Contributor

As an additional data point wrt maybeInitialize I can confirm that shapeless builds and tests perfectly with this patch applied. It has several uses of knownDirectSubclasses and these are exercised heavily in its tests and examples.

Contributor

milessabin commented Jul 20, 2016

As an additional data point wrt maybeInitialize I can confirm that shapeless builds and tests perfectly with this patch applied. It has several uses of knownDirectSubclasses and these are exercised heavily in its tests and examples.

@retronym

This comment has been minimized.

Show comment
Hide comment
@retronym

retronym Jul 21, 2016

Member

Here's a very contrived example to that shows a leakage in maybeInitialize.

package p1

import scala.reflect.macros.blackbox._
import language.experimental._

object Macro {
  def impl(c: Context): c.Tree = {
    import c.universe._
    println(rootMirror.staticClass("p1.Base").knownDirectSubclasses)
    q"()"
  }
  def p1_Base_knownDirectSubclasses: Unit = macro impl
}
package p1

sealed trait Base

object Test {
  val x = {
    Macro.p1_Base_knownDirectSubclasses
    ""
  } 
}

case class B(val b: Test.x.type)

This issues a cyclic error under this PR. I think this is because the CyclicError is being caught in typedInternal:

      try runTyper() catch {
        case ex: TypeError =>
          tree.clearType()
          // The only problematic case are (recoverable) cyclic reference errors which can pop up almost anywhere.
          typingStack.printTyping(tree, "caught %s: while typing %s".format(ex, tree)) //DEBUG
          reportTypeError(context, tree.pos, ex)
          setError(tree)
        case ex: Exception =>
          // @M causes cyclic reference error
          devWarning(s"exception when typing $tree, pt=$ptPlugins")
          if (context != null && context.unit.exists && tree != null)
            logError("AT: " + tree.pos, ex)
          throw ex
      }

rather than bubbling up to the catch-and-discard in maybeInitialize.

This isn't quite what I had in mind above, but it does qualify as a side-effect (issuing the type error).

The example itself needs a bit more minimization before its worthy of detailed discussion. But it shows basis for my unease about maybeInitialize is its current state.

I notice that you've put the forcing in directKnownSubclasses in the public reflection API which is good in the sense that this won't be triggered when the typer calls children in CheckabilityChecker; we know that the problems will only affect files that use shapeless-like macros.

I'm going to keep thinking about this one.

Member

retronym commented Jul 21, 2016

Here's a very contrived example to that shows a leakage in maybeInitialize.

package p1

import scala.reflect.macros.blackbox._
import language.experimental._

object Macro {
  def impl(c: Context): c.Tree = {
    import c.universe._
    println(rootMirror.staticClass("p1.Base").knownDirectSubclasses)
    q"()"
  }
  def p1_Base_knownDirectSubclasses: Unit = macro impl
}
package p1

sealed trait Base

object Test {
  val x = {
    Macro.p1_Base_knownDirectSubclasses
    ""
  } 
}

case class B(val b: Test.x.type)

This issues a cyclic error under this PR. I think this is because the CyclicError is being caught in typedInternal:

      try runTyper() catch {
        case ex: TypeError =>
          tree.clearType()
          // The only problematic case are (recoverable) cyclic reference errors which can pop up almost anywhere.
          typingStack.printTyping(tree, "caught %s: while typing %s".format(ex, tree)) //DEBUG
          reportTypeError(context, tree.pos, ex)
          setError(tree)
        case ex: Exception =>
          // @M causes cyclic reference error
          devWarning(s"exception when typing $tree, pt=$ptPlugins")
          if (context != null && context.unit.exists && tree != null)
            logError("AT: " + tree.pos, ex)
          throw ex
      }

rather than bubbling up to the catch-and-discard in maybeInitialize.

This isn't quite what I had in mind above, but it does qualify as a side-effect (issuing the type error).

The example itself needs a bit more minimization before its worthy of detailed discussion. But it shows basis for my unease about maybeInitialize is its current state.

I notice that you've put the forcing in directKnownSubclasses in the public reflection API which is good in the sense that this won't be triggered when the typer calls children in CheckabilityChecker; we know that the problems will only affect files that use shapeless-like macros.

I'm going to keep thinking about this one.

@milessabin

This comment has been minimized.

Show comment
Hide comment
@milessabin

milessabin Jul 21, 2016

Contributor

@retronym would be keen to get your take on the CyclicReference propagation fix in the most recent commit.

Contributor

milessabin commented Jul 21, 2016

@retronym would be keen to get your take on the CyclicReference propagation fix in the most recent commit.

@milessabin

This comment has been minimized.

Show comment
Hide comment
@milessabin

milessabin Jul 21, 2016

Contributor

Failures appear to be transient ...

Contributor

milessabin commented Jul 21, 2016

Failures appear to be transient ...

@milessabin

This comment has been minimized.

Show comment
Hide comment
@milessabin

milessabin Jul 21, 2016

Contributor

/rebuild

Contributor

milessabin commented Jul 21, 2016

/rebuild

Show outdated Hide outdated src/compiler/scala/tools/nsc/typechecker/Contexts.scala
@@ -265,6 +265,9 @@ trait Contexts { self: Analyzer =>
def inSecondTry_=(value: Boolean) = this(SecondTry) = value
def inReturnExpr = this(ReturnExpr)
def inTypeConstructorAllowed = this(TypeConstructorAllowed)
def propagateCyclicReferences_=(value: Boolean)
= this(PropagateCyclicReferences) = value
def propagateCyclicReferences = this(PropagateCyclicReferences)

This comment has been minimized.

@retronym

retronym Jul 22, 2016

Member

I think this is more appropriate as a single boolean for the entire Global, rather than one per context. Typechecking in one context can call an type competer in another context which wouldn't have this flag set.

I say that we have another thing called RecoverableCyclicReference. Unfortunately, I'm not familar with it. But it seems like it might be something that should be incorporated into this this design. Maybe when the compiler is in propagate mode, cyclic errors should be thrown as the recoverable version?

@retronym

retronym Jul 22, 2016

Member

I think this is more appropriate as a single boolean for the entire Global, rather than one per context. Typechecking in one context can call an type competer in another context which wouldn't have this flag set.

I say that we have another thing called RecoverableCyclicReference. Unfortunately, I'm not familar with it. But it seems like it might be something that should be incorporated into this this design. Maybe when the compiler is in propagate mode, cyclic errors should be thrown as the recoverable version?

@milessabin

This comment has been minimized.

Show comment
Hide comment
@milessabin

milessabin Jul 22, 2016

Contributor

@retronym apropos RecoverableCyclicReference vs. CyclicReference my understanding is that a RecoverableCyclicReference indicates that a cycle has been encountered but that the typer should continue to follow alternative paths if any ... only once the alternatives are exhausted is a non-recoverable CyclicReference thrown. I think that RecoverableCyclicReference is recoverable precisely because it's just an ordinary subtype of TypeError with no special handling.

If this is the case then I think what I'm doing here is correct: the only things which result in a CyclicReference being thrown are definitely not-recoverable and should be reported in the normal case. In the maybeInitialize case I believe that it's correct to suppress the report and propagate to the try/catch at the top.

Apropos the flag being context-based vs. global: I take your point about the flag not being propagated to the context associated with a different completer. What would you recommend as a mechanism for representing the flag globally? A counter var on Global?

Contributor

milessabin commented Jul 22, 2016

@retronym apropos RecoverableCyclicReference vs. CyclicReference my understanding is that a RecoverableCyclicReference indicates that a cycle has been encountered but that the typer should continue to follow alternative paths if any ... only once the alternatives are exhausted is a non-recoverable CyclicReference thrown. I think that RecoverableCyclicReference is recoverable precisely because it's just an ordinary subtype of TypeError with no special handling.

If this is the case then I think what I'm doing here is correct: the only things which result in a CyclicReference being thrown are definitely not-recoverable and should be reported in the normal case. In the maybeInitialize case I believe that it's correct to suppress the report and propagate to the try/catch at the top.

Apropos the flag being context-based vs. global: I take your point about the flag not being propagated to the context associated with a different completer. What would you recommend as a mechanism for representing the flag globally? A counter var on Global?

@adriaanm adriaanm modified the milestone: 2.12.1 Jul 26, 2016

@adriaanm

This comment has been minimized.

Show comment
Hide comment
@adriaanm

adriaanm Jul 26, 2016

Member

For now, I've assigned this to 2.12.1. We can discuss whether we can make an exception to the RC cycle rules for this one (if it can be fixed post 2.12.0, it should be), but we first need to focus on clearing the PRs blocking RC1.

Member

adriaanm commented Jul 26, 2016

For now, I've assigned this to 2.12.1. We can discuss whether we can make an exception to the RC cycle rules for this one (if it can be fixed post 2.12.0, it should be), but we first need to focus on clearing the PRs blocking RC1.

@milessabin

This comment has been minimized.

Show comment
Hide comment
@milessabin

milessabin Aug 9, 2016

Contributor

I've done as @retronym suggested and replaced the context-based flag with a global counted one. It seems that the existing tests already exercise typechecking in one context calling a type completer in another context. I'd welcome suggestions for more tests which bear on that point.

Contributor

milessabin commented Aug 9, 2016

I've done as @retronym suggested and replaced the context-based flag with a global counted one. It seems that the existing tests already exercise typechecking in one context calling a type completer in another context. I'd welcome suggestions for more tests which bear on that point.

@milessabin

This comment has been minimized.

Show comment
Hide comment
@milessabin

milessabin Aug 9, 2016

Contributor

FTR, the use of global mutable state makes me ... uncomfortable. But it's in good company in this area and there doesn't seem to be any straightforward way of avoiding it.

Contributor

milessabin commented Aug 9, 2016

FTR, the use of global mutable state makes me ... uncomfortable. But it's in good company in this area and there doesn't seem to be any straightforward way of avoiding it.

@milessabin milessabin changed the title from Proof of concept fix for SI-7046 to Fix for SI-7046 Aug 9, 2016

@milessabin

This comment has been minimized.

Show comment
Hide comment
@milessabin

milessabin Aug 9, 2016

Contributor

Title and description updated to reflect where we've got to.

Contributor

milessabin commented Aug 9, 2016

Title and description updated to reflect where we've got to.

@milessabin milessabin changed the title from Fix for SI-7046 to Partial fix for SI-7046 Aug 9, 2016

@milessabin

This comment has been minimized.

Show comment
Hide comment
@milessabin

milessabin Aug 10, 2016

Contributor

Squashed and rebased.

Contributor

milessabin commented Aug 10, 2016

Squashed and rebased.

@soc

This comment has been minimized.

Show comment
Hide comment
@soc

soc Aug 26, 2016

Member

Does anyone know what happened to CI? It seems to be stuck here.

Member

soc commented Aug 26, 2016

Does anyone know what happened to CI? It seems to be stuck here.

@milessabin

This comment has been minimized.

Show comment
Hide comment
@milessabin

milessabin Aug 26, 2016

Contributor

Backport to 2.11.8 available here.

Contributor

milessabin commented Aug 26, 2016

Backport to 2.11.8 available here.

@ikhoon

This comment has been minimized.

Show comment
Hide comment

ikhoon commented Aug 28, 2016

👍

@philwills philwills referenced this pull request Sep 6, 2016

Merged

Unions #3

philwills added a commit to guardian/marley that referenced this pull request Sep 6, 2016

Move test thrift into separate project
By forcing it into a separate compilation unit we can get scalac to be happy. This is a workaround for  https://issues.scala-lang.org/browse/SI-7046 for which there's an open PR scala/scala#5284.

philwills added a commit to guardian/marley that referenced this pull request Sep 6, 2016

Move test thrift into separate project
By forcing it into a separate compilation unit we can get scalac to be happy. This is a workaround for  https://issues.scala-lang.org/browse/SI-7046 for which there's an open PR scala/scala#5284.

philwills added a commit to guardian/marley that referenced this pull request Sep 6, 2016

Move test thrift into separate project
By forcing it into a separate compilation unit we can get scalac to be happy. This is a workaround for  https://issues.scala-lang.org/browse/SI-7046 for which there's an open PR scala/scala#5284.
@SethTisue

This comment has been minimized.

Show comment
Hide comment
@SethTisue

SethTisue Sep 8, 2016

Member

Jenkins says:

# Failed test paths (this command will update checkfiles)
partest --update-check \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/t5418.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/t5418b.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/t5463.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/t5488-fn.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/t5488.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/t5500.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/t5500b.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/t5535.scala
Member

SethTisue commented Sep 8, 2016

Jenkins says:

# Failed test paths (this command will update checkfiles)
partest --update-check \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/t5418.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/t5418b.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/t5463.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/t5488-fn.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/t5488.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/t5500.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/t5500b.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/t5535.scala
@thedmitriyk

This comment has been minimized.

Show comment
Hide comment
@thedmitriyk

thedmitriyk Sep 19, 2016

👍 for this fix.
My project uses automatic typeclass derivation and was affected by SI-7046. The scenario was pretty trivial and not far off from known examples. This fix, as implemented in the Typelevel fork, resolved my issue brilliantly. Would love to see this merged into Lightbend Scala.

👍 for this fix.
My project uses automatic typeclass derivation and was affected by SI-7046. The scenario was pretty trivial and not far off from known examples. This fix, as implemented in the Typelevel fork, resolved my issue brilliantly. Would love to see this merged into Lightbend Scala.

@milessabin

This comment has been minimized.

Show comment
Hide comment
@milessabin

milessabin Oct 20, 2016

Contributor

Rebased ...

Contributor

milessabin commented Oct 20, 2016

Rebased ...

@milessabin

This comment has been minimized.

Show comment
Hide comment
@milessabin

milessabin Nov 16, 2016

Contributor

Rebased.

This has been well received in TLS and merged for 2.11.9 ... is there anything more that needs to be done before merging for 2.12.1?

Contributor

milessabin commented Nov 16, 2016

Rebased.

This has been well received in TLS and merged for 2.11.9 ... is there anything more that needs to be done before merging for 2.12.1?

@adriaanm

This comment has been minimized.

Show comment
Hide comment
@adriaanm

adriaanm Nov 16, 2016

Member

I expect this will be fine, but have not had time for a final review pass. Should get to it this week.

Member

adriaanm commented Nov 16, 2016

I expect this will be fine, but have not had time for a final review pass. Should get to it this week.

@milessabin

This comment has been minimized.

Show comment
Hide comment
@milessabin

milessabin Nov 28, 2016

Contributor

Rebased.

Contributor

milessabin commented Nov 28, 2016

Rebased.

@adriaanm adriaanm self-assigned this Nov 30, 2016

@adriaanm

This comment has been minimized.

Show comment
Hide comment
@adriaanm

adriaanm Dec 1, 2016

Member

LGTM.

I'm a little concerned about going into the type checker from a method on Symbol to force all symbols in the enclosing package, but since this is a specialized enough use case, I guess we can see how this plays out in practice.

In future, please include design motivation in the commit message. Since this has been discussed in the PR, I guess we can make an exception.

Member

adriaanm commented Dec 1, 2016

LGTM.

I'm a little concerned about going into the type checker from a method on Symbol to force all symbols in the enclosing package, but since this is a specialized enough use case, I guess we can see how this plays out in practice.

In future, please include design motivation in the commit message. Since this has been discussed in the PR, I guess we can make an exception.

@adriaanm adriaanm merged commit c2eb299 into scala:2.12.x Dec 1, 2016

5 checks passed

cla @milessabin signed the Scala CLA. Thanks!
Details
integrate-ide [3807] SUCCESS. Took 14 s.
Details
validate-main [4311] SUCCESS. Took 73 min.
Details
validate-publish-core [4181] SUCCESS. Took 4 min.
Details
validate-test [3682] SUCCESS. Took 65 min.
Details

@SethTisue SethTisue changed the title from Partial fix for SI-7046 to SI-7046 partial fix to knownDirectSubclasses for reflection users and macro authors Dec 5, 2016

dhpiggott added a commit to dhpiggott/liquidity that referenced this pull request Oct 29, 2017

Convert ZoneValidator actor protocol to Protobuf
The ZoneValidator protocol is now:
1. Independent of the WsProtocol.
2. Still represented by the same types that they always were, as far as speakers
   of the protocol are concerned.
3. Serialized to Protobuf messages.

(3) was the goal of this change, while (2) was a constraint on the solution that
I've imposed in order to retain meaningful types throughout most of the codebase
without resorting to using ScalaPB's TypeMappers (which can only do so much).

The introduction of several new `case object Empty ...` types on the application
side is in order to enable inductive derivation of ProtoConverter instances for
the wrapper coproducts that ScalaPB generates for `oneof` message types.

The movement of ZoneValidatorMessageSerializer fromm the actor-protocol module
to the server module was done in order ensure that its compilation happens
_after_ the compilation of both the generated Protobuf protocol types and the
original protocol types (both of which are compiled in the actor-protocol
module). This was necessary because ScalaPB generates the _instances_ of the
wrapper coproducts that it creats for `oneof` message types _within_ the
companion object that it creates for the trait of which they are children.
This property of the generated code interacted badly with the way that
scalac's knownDirectSubclasses macro method works, causing errors of the form
`error: knownDirectSubclasses of Foo observed before subclass Bar registered`.

I found that commenting out the three lines in ZoneValidatorMessageSerializer
that trigger the derivation of ProtoConverter instances that depend on coproduct
ProtoConverter instances being derived was sufficient for compilation to
succeed. I then found that a second compilation attempt with them _un_commented
would somewhat suprisingly also succeed. This finding coupled with the notes on
scala/scala#5284 confirmed to me that the issue was
only happening because of the above property of the generated ScalaPB code
(ProtoConverterSpec includes test cases for coproducts which work fine, so I
knew that other derivations were working - although somewhat surprisingly it
also works if the test coproduct instances are moved to be children of
companions of their parent traits...).

Hence the workaround of simply moving the serializer instances into the server
module (which is the only place they're used anyway*). This ensures that
derivation happens _after_ the necessary compiler phases have completed on the
coproducts in question.

*ZoneEventSerializer was in a separate module because analytics used to run as a
separate deployable which built against the persistence module (but not the
server module).

Other notes:

1. There are two TODOs re. unused import warnings in the ZoneEvent and
   ZoneValidatorMessage serializers. I've left these for later resolution
   because: (a) it's taken a fair amount of time already to get this change to
   where it is, (b) the size of the diff is getting large, and (c) they aren't
   issues with the code anyway - they really seem to be an IntelliJ issue, so
   the TODOs are really about finding a fix or workaround for that.
2. ClientConnectionActor (and its Spec) are using ProtoConverter instances to
   convert WsProtocol ZoneCommands to ZoneValidatorMessage ZoneCommands. As the
   numerous comments where this occurs explain, the terms asScala and asProto
   are not appropriate here. Indeed, that's not the only way this is not ideal.
   The more preferable solution would have been to _move_ the ZoneCommands from
   WsProtocol, rather than _copy_ them. That's actually what I first attempted
   to do. However, the issue I ran into was in adapting the scala-json-rpc
   message format constructors to support wrapping an external message in a way
   that would preserve the original JSON-RPC wire format. There would have been
   a solution, but using the recently developed ProtoConverter functionality
   ended up being a much faster solution to implement (and is also less
   verbose). Since the JSON-RPC protocol is now living on borrowed time and
   should be deprecated and then removed in the near future now that all the
   prerequisite steps are complete (this being the last of them), this solution
   is reasonable enough (it will disappear when the JSON-RPC protocol is
   removed). This leads nicely on to the final point:
3. play-json removal: as part of the deprecation and later removal of the
   JSON-RPC protocol, the number of modules which include play-json as a
   dependency should decrease and eventually reach zero. For now it remains as
   a dependency of most things because I've deliberately left the application
   side ADTs with JsObject as the metadata type in order to minimise the volume
   of individual Protobuf-conversion-related commits.

@kailuowang kailuowang referenced this pull request Dec 1, 2017

Merged

Restore full auto derivation for Empty and Functor #73

2 of 2 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment