Backport compiler determinism to 2.12.x [ci: last-only] #7203
Conversation
|
Going through the reviews of the 2.13 versions. Some concerns (will edit as I make my way through):
|
By slightly modifying an existing test to force creation of default getters for both `bar` methods _before_ typechecking the application, I was able to show a latent bug in the way the default getter is looked up in scope. The bespoke `Context.lookup` method did not respect shadowing, but rather considered the two, same-named default getters as overloaded. Because the overloaded symbol had NoSymbol as its owner, which didn't match the expected owner, neither default was eligible. This commit brings the code more into line with `Context.lookupSymbol` and respects shadowing. (cherry picked from commit 86f2028)
In scala#5700, I fixed a bug in the companion lookup, which ensured they were defined in the same scope. The same approach applies well to the lookup of default getters. You may ask, we can't just use: ``` context.lookupSymbol(name, _.owner == expectedOwner) ``` That doesn't individually lookup the entry in each enclosing nested scopes, but rather relies on the outer scope delegation in `Scope.lookupEntry` itself. This in turn relies on the way that nested scopes share the `elems` table with the enclosing scope: ``` final def newNestedScope(outer: Scope): Scope = { val nested = newScope nested.elems = outer.elems nested.nestinglevel = outer.nestinglevel + 1 ... } ``` If the outer scope is later mutated, in our case by lazily adding the default getter, the inner scope won't see the new elems. Context.lookupSymbol will jump immediately jump to search of the enclosing prefix. Perhaps a better design would be for the inner scope to retain a reference to the outer one, rather than just to the head of its elems linked list at the time the nested scope was created. (cherry picked from commit da14e9c)
The annotation removed in this diff was actually from `R`! (cherry picked from commit 623589a)
Small refactoring designed to make the subsequent commit more reviewable. Tightens up the definition of "companion"-s so that we no longer add a type alias in a package object into the pickle of a same-named module. (cherry picked from commit b41e651)
The pickle format does not explicitly enccode the order of decls. Instead, symbols are entered into an index in the order that they are found by the pickler, either as a definition or as a reference. During unpickling, symbols are read and entered into the owner's decls in that order. This is a cause of unstable compiler output: a class that mixes in members from some trait will have a different order if it is compiled jointly with / separately from that trait. This commit modifies the pickler with an initial pass that reserves index entries for all declarations in the declaration order. The pickle format and the unpickler are unchanged. (cherry picked from commit 8cc7e56)
- Factor out differences between constructors and regular methods
into an virtual call to a helper class
- Tease apart symbol creation/entry from synthesis of the default
getter tree to prepare for a subsequent commit that will perform
the first part eagerly.
- Add a test to show the unstable order of the default getter
symbols in the owner's scope.
(cherry picked from commit f44e1bf)
This stabilizes the order they appear in the owners scope. Previously, their order was goverened by the order that the methods bearing default parameters were type completed. Make macro annotations compatible with these changes (cherry picked from commit 87be453)
(cherry picked from commit 2c26cf2)
(cherry picked from commit befc337)
Fresh names are created using a FreshNameCreator, which appends
an increasing number to the given prefix.
```
scala> val fresh = new scala.reflect.internal.util.FreshNameCreator()
fresh: scala.reflect.internal.util.FreshNameCreator = scala.reflect.internal.util.FreshNameCreator@42b84286
scala> List("foo$", "bar$", "foo$").map(fresh.newName(_))
res1: List[String] = List(foo$1, bar$1, foo$2)
```
Each compilation unit had its own fresh name creator, which is used
in the regular compiler. Macros and quasiquotes make use of a global
creator (at least, as of scala#3401).
Both of these are too broadly scoped to help achieve deterministic
fresh names: if sources are recompiled in a different order or separately
recompiled, the fresh name counters can be different. Methods in a given
compilation unit are not necessarily typechecked in a linear fashion;
they might be typechecked ahead of time to provide an inferred type
to a caller.
This commit:
- Changes all known fresh name creations within the typer phase (in which
out-of-order typechecking is a factor) to use a fineer grained fresh
name creator. How fine grained? A fresh name generated as some position
`p` shares the fresh name generator scoped at the closest method or
class that encloses that the outermost enclosing tree at the same
position. This definition is designed to give a shared fresh name
creator for all fresh names generated in `macro1(macro2())`, even if
the fresh names are requiested from with a Typer in the macro enclosed
by a synthetic method.
- Changes macro fresh names to use the same fresh naming scheme as the regular
typechecker. An opt-out compiler option allows the old behaviour, but I'm
interested to find real-world cases where the new scheme actually causes
a problem
In addition, a small change is made to lambda lift to lift local methods in the
order that they are encountered during traversal, rather than sorting them
based on `Symbol.isLess` (which include `Symbol.id`, an order-of-typechecking
dependent value).
(cherry picked from commit 69d60cb)
I've used this to flush out the corner cases fixed in the previous commit. (cherry picked from commit dfaefa0)
(cherry picked from commit c5cc71f)
Previously, it was done while typechecking super calls, and would fail to see the fact that a yet-to-be-typechecked super constructor itself had a parameter aliased by a grand-parent class. (cherry picked from commit 3ae11c1)
typedRefinement defers the setting of this flag until the end of the compilation unit, which means that inferred types that are derived from the written refinement type can be unstable depending on whether they were computed before or after the flag was set. An alternative fix might be to just remove the setting of OVERRIDE in typedRefinement.unitToCheck. (cherry picked from commit f6ca3dd)
As per the discussion at scala@7184fe0#r29651930, all we want to avoid is name collision among the holes in a single pattern. For that, `c.freshName(<a constant>)` itself is sufficient and the randomness is not needed. This is what scala.meta does, and works just as well. This fixes scala/bug#11008 (cherry picked from commit 0336c14)
adriaanm
force-pushed
the
backport/determinism
branch
from
051379d
to
e764204
Compare
|
Preparing macro-annot PR at https://github.com/adriaanm/paradise/pull/new/2.12.8 |
|
Great to see this being backported. |
|
I added a community build for 2.12.8-bin-e764204-SNAPSHOT to the queue (https://scala-ci.typesafe.com/job/scala-2.12.x-integrate-community-build/3650). It's using a custom config (adriaanm/community-builds@bddc889) that uses macro-paradise at https://github.com/scalamacros/paradise/tree/2.12.8 (I incorporated the community build fixes) /cc @SethTisue FYI on queuing the build |
|
After a green community build, this is ready to merge! |
|
there were some failures in run 3650. the enumeratum one, at least, looks spurious, but let's see how a rebuild does: |
|
|
All test failures -- could they all be spurious?? Hitting rebuild one more time |
|
for comparison, we should see whether any of the same failures crop up in https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/3653/ which is a plain run that's running now |
|
The lift-json looks like a change in behaviour, or bug, in scalap. Scalap contains an independant implementation of the unpickler. I thought the changes to the pickler were backwards compatible, I just changed made the order of entries predictable. Oh, that's not scalap, it is yet-another-unpickler implementation. |
This fixed a regression in lift-json, which programattically uses scalap, and expects that method parameters are children of the method symbol.
|
I've pushed a fix for lift-json. Scheduled a community build: https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/3655/ |
|
I merged Seth's changes to the community build config into my branch that points to a different branch of paradise, and bumped the scala version to use 3dcbf47. Let's see how this one fares: https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/3656 |
|
followup run https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/3662/ failed: classutil, curryhoward, scala-refactoring, enumeratum, scalatex the enumeratum failure is unrelated (and now fixed by scala/community-builds/commit/32151444479ae4e1ecdc8d0202336ff64913f0a6) the scalatex failure is unrelated classutil, curryhoward, scala-refactoring look like they might be overly sensitive tests, hard to tell at a glance only 3 projects didn't run because of upstream failures (pureconfig, refined, scalasti) |
|
|
|
The |
Makes these tests work with scala/scala#7203
|
Scala refactoring: scala-ide/scala-refactoring#201 |
|
I'm happy that the community build hasn't found any bugs in this change. @SethTisue Should we update the community build to use my patches to the three projects above before we merge this? |
|
I think due diligence has now been done, I don't think it's necessary to do another run |
|
Is this waiting on any more reviews or can it be merged ? |
| @@ -27,7 +27,7 @@ abstract class ScalaSigSymbol extends Symbol { | |||
| def entry: ScalaSig#Entry | |||
| def index = entry.index | |||
|
|
|||
| lazy val children: Seq[Symbol] = applyScalaSigRule(ScalaSigParsers.symbols) filter (sym => sym.parent == Some(this) && !sym.isParam) | |||
| lazy val children: Seq[Symbol] = applyScalaSigRule(ScalaSigParsers.symbols) filter (sym => sym.parent == Some(this) && (this match { case _: MethodSymbol => true case _ => !sym.isParam})) | |||
There was a problem hiding this comment.
This is mabye a little ad-hoc, to filter class type params, but not method type params?
|
It looks ready, but I'll let @adriaanm merge it. |
Backports:
Which together resolve scala/scala-dev#405
A risk of the backport is that we create binary incompatibilities
with code compiled with
-opt:inlinethat refers to fresh namesthat are now changed. I had this problem with partest on 2.13.x:
scala/sbt-scala-module@7aee2b5#diff-b296be07ef3dd86a176475283d8583a4
Another risk is that there are serialization incompatibilities
that aren't covered by
SerializationStabilityTest.