Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Full bootstrap: Use Dotty as the reference compiler #5923

Merged
merged 4 commits into from Apr 22, 2019

Conversation

@smarter
Copy link
Member

@smarter smarter commented Feb 14, 2019

Before merging this I'd like to:

  • Wait for 0.13.0-RC1 to be released.
  • Change the nomenclature used in the build ("non-bootstrapped dotty"
    and "bootstrapped dotty" are not descriptive enough when the reference
    compiler is also dotty).
    Will do this later.
  • Cross-compile the build with 2.12 on the CI so that we don't
    accidentally start using Dotty features in case we need to revert this
    PR (we'll start usin Dotty features eventually, but let's wait
    until we're confident that this setup works well enough).
@smarter
Copy link
Member Author

@smarter smarter commented Feb 14, 2019

Tasty UUID (001eee9b-f167-8700-005e-a7bc5063cc00) file did not correspond the tasty UUID (00b923fc-a749-ef00-00c0-b993bb014d00) declared in the classfile /tmp/1/library/../out/bootstrap/dotty-library-bootstrapped/scala-0.13/dotty-library_0.13-0.13.0-bin-SNAPSHOT.jar(scala/quoted/package.class).

Hmm, @nicolasstucki do you remember how to debug this ?

@nicolasstucki
Copy link
Contributor

@nicolasstucki nicolasstucki commented Feb 21, 2019

I usually printed the tasty files (source file or full tasty) along with the UUID. For this kind of bug, it usually was an older version of the tasty file with a newer class file.

@@ -30,6 +30,7 @@ import scala.util.Properties.isJavaAtLeast

object Build {
val scalacVersion = "2.12.8"
val referenceVersion = "0.13.0-bin-20190213-d7090ab-NIGHTLY"
Copy link
Collaborator

@abeln abeln Mar 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll need a more recent nightly, because of quotes in StagedTuple and b32a4a8

@abeln
Copy link
Collaborator

@abeln abeln commented Mar 7, 2019

@nicolasstucki I'm trying to patch this in locally for our explicit nulls work (we want to bootstrap the compiler with explicit nulls), and I'm seeing the same error

Tasty UUID (0087cb62-52b8-0e00-003f-c5611a437a00) file did not correspond the tasty UUID (0074c917-b8c8-e900-00f9-8f1e95ce9c00) declared in the classfile /Users/abeln/src/dotty2/dotty/library/../out/bootstrap/dotty-library-bootstrapped/scala-0.14/dotty-library_0.14-0.14.0-bin-SNAPSHOT.jar(dotty/DottyPredef.class).

When you say

For this kind of bug, it usually was an older version of the tasty file with a newer class file.

does that suggest that the fix is cleaning some sbt cache? Or do I need to do something else? Thanks!

@smarter
Copy link
Member Author

@smarter smarter commented Mar 7, 2019

There's probably some classpath confusion somewhere, I'll investigate in the near future, but I'm not sure why this is needed for you given that dotty-bootstrapped/compile should already give you the information you want

@abeln
Copy link
Collaborator

@abeln abeln commented Mar 7, 2019

There's probably some classpath confusion somewhere, I'll investigate in the near future, but I'm not sure why this is needed for you given that dotty-bootstrapped/compile should already give you the information you want

Ok, but I see the error when I run dotc, so I'm not able to run the boostrapped compiler on any files. In any case, thanks for looking into this.

@smarter
Copy link
Member Author

@smarter smarter commented Mar 7, 2019

What error ?

@smarter
Copy link
Member Author

@smarter smarter commented Mar 7, 2019

To clarify: Iwas saying that you don't need this PR to test the bootstrapped compiler since you can already do one stage of bootstrapping with the current build

@abeln
Copy link
Collaborator

@abeln abeln commented Mar 7, 2019

To clarify: Iwas saying that you don't need this PR to test the bootstrapped compiler since you can already do one stage of bootstrapping with the current build

I'm confused. When we switch to explicit nulls, we'll need to use union types inside dotty. My understanding of the current bootstrapping process is

  1. scalac compiles dotty
  2. dotty compiles dotty

If we use union types inside dotty, then step 1) will fail, because scalac doesn't have union types, right? What am I missing?

@smarter
Copy link
Member Author

@smarter smarter commented Mar 7, 2019

Ah ok, I see now, but can't you work around that by defining something like:

type Nullable[T] = T | Null

In a file which is only compiled by the bootstrapped compiler, and:

type Nullable[T] = T

in a file that is only compiled by Scala 2. Then everything else can cross-compile.

@smarter smarter force-pushed the full-bootstrap branch 3 times, most recently from a2dccf6 to 7d1f407 Mar 26, 2019
@anatoliykmetyuk anatoliykmetyuk removed this from the 0.14 Tech Preview milestone Apr 11, 2019
@anatoliykmetyuk anatoliykmetyuk added this to the 0.15 Tech Preview milestone Apr 11, 2019
@smarter smarter changed the title [WIP] Full bootstrap: Use Dotty as the reference compiler Full bootstrap: Use Dotty as the reference compiler Apr 21, 2019
@smarter smarter force-pushed the full-bootstrap branch from 7d1f407 to 0a8ad93 Apr 21, 2019
@smarter smarter requested a review from nicolasstucki Apr 21, 2019
@nicolasstucki
Copy link
Contributor

@nicolasstucki nicolasstucki commented Apr 22, 2019

Both tests/run/i5536/Macro_1.scala and tests/run/i5533b/Macro_1.scala are using implicit val toolbox ... by mistake, if you remove that line you will be able to place them in tests/run-macros.

Copy link
Contributor

@nicolasstucki nicolasstucki left a comment

Otherwise LGTM

project/Build.scala Show resolved Hide resolved
The non-bootstrapped compiler is now compiled by Dotty 0.14.0-RC1
instead of Scala 2.12.8. The bootstrapped compiler is compiled using the
non-bootstrapped compiler as before.

For now, we still support compilation with Scala 2.12, this is achieved
by having separate `src-2.x` and `src-3.x` directory for the library.

To use Scala 2.12 as the reference compiler, run:

++2.12.8

in sbt, or change `referenceVersion` to "2.12.8" in Build.scala.
@smarter smarter force-pushed the full-bootstrap branch from 0a8ad93 to d75092c Apr 22, 2019
Copy link
Contributor

@nicolasstucki nicolasstucki left a comment

Otherwise LGTM

tests/run/i5536/Macro_1.scala Show resolved Hide resolved
smarter added 3 commits Apr 22, 2019
Compiling a macro with a non-bootstrapped compiler is fine, but running
that macro in a subsequent compiler run as is usually done in macro
tests is problematic since the MacroClassLoader is created using the
compiler classpath (which contains a bootstrapped library) with the
regular JVM classpath as parent classloader (which contains a
non-bootstrapped library), this is inconsistent and could lead to various
weird problems depending on the order in which things get loaded.

One possible way to run these tests would be to always put the
bootstrapped library on the JVM classpath, even if we're using the
non-bootstrapped compiler, but this would cause test failures every time
we would break binary compatibility.

Instead, this commit moves the relevant tests to {pos,neg,run}-macros
which will only be run with a bootstrapped compiler. To speed up
development, it might be useful to add a flag to `testCompilation` that
would put the bootstrapped library on the JVM classpath and still run
these tests, but this is left as future work.
Like explained in the previous commit this kind of tests should be
bootstrapped-only.
Some check files were not deleted or moved when the corresponding test
was deleted or moved.
@smarter smarter force-pushed the full-bootstrap branch from d75092c to ee7cc53 Apr 22, 2019
@smarter smarter merged commit b8fddaf into lampepfl:master Apr 22, 2019
2 checks passed
@smarter
Copy link
Member Author

@smarter smarter commented Apr 22, 2019

@odersky @anatoliykmetyuk @biboudis @OlivierBlanvillain @liufengyun @AleksanderBG @milessabin @allanrenucci @sjrd This PR is finally in! Here's a few things to watch out for now that we use Dotty instead of Scala 2.12 as our reference compiler:

  • Incremental compilation issues, if you notice them please open an issue with a branch where the problem can be reproduced:
    • Undercompilation: When we make a change, hit compile, and some files that should be recompiled are not. This can manifest itself in various ways such as a MethodNotFoundException at runtime (as an aside, switching to Dotty fixes one long-standing undercompilation issue: if you change the body of a final val, every usage of that final val should now get correctly recompiled).
    • Overcompilation: When we make a change, hit compile, and a lot more files than expected end up being recompiled. This is benign but annoying.
  • Classpath confusion: Especially when running the non-bootstrapped compiler, it's easy to get confused: on the JVM classpath (the one set using java -classpath ... used to load classes in the JVM) we have a non-bootstrapped library and compiler, but on the compiler classpath (the one set using dotc -classpath ... used to create symbols in the compiler) we always have the bootstrapped library (since we may not be able to load symbols from the tasty files generated by the reference compiler). I've tried to make sure we use the correct classpath everywhere but there might still be some problems lurking, they can manifest themselves in various ways such as a "TASTy signature has wrong version." exception.
    Things get even more complicated when using macros since running a macro involves loading the compiler classpath into the JVM classpath, therefore macro tests are now limited to the bootstrapped compiler, see aed2582

If at any point you need to go back to using Scala 2.12 as the reference compiler, you can do it very easily by running:

++2.12.8

Inside sbt. Or by changing

val referenceVersion = "0.14.0-RC1"
to "2.12.8".

Happy hacking!

@allanrenucci allanrenucci deleted the full-bootstrap branch Apr 22, 2019
@smarter smarter mentioned this pull request May 2, 2019
smarter added a commit to lampepfl/dotty-ci that referenced this issue Sep 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants