Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add scala2-library-cc-tasty project #18967

Merged

Conversation

nicolasstucki
Copy link
Contributor

@nicolasstucki nicolasstucki commented Nov 17, 2023

We can override the sources of the standard library with cc annotation in this project and generate a TASTy version of the standard library.

Usage:

sbt> set ThisBuild/Build.scala2Library := Build.Scala2LibraryCCTasty
sbt> scala3-bootstrapped/testCompilation
sbt> scala3-bootstrapped/test
sbt> scala3-bootstrapped/run ...
sbt> scala2-library-cc/run diff
sbt> scala2-library-cc/run diff scala/Option.scala
sbt> scala2-library-cc/run diff scala/collection/immutable

Current limitations

Empty self types in library

We need to override Function1 and PartialFunction to remove their empty self type. These cause some problem with capture checking. We need to fix these cases. I will open an issue once we have a way to reproduce it with this PR.

Cannot use the TASTy jar yet

There is a bug when we try to class load the captured checked library from TASTy. These are due to some cyclic reference while unpickling the TASTy.

The current theory is that this due to the CC annotations in the top-level classes.
If this is the case, we can fix this issue by adding these language features as TASTy attributes (see #19033).

This was wrong or only part of the problems. The current issue can be reproduced with #19115.

@nicolasstucki nicolasstucki self-assigned this Nov 17, 2023
@nicolasstucki nicolasstucki force-pushed the add-scala2-library-cc-tasty-project branch 3 times, most recently from c4cdbc4 to 7d23d16 Compare November 20, 2023 09:03
@nicolasstucki nicolasstucki marked this pull request as draft November 20, 2023 16:32
@nicolasstucki nicolasstucki force-pushed the add-scala2-library-cc-tasty-project branch 10 times, most recently from 109d839 to 2bcc5a2 Compare November 24, 2023 08:14
@nicolasstucki
Copy link
Contributor Author

The commit that adds a large number of files (64f3fdc) was created uisng

sbt> scala2-library-cc/run clone ...

where ... was the list of all sources that where cloned in that commit.

@nicolasstucki nicolasstucki marked this pull request as ready for review November 24, 2023 21:17
@nicolasstucki nicolasstucki assigned sjrd and unassigned nicolasstucki Nov 27, 2023
@nicolasstucki nicolasstucki force-pushed the add-scala2-library-cc-tasty-project branch from 2cdeff3 to f995c4e Compare November 28, 2023 10:40
odersky added a commit that referenced this pull request Dec 15, 2023
)

#### New behaviour, tested on #18967

```scala
// Test.scala
def test: Option[Int] = ???
```

```
sbt> set ThisBuild/Build.scala2Library := Build.Scala2LibraryCCTasty
sbt> scala3-bootstrapped/scalac Test.scala
-- Error: Test.scala:1:10 ----------------------------------------------------
1 |def test: Option[Int] = ???
  |          ^
  |Could not read definition class Option in out/bootstrap/scala2-library-cc-tasty/scala-3.4.0-RC1-bin-SNAPSHOT-nonbootstrapped/scala2-library-cc-tasty_3-3.4.0-RC1-bin-SNAPSHOT.jar(scala/Option.tasty). Caused by the following exception:
  |dotty.tools.dotc.core.UnpicklingError: Could not read definition type IterableOnce in out/bootstrap/scala2-library-cc-tasty/scala-3.4.0-RC1-bin-SNAPSHOT-nonbootstrapped/scala2-library-cc-tasty_3-3.4.0-RC1-bin-SNAPSHOT.jar(scala/package.tasty). Caused by the following exception:
  |dotty.tools.dotc.core.UnpicklingError: Could not read definition trait IterableOnce in out/bootstrap/scala2-library-cc-tasty/scala-3.4.0-RC1-bin-SNAPSHOT-nonbootstrapped/scala2-library-cc-tasty_3-3.4.0-RC1-bin-SNAPSHOT.jar(scala/collection/IterableOnce.tasty). Caused by the following exception:
  |dotty.tools.dotc.core.UnpicklingError: Could not read definition trait Iterable in out/bootstrap/scala2-library-cc-tasty/scala-3.4.0-RC1-bin-SNAPSHOT-nonbootstrapped/scala2-library-cc-tasty_3-3.4.0-RC1-bin-SNAPSHOT.jar(scala/collection/immutable/Iterable.tasty). Caused by the following exception:
  |dotty.tools.dotc.core.UnpicklingError: Could not read definition trait Iterable out/bootstrap/scala2-library-cc-tasty/scala-3.4.0-RC1-bin-SNAPSHOT-nonbootstrapped/scala2-library-cc-tasty_3-3.4.0-RC1-bin-SNAPSHOT.jar(scala/collection/Iterable.tasty). Caused by the following exception:
  |dotty.tools.dotc.core.CyclicReference: Cyclic reference involving trait Seq
  |
  |Run with -Ydebug-unpickling to see full stack trace.

sbt> scala3-bootstrapped/scalac -Ydebug-unpickling Test.scala
-- Error: Test.scala:1:10 ----------------------------------------------------
1 |def test: Option[Int] = ???
  |          ^
  |Could not read definition class Option in out/bootstrap/scala2-library-cc-tasty/scala-3.4.0-RC1-bin-SNAPSHOT-nonbootstrapped/scala2-library-cc-tasty_3-3.4.0-RC1-bin-SNAPSHOT.jar(scala/Option.tasty). Caused by the following exception:
  |dotty.tools.dotc.core.UnpicklingError: Could not read definition type IterableOnce in out/bootstrap/scala2-library-cc-tasty/scala-3.4.0-RC1-bin-SNAPSHOT-nonbootstrapped/scala2-library-cc-tasty_3-3.4.0-RC1-bin-SNAPSHOT.jar(scala/package.tasty). Caused by the following exception:
  |dotty.tools.dotc.core.UnpicklingError: Could not read definition trait IterableOnce in out/bootstrap/scala2-library-cc-tasty/scala-3.4.0-RC1-bin-SNAPSHOT-nonbootstrapped/scala2-library-cc-tasty_3-3.4.0-RC1-bin-SNAPSHOT.jar(scala/collection/IterableOnce.tasty). Caused by the following exception:
  |dotty.tools.dotc.core.UnpicklingError: Could not read definition trait Iterable in out/bootstrap/scala2-library-cc-tasty/scala-3.4.0-RC1-bin-SNAPSHOT-nonbootstrapped/scala2-library-cc-tasty_3-3.4.0-RC1-bin-SNAPSHOT.jar(scala/collection/immutable/Iterable.tasty). Caused by the following exception:
  |dotty.tools.dotc.core.UnpicklingError: Could not read definition trait Iterable in out/bootstrap/scala2-library-cc-tasty/scala-3.4.0-RC1-bin-SNAPSHOT-nonbootstrapped/scala2-library-cc-tasty_3-3.4.0-RC1-bin-SNAPSHOT.jar(scala/collection/Iterable.tasty). Caused by the following exception:
  |dotty.tools.dotc.core.CyclicReference: Cyclic reference involving trait Seq
  |    dotty.tools.dotc.core.CyclicReference$.apply(TypeErrors.scala:186)
  |    dotty.tools.dotc.core.SymDenotations$BaseDataImpl.apply(SymDenotations.scala:2973)
  |    dotty.tools.dotc.core.SymDenotations$ClassDenotation.baseData(SymDenotations.scala:1954)
  |    dotty.tools.dotc.core.SymDenotations$ClassDenotation.baseClassSet(SymDenotations.scala:1970)
  |    dotty.tools.dotc.core.SymDenotations$ClassDenotation.isValueClass(SymDenotations.scala:2032)
  |    dotty.tools.dotc.core.SymUtils.isDerivedValueClass(SymUtils.scala:85)
  |    dotty.tools.dotc.core.TypeErasure$.dotty$tools$dotc$core$TypeErasure$$$erasureDependsOnArgs(TypeErasure.scala:74)
  |    ...
```

#### Old behaviour
```
sbt> set ThisBuild/Build.scala2Library := Build.Scala2LibraryCCTasty
sbt> scala3-bootstrapped/scalac Test.scala 
-- Error: Test.scala:1:10 ----------------------------------------------------
1 |def test: Option[Int] = ???
  |          ^
  |Could not read definition of class Option in /Users/nicolasstucki/Github/dotty/scala2-library-cc-tasty/../out/bootstrap/scala2-library-cc-tasty/scala-3.4.0-RC1-bin-SNAPSHOT-nonbootstrapped/scala2-library-cc-tasty_3-3.4.0-RC1-bin-SNAPSHOT.jar(scala/Option.tasty)
  |An exception was encountered:
  |  dotty.tools.dotc.core.TypeError$$anon$1: Could not read definition of type IterableOnce in /Users/nicolasstucki/Github/dotty/scala2-library-cc-tasty/../out/bootstrap/scala2-library-cc-tasty/scala-3.4.0-RC1-bin-SNAPSHOT-nonbootstrapped/scala2-library-cc-tasty_3-3.4.0-RC1-bin-SNAPSHOT.jar(scala/package.tasty)
  |An exception was encountered:
  |  dotty.tools.dotc.core.TypeError$$anon$1: Could not read definition of trait IterableOnce in /Users/nicolasstucki/Github/dotty/scala2-library-cc-tasty/../out/bootstrap/scala2-library-cc-tasty/scala-3.4.0-RC1-bin-SNAPSHOT-nonbootstrapped/scala2-library-cc-tasty_3-3.4.0-RC1-bin-SNAPSHOT.jar(scala/collection/IterableOnce.tasty)
  |An exception was encountered:
  |  dotty.tools.dotc.core.TypeError$$anon$1: Could not read definition of trait Iterable in /Users/nicolasstucki/Github/dotty/scala2-library-cc-tasty/../out/bootstrap/scala2-library-cc-tasty/scala-3.4.0-RC1-bin-SNAPSHOT-nonbootstrapped/scala2-library-cc-tasty_3-3.4.0-RC1-bin-SNAPSHOT.jar(scala/collection/immutable/Iterable.tasty)
  |An exception was encountered:
  |  dotty.tools.dotc.core.TypeError$$anon$1: Could not read definition of trait Iterable in /Users/nicolasstucki/Github/dotty/scala2-library-cc-tasty/../out/bootstrap/scala2-library-cc-tasty/scala-3.4.0-RC1-bin-SNAPSHOT-nonbootstrapped/scala2-library-cc-tasty_3-3.4.0-RC1-bin-SNAPSHOT.jar(scala/collection/Iterable.tasty)
  |An exception was encountered:
  |  dotty.tools.dotc.core.CyclicReference: Cyclic reference involving trait Seq
  |Run with -Ydebug-unpickling to see full stack trace.
  |Run with -Ydebug-unpickling to see full stack trace.
  |Run with -Ydebug-unpickling to see full stack trace.
  |Run with -Ydebug-unpickling to see full stack trace.
  |Run with -Ydebug-unpickling to see full stack trace.
1 error found

sbt> scala3-bootstrapped/scalac t/Test.scala -Ydebug-unpickling
-- Error: t/Test.scala:1:10 ----------------------------------------------------
1 |def test: Option[Int] = ???
  |          ^
  |          Cyclic reference involving trait Seq
  |
  | longer explanation available when compiling with `-explain`
1 error found
```
@nicolasstucki nicolasstucki force-pushed the add-scala2-library-cc-tasty-project branch from f995c4e to a7108d7 Compare December 20, 2023 09:13
nicolasstucki added a commit that referenced this pull request Dec 20, 2023
This command allow us to quickly see the diff between the original
library and the overwritten files in `scala2-library-bootstrapped/src`.

Instructions are printed with `scala2-library-bootstrapped/run` as with
other commands for `scala2-library-bootstrapped`.

Extracted from #18967.
Also useful for #19231.
@nicolasstucki nicolasstucki force-pushed the add-scala2-library-cc-tasty-project branch from a7108d7 to 38770b8 Compare December 20, 2023 16:22
Copy link
Member

@sjrd sjrd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should reuse the sources of the CC stdlib instead of copy it once more.

Comment on lines 661 to 667
jars.get(jarName) match {
case Some(jar) =>
Seq(s"-Ddotty.tests.tasties.scalaLibrary=$jar")
case None =>
log.warn("Scala 2 library TASTy is ignored on non-bootstrapped compiler")
Seq.empty
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
jars.get(jarName) match {
case Some(jar) =>
Seq(s"-Ddotty.tests.tasties.scalaLibrary=$jar")
case None =>
log.warn("Scala 2 library TASTy is ignored on non-bootstrapped compiler")
Seq.empty
}
jars.get(jarName) match {
case Some(jar) =>
Seq(s"-Ddotty.tests.tasties.scalaLibrary=$jar")
case None =>
log.warn("Scala 2 library TASTy is ignored on non-bootstrapped compiler")
Seq.empty
}

We can override the sources of the standard library with cc annotation
in this project and generate a TASTy version of the standard library.

Usage:
```
sbt> set ThisBuild/Build.scala2Library := Build.Scala2LibraryCCTasty
sbt> scala3-bootstrapped/testCompilation
sbt> scala3-bootstrapped/test
sbt> scala3-bootstrapped/run ...
```
Note that to compile this version of the library we do not patch the
library with `scala.runtime.stdLibPatches`.
Also reduce diff with the original Scala 2 library sources.
@nicolasstucki nicolasstucki force-pushed the add-scala2-library-cc-tasty-project branch from 38770b8 to 5c6e542 Compare December 21, 2023 16:05
@nicolasstucki nicolasstucki merged commit 25eb48e into scala:main Jan 3, 2024
19 checks passed
@nicolasstucki nicolasstucki deleted the add-scala2-library-cc-tasty-project branch January 3, 2024 08:37
nicolasstucki added a commit to dotty-staging/dotty that referenced this pull request Jan 4, 2024
Update docs to align with update in scala#18967.
bishabosha added a commit that referenced this pull request Jan 4, 2024
Update docs to align with update in #18967.


[skip ci]
odersky pushed a commit to dotty-staging/dotty that referenced this pull request Jan 6, 2024
Update docs to align with update in scala#18967.
@Kordyjan Kordyjan added this to the 3.4.1 milestone Feb 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants