Switch to the 2.13 standard library #7019

smarter · 2019-08-08T21:21:09Z

Still to do:

Fix all the deprecation warnings 😱
Get all the tests to pass.
Update all the projects of the community build (or temporarily disable some of them if we can't easily update them)
Switch to the 2.13 case class hashing scheme: fixes Mix the productPrefix.hashCode into the hash code of case classes, like Scala 2.13.0-RC1 #5732

As soon as this PR is merged we should publish a new nightly and use it as the reference compiler, so that we don't spend too much time with a build that uses both the 2.12 and 2.13 stdlib (it works, but it can be confusing and cumbersome).

dottybot

Hello, and thank you for opening this PR! 🎉

All contributors have signed the CLA, thank you! ❤️

Commit Messages

We want to keep history, but for that to actually be useful we have
some rules on how to format our commit messages (relevant xkcd).

Please stick to these guidelines for commit messages:

Separate subject from body with a blank line

When fixing an issue, start your commit message with Fix #<ISSUE-NBR>:

Limit the subject line to 72 characters

Capitalize the subject line

Do not end the subject line with a period

Use the imperative mood in the subject line ("Add" instead of "Added")

Wrap the body at 80 characters

Use the body to explain what and why vs. how

adapted from https://chris.beams.io/posts/git-commit

Have an awesome day! ☀️

smarter · 2019-08-08T21:31:20Z

@bishabosha I'll look into the failing test cases, meanwhile, feel free to help with any of the other items in the todo list above.

smarter · 2019-08-08T21:36:59Z

On the CI the community build fails to build every project with:

module not found: ch.epfl.lamp#sbt-dotty;0.3.4-SNAPSHOT

But I can't reproduce that locally. Weird, but this doesn't prevent us from working on it locally at least.

tests/neg-custom-args/repeatedArgs213.scala

tests/pos-special/repeatedArgs213.scala

smarter · 2019-08-08T21:51:23Z

@milessabin There's (at least) one compilation error for shapeless on 2.13:

[error] -- [E007] Type Mismatch Error: /home/smarter/opt/dotty/community-build/community-projects/shapeless/core/src/main/scala/shapeless/shapeless.scala:64:51
[error] 64 |      WrappedArray.make[String](summonValuesAsArray[mirror.MirroredElemLabels])
[error]    |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[error]    |                                Found:    Array[Any]
[error]    |                                Required: Array[String]
[error] one error found

If you have the time, it'd be great if you could take a look at it.

bishabosha · 2019-08-09T08:42:44Z

now I see why there is 😱 for deprecation

compiler/src/dotty/tools/dotc/core/classfile/ClassfileParser.scala

milessabin · 2019-08-09T11:47:08Z

@smarter you should be good after #7020.

smarter · 2019-08-09T12:15:20Z

@milessabin There's still an error

[error] -- [E007] Type Mismatch Error: /home/smarter/opt/dotty/community-build/community-projects/shapeless/core/src/main/scala/shapeless/shapeless.scala:65:31
[error] 65 |      WrappedArray.make[String](summonValuesAsArray[mirror.MirroredElemLabels, String])
[error]    |      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[error]    |      Found:    scala.collection.mutable.ArraySeq[String]
[error]    |      Required: Seq[String]

This can be solved by adding a .toSeq to get an immutable Seq, but there might be a nicer way. Note that you don't necessarily need to keep compiling against both the 2.12 and 2.13 stdlib, I'm happy with a 2.13-only git branch.

milessabin · 2019-08-09T12:18:43Z

@smarter I'll add the .toSeq for now and switch over to 2.13-only once this PR is merged.

milessabin · 2019-08-09T13:21:16Z

Done in #7021.

smarter

@bishabosha Would be good to add some test cases for the hashcode changes.

compiler/src/dotty/tools/dotc/transform/SyntheticMembers.scala

smarter · 2019-08-09T21:04:07Z

Would be good to add some test cases for the hashcode changes.

Actually we already have a related test that should be updated: https://github.com/lampepfl/dotty/blob/master/tests/run/caseClassHash.scala https://github.com/lampepfl/dotty/blob/master/tests/run/caseClassHash.check

bishabosha · 2019-08-09T21:05:52Z

with the new update ## and MurmurHash produce the same value

milessabin · 2019-08-14T09:12:20Z

@bishabosha what's with the shapeless bump? That's already merged on master ... wouldn't it be easier to rebase?

smarter · 2019-08-14T16:27:31Z

@milessabin Rebased with your patch in.

smarter · 2019-08-14T23:08:32Z

@liufengyun Unfortunately, https://github.com/liufengyun/bench/tree/master/tests/scala-library does not compile anymore now that the compiler runs using the 2.13 stdlib. I tried to hack it to make it compile, but it seems pretty hard, so I think we'll have to remove it from the benchmarks, and replace it by the new stdlib.

The value of type Unit is `()`, using `Unit` usually also works because it refers to the companion object of the class Unit, and because of automatic `()` insertion when the expected type is `Unit`, but it's still wrong. 2.13 marked it as `@compileTimeOnly` which is how I noticed.

This is easier to reason about, in particular because it avoids having to think about what kind of classloader sbt decided to inflict upon us. This fixes running the community-build test after switching to the 2.13 stdlib in the bootstrapped build.

This matches the behavior of Scala 2, this is necessary when using the 2.13 stdlib where scala.Serializable is a type alias of java.lang.Serializable, otherwise the compiler will disallow extending both Serializable and AnyVal.

This required some surgery in Build.scala: when compiling the non-bootstrapped compiler, we need the 2.12 stdlib on the JVM and compiler classpaths, because the current reference compiler cannot be run using the 2.13 stdlib, on the other hand when _running_ the non-bootstrapped compiler we need to the 2.13 stdlib on the compiler classpath to produce a bootstrapped compiler that links against 2.13. Upgrade the Scala 2 PickleFormat version to 5.2 (which is in fact identical to 5.0). Remove no longer needed -Ynew-collections flag and update the related test files.

Replace getExternalDep and findLib by findArtifact, which is safer than getLib was (it doesn't use `contains` to find the name which might yield false positives).

productElementName is no longer generated in Desugar after the previous commit.

It doesn't output an error anymore for some reason, but that's fine: we only care about it not crashing the compiler, which is what fuzzy tests check.

This fixes a pickling/unpickling difference in sip23-valueof.scala and makes sense in general.

Needs to be replaced by something that doesn't use the parallel collections, I attempted to do so in b9ba16e but the result caused deadlocks, don't have time to investigate right now.

I accidentally replaced `"` by `'` in the generated .dotty-ide.json file in 2ae438d which broke everything.

We now need to dealias deeply to get tests/run/Course-2002-13.scala and tests/run/enrich-gentraversable.scala to pass.

`membersBasedOnFlags` internally uses `memberNames` which returns a Set, so the order of its elements is unstable. We were using it when generating static forwarder methods which lead to some idempotency tests failing (no idea why this didn't manifest itself before). Fixed by adding a new `sortedMembersBasedOnFlags` which sorts members by names and signatures (this required adding a somewhat arbitrary lexicographic ordering to Signature).

So far, Jar was defined as an Iterable implementation that overrides `foreach` and then define `override def iterator = this.toList.iterator` That was OK with the 2.12 stdlib, but stackoverflows with the 2.13 one due to `toList` now being implemented in terms of `iterator`. Any way, it turns out that we don't really need Jar to be an Iterable, so the simplest fix was to just remove these methods, and add back a `toList` for the one usesite that needed it (not very efficient, but not any worse than before).

Not needed with 2.13 where CanBuildFrom is mostly gone.

Originally fixed in f8a746e, lost in b9ba16e when tests were updated based on their version in scala/scala.

To pick up the improvements since the previous one.

And the revert the CI changes from the previous commit which are no longer needed.

bishabosha · 2019-08-25T15:15:21Z

🔥🔥🔥

dottybot reviewed Aug 8, 2019

View reviewed changes

smarter force-pushed the upgrade/scalac-2.13 branch from 3a317ae to 40c9fee Compare August 8, 2019 21:24

allanrenucci reviewed Aug 8, 2019

View reviewed changes

tests/neg-custom-args/repeatedArgs213.scala Show resolved Hide resolved

tests/pos-special/repeatedArgs213.scala Show resolved Hide resolved

smarter force-pushed the upgrade/scalac-2.13 branch from 40c9fee to f8f8a18 Compare August 8, 2019 21:55

allanrenucci reviewed Aug 9, 2019

View reviewed changes

compiler/src/dotty/tools/dotc/core/classfile/ClassfileParser.scala Show resolved Hide resolved

bishabosha force-pushed the upgrade/scalac-2.13 branch from 5cc7de5 to 0099330 Compare August 9, 2019 14:44

smarter commented Aug 9, 2019

View reviewed changes

bishabosha force-pushed the upgrade/scalac-2.13 branch from 1ded229 to abb75a0 Compare August 12, 2019 11:55

bishabosha force-pushed the upgrade/scalac-2.13 branch from 88dfe50 to 44c1a98 Compare August 14, 2019 10:00

smarter force-pushed the upgrade/scalac-2.13 branch from 490fc95 to 4762d30 Compare August 14, 2019 16:27

bishabosha force-pushed the upgrade/scalac-2.13 branch from 45c46d2 to fffc3c4 Compare August 15, 2019 14:12

smarter force-pushed the upgrade/scalac-2.13 branch from e0a6fab to be5e67d Compare August 16, 2019 13:02

smarter added 5 commits August 19, 2019 10:43

Always fork when running stuff

f3ab8b9

This is easier to reason about, in particular because it avoids having to think about what kind of classloader sbt decided to inflict upon us. This fixes running the community-build test after switching to the 2.13 stdlib in the bootstrapped build.

Treat some Java interfaces as universal traits

f041b1d

This matches the behavior of Scala 2, this is necessary when using the 2.13 stdlib where scala.Serializable is a type alias of java.lang.Serializable, otherwise the compiler will disallow extending both Serializable and AnyVal.

Slight build refactoring

a6efeb6

Replace getExternalDep and findLib by findArtifact, which is safer than getLib was (it doesn't use `contains` to find the name which might yield false positives).

smarter and others added 11 commits August 23, 2019 16:09

Neg test i1650 now needs an extra error marker

6e32d39

scalajs-ir has now been published for 2.13

f82d9a3

move productElementName generation to SyntheticMethods

c5f9b2a

Remove dead case and update checkfile

87416ea

productElementName is no longer generated in Desugar after the previous commit.

Adapt sbt scripted tests for the 2.13 stdlib

4fd550f

Make i3187 a fuzzy test

acfe55c

It doesn't output an error anymore for some reason, but that's fine: we only care about it not crashing the compiler, which is what fuzzy tests check.

Inlining: Constant-fold select trees

a553078

This fixes a pickling/unpickling difference in sip23-valueof.scala and makes sense in general.

Move original t5375 to pending

b4b9fe5

Needs to be replaced by something that doesn't use the parallel collections, I attempted to do so in b9ba16e but the result caused deadlocks, don't have time to investigate right now.

Fix language server tests

79bcc5c

I accidentally replaced `"` by `'` in the generated .dotty-ide.json file in 2ae438d which broke everything.

Fix -Ytest-pickler handling of aliases

960498b

We now need to dealias deeply to get tests/run/Course-2002-13.scala and tests/run/enrich-gentraversable.scala to pass.

smarter force-pushed the upgrade/scalac-2.13 branch 2 times, most recently from dfbb87a to 1ba7e75 Compare August 25, 2019 12:00

bishabosha and others added 10 commits August 25, 2019 15:42

update classpath for InterfaceEntryPointTest

13e76a7

Update SignatureHelpTest for the 2.13 stdlib

64bcaad

Remove hiding of CanBuildFrom params from the IDE/REPL

3e4a9cc

Not needed with 2.13 where CanBuildFrom is mostly gone.

Make Scala.js happy

a43efe3

Re-fix test failure on Java 11

9e77e21

Originally fixed in f8a746e, lost in b9ba16e when tests were updated based on their version in scala/scala.

Fix a typo in Signature

ce05244

Delete obsolete comment

cd56de3

Add reference to issue with union types

5a3ad0d

Force another nightly to be published

6960f8d

To pick up the improvements since the previous one.

smarter force-pushed the upgrade/scalac-2.13 branch from 823254e to 6960f8d Compare August 25, 2019 13:51

Bump reference compiler to the previous commit

37fab22

And the revert the CI changes from the previous commit which are no longer needed.

smarter merged commit 38e87b0 into scala:master Aug 25, 2019

smarter deleted the upgrade/scalac-2.13 branch August 25, 2019 15:16

anatoliykmetyuk added this to the 0.18 Tech Preview milestone Aug 28, 2019

nicolasstucki mentioned this pull request Sep 2, 2019

Bad symbolic reference to a Scala 2 symbol #7157

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to the 2.13 standard library #7019

Switch to the 2.13 standard library #7019

smarter commented Aug 8, 2019 •

edited by bishabosha

Loading

dottybot left a comment

smarter commented Aug 8, 2019

smarter commented Aug 8, 2019

smarter commented Aug 8, 2019

bishabosha commented Aug 9, 2019

milessabin commented Aug 9, 2019

smarter commented Aug 9, 2019

milessabin commented Aug 9, 2019

milessabin commented Aug 9, 2019

smarter left a comment

smarter commented Aug 9, 2019

bishabosha commented Aug 9, 2019

milessabin commented Aug 14, 2019

smarter commented Aug 14, 2019

smarter commented Aug 14, 2019

bishabosha commented Aug 25, 2019

Switch to the 2.13 standard library #7019

Switch to the 2.13 standard library #7019

Conversation

smarter commented Aug 8, 2019 • edited by bishabosha Loading

dottybot left a comment

Choose a reason for hiding this comment

Commit Messages

smarter commented Aug 8, 2019

smarter commented Aug 8, 2019

smarter commented Aug 8, 2019

bishabosha commented Aug 9, 2019

milessabin commented Aug 9, 2019

smarter commented Aug 9, 2019

milessabin commented Aug 9, 2019

milessabin commented Aug 9, 2019

smarter left a comment

Choose a reason for hiding this comment

smarter commented Aug 9, 2019

bishabosha commented Aug 9, 2019

milessabin commented Aug 14, 2019

smarter commented Aug 14, 2019

smarter commented Aug 14, 2019

bishabosha commented Aug 25, 2019

smarter commented Aug 8, 2019 •

edited by bishabosha

Loading