Add Scala 2.12 support #1877

errikos · 2020-08-17T10:49:38Z

Summary

Add Scala 2.12 support for Scala Native. You can consult the Scala 2.12 changelog here. This PR takes into account all the changes and adapts them for Scala Native.

Lambdas

In order to keep the NIR changes to a minimum, we continue generating a FunctionN class with an apply method for each lambda. All the lambda captures are stored inside the class instance and initialised via its constructor. The apply method calls the generated $anonfun method, also extracting and forwarding the captures.

NIR compatibility

Scala 2.12 now compiles trait methods as default methods in Java interfaces. This means that the NIR compiled with 2.12 is not compatible with that compiled with 2.11.

Scala 2.11 cross-compilation

In order to keep the project compilable with Scala 2.11, a scala.scalanative.nscplugin.NirCompat component is introduced in nscplugin. This allows the compiler plugin to function with both Scala 2.11 and 2.12.

In addition, some tests need to know under which Scala version they are running. We use sbt-buildinfo to provide Scala version information to the projects that need them (mainly for unit-tests and tools tests).

sjrd · 2020-08-17T11:32:08Z

For the issue with Arrays, I think the underlying issue is that you're not handling value classes correctly in genFunction. The problem is that you need to recover value classes in parameters and result types, and box/unbox them appropriately. You'll have to replicate the uses of enteringPhase(currentRun.posterasurePhase) { ... } and handling of ErasedValueTypes that we do in Scala.js. Some pointers:

ensureBoxed and fromAny, which deal with the ErasedValueTypes
one example of where we use enteringPhase to compute the correct tpes to give to ensureBoxed and fromAny

tools/src/main/scala/scala/scalanative/linker/Reach.scala

nscplugin/src/main/scala/scala/scalanative/nscplugin/NirGenExpr.scala

sandbox/Test.scala

ekrich · 2020-08-17T22:15:56Z

scalalib/overrides-2.12/scala/App.scala

@@ -0,0 +1,77 @@
+/*


Are the overrides just standard for 2.12 even in the Scala repo?

Hmm, what do you mean?

I am just asking a general question about why these are here. They are not changed for Scala Native are they?

Oh yeah, we leave them as is. Basically, where applicable, I applied the changelog from the overrides-2.11 to the 2.12 files.

sjrd

This is a first review. I still need to really understand what's happening now in the linker for default methods.

.travis.yml

build.sbt

unit-tests/src/test/scala/java/util/ArrayDequeSuite.scala

unit-tests/src/test/scala/scala/AsInstanceOfSuite.scala

unit-tests/src/test/scala/tests/SuiteSuite.scala

util/src/main/scala/scala/scalanative/util/Platform.scala

errikos

The changes in commit 9f7d2fc were introduced because IntelliJ IDEA treats inline as a keyword, thus the whole files are red and impossible to work with.

nir/src/main/scala/scala/scalanative/nir/Attrs.scala

errikos · 2020-08-22T07:23:45Z

nir/src/main/scala/scala/scalanative/nir/Versions.scala

-  final val compat: Int   = 4 // a.k.a. MAJOR version
-  final val revision: Int = 7 // a.k.a. MINOR version
+  final val compat: Int   = 5 // a.k.a. MAJOR version
+  final val revision: Int = 8 // a.k.a. MINOR version


We had increased the revision when we introduced reflection. We did not have a release in between, but I figured I would increase it again with compat, since this is a separate PR (i.e. commit hash).

Shouldn't be 5.0 instead of 5.8 ?
How does semantic versioning work here?

It should definitely be 5.0 if we are actually breaking backward binary compat. But if we are doing so, we must make that into a separate commit (hence PR) that also removes all the deserialization hacks that were introduced so far.

Hmm, okay. So, are we changing the increment behaviour described in the docstring above?

Let's keep the system in place for now, hence leave 5.8.

nir/src/main/scala/scala/scalanative/nir/Types.scala

ishubham-k · 2020-09-22T16:15:25Z

Hi. I was wondering if this pr is been under active review. Is there anything pending that community can take a stab at ?

WojciechMazur · 2020-09-22T16:20:39Z

@ikamthania Hey, it's still under review. I hope we can merge it (as well as other remaining 0.4.0 blockers) and make a release by end of this month

ishubham-k · 2020-09-22T20:02:57Z

@ikamthania Hey, it's still under review. I hope we can merge it (as well as other remaining 0.4.0 blockers) and make a release by end of this month

Thanks for update. No issues. These things take time understandably.

sjrd

I finally dedicated enough continuous time to properly review this.

This looks very good. I only minor comments here and there. I'm looking forward to merging it :)

build.sbt

sjrd · 2020-10-01T11:47:02Z

tools/src/main/scala/scala/scalanative/interflow/Opt.scala

@@ -86,7 +86,7 @@ trait Opt { self: Interflow =>
    val retty = rets match {
      case Seq()   => Type.Nothing
      case Seq(ty) => ty
-      case tys     => Sub.lub(tys)
+      case tys     => Sub.lub(tys, Type.Ref(Global.Top("java.lang.Object")))


This is not resolve. We need to pass what used to be origRetty as the second argument to Sub.lub on this line (moving the computation of origRetty above val retty = ....

Suggested change

case tys => Sub.lub(tys, Type.Ref(Global.Top("java.lang.Object")))

case tys => Sub.lub(tys, origRetty)

Then we can remove resRetty and use retty on the last line of the method instead.

The current changes are sound, but they removed the type specialization entirely, instead of taking advantage of what Sub.lub can do for us.

tools/src/main/scala/scala/scalanative/linker/ClassLoader.scala

tools/src/main/scala/scala/scalanative/linker/Reach.scala

sjrd · 2020-10-02T10:08:48Z

This will also need to be rebased on master to sit on top of #1912.

errikos · 2020-10-02T17:18:34Z

There seems to be a test failing in tools due to recent changes, I will look into it.

WojciechMazur · 2020-10-02T19:45:28Z

There seems to be a test failing in tools due to recent changes, I will look into it.

From what I can see there is problem with mangled name of closure, exactly it's order of it's arguments:
expected: D6loop$1iL16java.lang.StringzL16java.lang.StringEPT6xyz.B$
actual: D6loop$1zL16java.lang.StringiL16java.lang.StringEPT6xyz.B$

And here's actual method:

package xyz
object B extends A {
def encodeLoop(arg1: Int, arg2: String): String = {
  println("init_logic")
  val bool: Boolean = false
  def loop(): String = {
    if (bool) "asd" else arg2 * arg1
  }
  loop()
}
}

I'm sure that's not a problem with our mangling logic (cause it passes other tests)
I think that order in which Scala compiler creates mangled names for closures have changed in 2.12. Since our signatures/mangled names are based on Scala compiler output we won't be able to change it.

I would suggest to change test definition to expect method signature in both orders on arguments. I can provide patch resolving this issue tomorrow.

errikos · 2020-10-06T20:00:21Z

This should be ready for another pass @sjrd. I think that all the comments are resolved and the CI seems to be green!

errikos · 2020-10-06T20:12:33Z

nscplugin/src/main/scala/scala/scalanative/nscplugin/NirPrimitives.scala

+      import scala.tools.nsc.settings._
+      ScalaVersion.current match {
+        case SpecificScalaVersion(2, 11, _, _) =>
+          HashMethods.foreach(addPrimitive(_, HASH))


I was not able to use the sbt-buildinfo plugin here to condition for the Scala version, because it resulted to an infinite recursion in sbt when resolving the projects that depend on nscplugin (tried to enable sbt-buildinfo for nscplugin).

I went with nsc.settings.ScalaVersion instead. I don't know if the latter has any version guarantees.

sjrd

Ship it!

Thank you @errikos for this huge work!

The Scala 2.12 changelog can be found at https://www.scala-lang.org/news/2.12.0/ This PR takes into account all the changes and adapts them for Scala Native. --- Lambdas In order to keep the NIR changes to a minimum, we continue generating a `FunctionN` class with an `apply` method for each lambda. All the lambda captures are stored inside the class instance and initialised via its constructor. The `apply` method calls the generated `$anonfun` method, also extracting and forwarding the captures. --- NIR changes Scala 2.12 now compiles trait methods as default methods in Java interfaces. This means that the NIR was enhanced to support default methods. The biggest impact of that is found in the reachability analysis (`Reach.scala`). --- Scala 2.11 cross-compilation In order to keep the project compilable with Scala 2.11, a `scala.scalanative.nscplugin.NirCompat` component is introduced in `nscplugin`. This allows the compiler plugin to function with both Scala 2.11 and 2.12. In addition, some tests need to know under which Scala version they are running. We use `sbt-buildinfo` to provide Scala version information to the projects that need them (mainly for `unit-tests` and `tools` tests). --- Co-authored-by: Denys Shabalin <denys.shabalin@epfl.ch>

This reverts commit 6f57cda.

The Scala 2.12 changelog can be found at https://www.scala-lang.org/news/2.12.0/ This PR takes into account all the changes and adapts them for Scala Native. --- Lambdas In order to keep the NIR changes to a minimum, we continue generating a `FunctionN` class with an `apply` method for each lambda. All the lambda captures are stored inside the class instance and initialised via its constructor. The `apply` method calls the generated `$anonfun` method, also extracting and forwarding the captures. --- NIR changes Scala 2.12 now compiles trait methods as default methods in Java interfaces. This means that the NIR was enhanced to support default methods. The biggest impact of that is found in the reachability analysis (`Reach.scala`). --- Scala 2.11 cross-compilation In order to keep the project compilable with Scala 2.11, a `scala.scalanative.nscplugin.NirCompat` component is introduced in `nscplugin`. This allows the compiler plugin to function with both Scala 2.11 and 2.12. In addition, some tests need to know under which Scala version they are running. We use `sbt-buildinfo` to provide Scala version information to the projects that need them (mainly for `unit-tests` and `tools` tests). --- Co-authored-by: Denys Shabalin <denys.shabalin@epfl.ch>

lolgab suggested changes Aug 17, 2020

View reviewed changes

tools/src/main/scala/scala/scalanative/linker/Reach.scala Outdated Show resolved Hide resolved

ekrich reviewed Aug 17, 2020

View reviewed changes

errikos force-pushed the topic/2.12.x-support branch 2 times, most recently from 3d1e5ce to 237e427 Compare August 19, 2020 13:28

sjrd requested changes Aug 20, 2020

View reviewed changes

errikos force-pushed the topic/2.12.x-support branch from 5b9a74f to 9717b6c Compare August 20, 2020 19:34

errikos commented Aug 21, 2020

View reviewed changes

ekrich reviewed Aug 21, 2020

View reviewed changes

nir/src/main/scala/scala/scalanative/nir/Attrs.scala Outdated Show resolved Hide resolved

errikos commented Aug 22, 2020

View reviewed changes

errikos requested a review from sjrd September 14, 2020 16:50

WojciechMazur reviewed Sep 18, 2020

View reviewed changes

nir/src/main/scala/scala/scalanative/nir/Types.scala Outdated Show resolved Hide resolved

sjrd requested changes Oct 1, 2020

View reviewed changes

WojciechMazur mentioned this pull request Oct 2, 2020

Update NIR version to 5.8 #1912

Merged

densh and others added 12 commits October 2, 2020 16:34

Scala 2.12 support prototype

d9259cb

Fix javalib deprecations

e0d8502

Fix nativelib deprecations

858be5b

Fix comparison bug in Reach

d41ae97

Rebase nscplugin

dd614d6

Avoid unrelated comparison

7d839c1

Add note about ValTree and ContTree not being final

ca4d092

Fix param types in lambda generation

5bb5521

Upgrade to 2.12.12

6a70c38

Apply scalafmt

f8300ba

Bring TraitReachabilitySuite up-to-date with 2.12 traits

3ad3ee2

Rework and explain apply parameters unbox/cast

154e415

errikos and others added 9 commits October 2, 2020 16:34

Make scripted-tests cross-compilable

09f81e4

Resolve comments in NirGenExpr

bbb3920

Resolve comments in InterFlow

a949f89

Test tools with Scala versions 2.11 and 2.12

a14dc5b

Test runtime with Scala versions 2.11 and 2.12

6b06882

Rename doInline to inlineHint in Attrs

00966d2

Get rid of libScalaVersion

cd679e2

Rebase to master

2399336

Resolve comments, phase 1

4d05ca7

errikos force-pushed the topic/2.12.x-support branch from b080d1a to 4d05ca7 Compare October 2, 2020 15:53

Fix junitAsyncJVM, junitTestOutputsJVM cross-Scala version

69451a4

errikos force-pushed the topic/2.12.x-support branch from ca6b029 to 69451a4 Compare October 3, 2020 00:42

Fix private methods mangling test in tools for 2.12

4653520

WojciechMazur mentioned this pull request Oct 5, 2020

Add Scala 2.13.x support #1916

Merged

2 tasks

errikos added 3 commits October 6, 2020 16:35

Fix retty computation in Opt

e5acafb

Revert changes in 2.11 ScalaRunTime

a60929c

Bring back hash intrinsics for 2.11.x

fce1234

errikos requested a review from sjrd October 6, 2020 19:59

errikos commented Oct 6, 2020

View reviewed changes

sjrd approved these changes Oct 7, 2020

View reviewed changes

sjrd merged commit 12a305c into scala-native:master Oct 7, 2020

errikos deleted the topic/2.12.x-support branch October 8, 2020 17:27

vicopem added a commit to vicopem/scala-native that referenced this pull request Nov 19, 2020

Revert "Add Scala 2.12 support (scala-native#1877)"

2262a6c

This reverts commit 6f57cda.

vicopem added a commit to vicopem/scala-native that referenced this pull request Nov 19, 2020

Revert "Add Scala 2.12 support (scala-native#1877)"

94c67e3

This reverts commit 6f57cda.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Scala 2.12 support #1877

Add Scala 2.12 support #1877

errikos commented Aug 17, 2020 •

edited

Loading

sjrd commented Aug 17, 2020

ekrich Aug 17, 2020

errikos Aug 18, 2020

ekrich Aug 18, 2020

errikos Aug 18, 2020

sjrd left a comment

errikos left a comment

errikos Aug 22, 2020 •

edited

Loading

lolgab Aug 22, 2020 •

edited

Loading

sjrd Aug 22, 2020

errikos Aug 22, 2020

sjrd Oct 1, 2020

ishubham-k commented Sep 22, 2020

WojciechMazur commented Sep 22, 2020 •

edited

Loading

ishubham-k commented Sep 22, 2020

sjrd left a comment

sjrd Oct 1, 2020

sjrd commented Oct 2, 2020

errikos commented Oct 2, 2020

WojciechMazur commented Oct 2, 2020 •

edited

Loading

errikos commented Oct 6, 2020

errikos Oct 6, 2020

sjrd left a comment

	case tys => Sub.lub(tys, Type.Ref(Global.Top("java.lang.Object")))
	case tys => Sub.lub(tys, origRetty)

Add Scala 2.12 support #1877

Add Scala 2.12 support #1877

Conversation

errikos commented Aug 17, 2020 • edited Loading

Summary

Lambdas

NIR compatibility

Scala 2.11 cross-compilation

sjrd commented Aug 17, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjrd left a comment

Choose a reason for hiding this comment

errikos left a comment

Choose a reason for hiding this comment

errikos Aug 22, 2020 • edited Loading

Choose a reason for hiding this comment

lolgab Aug 22, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ishubham-k commented Sep 22, 2020

WojciechMazur commented Sep 22, 2020 • edited Loading

ishubham-k commented Sep 22, 2020

sjrd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjrd commented Oct 2, 2020

errikos commented Oct 2, 2020

WojciechMazur commented Oct 2, 2020 • edited Loading

errikos commented Oct 6, 2020

Choose a reason for hiding this comment

sjrd left a comment

Choose a reason for hiding this comment

errikos commented Aug 17, 2020 •

edited

Loading

errikos Aug 22, 2020 •

edited

Loading

lolgab Aug 22, 2020 •

edited

Loading

WojciechMazur commented Sep 22, 2020 •

edited

Loading

WojciechMazur commented Oct 2, 2020 •

edited

Loading