New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port Scalameta to Scala Native #772

Closed
DavidDudson opened this Issue Mar 29, 2017 · 25 comments

Comments

Projects
None yet
7 participants
@DavidDudson
Member

DavidDudson commented Mar 29, 2017

Given that SJS support was just added, It seems like SN support should be somewhat trivial now that we are not platform specific. I could be wrong, so lets see what actually needs to be done.

Here are the scalameta dependencies

  • scalatest
  • scalapb
  • utest
  • sourcecode
  • fastparse

Further tasks:

  • Compile testsNative
  • Link testsNative
  • Remove the s/CLASS/CLAZZ/g workaround (scala-native/scala-native#1155)
  • Successfully run testsNative
  • Configure Travis to test native

@DavidDudson DavidDudson added the Build label Mar 29, 2017

@DavidDudson

This comment has been minimized.

Show comment
Hide comment
@DavidDudson

DavidDudson Mar 29, 2017

Member

@densh I'm sure you can answer a lot of these questions I have.

Member

DavidDudson commented Mar 29, 2017

@densh I'm sure you can answer a lot of these questions I have.

@olafurpg

This comment has been minimized.

Show comment
Hide comment
@olafurpg

olafurpg Mar 29, 2017

Member

This was one big motivation for porting to Scala.js/crossProject. I already tested the waters by trying to build scalafmt against scala-native, which resulted in these linking errors scala-native/scala-native#557

Member

olafurpg commented Mar 29, 2017

This was one big motivation for porting to Scala.js/crossProject. I already tested the waters by trying to build scalafmt against scala-native, which resulted in these linking errors scala-native/scala-native#557

@xeno-by xeno-by changed the title from Investigate crossbuilding with ScalaNative to Port scala.meta to Scala Native Mar 30, 2017

@densh

This comment has been minimized.

Show comment
Hide comment
@densh

densh Apr 20, 2017

Member

@olafurpg Most of the linking errors from that list are done now, we should retry and see if there is anything missing still.

Member

densh commented Apr 20, 2017

@olafurpg Most of the linking errors from that list are done now, we should retry and see if there is anything missing still.

@olafurpg

This comment has been minimized.

Show comment
Hide comment
@olafurpg

olafurpg Apr 22, 2017

Member

@densh I just opened scalameta/scalafmt#898 porting scalafmt-core to Scala.js. We can work from that branch to try build scalafmt with native, I don't suspect we won't hit on any linking errors in core, but maybe we need more work in scalafmt-cli.

I will try to look into porting to native tomorrow.

Member

olafurpg commented Apr 22, 2017

@densh I just opened scalameta/scalafmt#898 porting scalafmt-core to Scala.js. We can work from that branch to try build scalafmt with native, I don't suspect we won't hit on any linking errors in core, but maybe we need more work in scalafmt-cli.

I will try to look into porting to native tomorrow.

@olafurpg

This comment has been minimized.

Show comment
Hide comment
@olafurpg

olafurpg Apr 27, 2017

Member

I am super excited to confirm that scalafmt links with scala-native 0.2!

scalafmt-native

Getting it to work only required removing a single Enumeration in the scalameta parser since it seems those use runtime reflection. The rest was simple build changes.

Member

olafurpg commented Apr 27, 2017

I am super excited to confirm that scalafmt links with scala-native 0.2!

scalafmt-native

Getting it to work only required removing a single Enumeration in the scalameta parser since it seems those use runtime reflection. The rest was simple build changes.

@xeno-by

This comment has been minimized.

Show comment
Hide comment
@xeno-by

xeno-by Apr 27, 2017

Member
Member

xeno-by commented Apr 27, 2017

@olafurpg

This comment has been minimized.

Show comment
Hide comment
@olafurpg

olafurpg Apr 27, 2017

Member

(Moved to description)

Member

olafurpg commented Apr 27, 2017

(Moved to description)

@xeno-by

This comment has been minimized.

Show comment
Hide comment
@xeno-by

xeno-by Apr 27, 2017

Member

@densh Can you comment on what it would take to support these libraries?

Member

xeno-by commented Apr 27, 2017

@densh Can you comment on what it would take to support these libraries?

@densh

This comment has been minimized.

Show comment
Hide comment
@densh

densh Apr 28, 2017

Member

I have no data on scalatest as they have rather complicated build that's not easily migratable to native.

utest/sourcecode/fastparse seem to have enough APIs implemented in 0.2 already, it's just a matter of cross-publishing it, build changes are trivial. We can't run the tests automatically from the sbt yet due to scala-native/scala-native#339, but @Duhemm might have time to look at it in 0.3 milestone.

Scalacheck should be either working already or require minimum changes from us to support it.

Member

densh commented Apr 28, 2017

I have no data on scalatest as they have rather complicated build that's not easily migratable to native.

utest/sourcecode/fastparse seem to have enough APIs implemented in 0.2 already, it's just a matter of cross-publishing it, build changes are trivial. We can't run the tests automatically from the sbt yet due to scala-native/scala-native#339, but @Duhemm might have time to look at it in 0.3 milestone.

Scalacheck should be either working already or require minimum changes from us to support it.

@olafurpg

This comment has been minimized.

Show comment
Hide comment
@olafurpg

olafurpg Aug 13, 2017

Member

Fastparse 0.4.4 is out now with native support. Only remaining dependency is ScalaTest. Our options are to either

  • don't run tests in native projects (booo!)
  • write a thin FunSuite wrapper using utests.

I lean towards the second option. utest assert macro error messages are IME better than scalatest macro assert errors so I wouldn't mind switching to utest. Once lihaoyi/utest#113 is merged then utest will also have pretty console reports.

Member

olafurpg commented Aug 13, 2017

Fastparse 0.4.4 is out now with native support. Only remaining dependency is ScalaTest. Our options are to either

  • don't run tests in native projects (booo!)
  • write a thin FunSuite wrapper using utests.

I lean towards the second option. utest assert macro error messages are IME better than scalatest macro assert errors so I wouldn't mind switching to utest. Once lihaoyi/utest#113 is merged then utest will also have pretty console reports.

@ShaneDelmore

This comment has been minimized.

Show comment
Hide comment
@ShaneDelmore

ShaneDelmore Aug 13, 2017

Contributor

👍 for writing a FunSuite wrapper for uTest.

Contributor

ShaneDelmore commented Aug 13, 2017

👍 for writing a FunSuite wrapper for uTest.

@xeno-by

This comment has been minimized.

Show comment
Hide comment
@xeno-by

xeno-by Aug 14, 2017

Member

👍

Member

xeno-by commented Aug 14, 2017

👍

@olafurpg olafurpg changed the title from Port scala.meta to Scala Native to Port scalameta to Scala Native Aug 18, 2017

@xeno-by xeno-by changed the title from Port scalameta to Scala Native to Port Scalameta to Scala Native Jan 13, 2018

@xeno-by

This comment has been minimized.

Show comment
Hide comment
@xeno-by

xeno-by Jan 23, 2018

Member

I tried crosscompiling a small part of Scalameta to Native, and it worked pretty well, except for a minor inconvenienced described in #1228 (comment). All in all, this looks doable, and I'm looking forward to a native build of Scalatest to commence a full-scale migration.

Member

xeno-by commented Jan 23, 2018

I tried crosscompiling a small part of Scalameta to Native, and it worked pretty well, except for a minor inconvenienced described in #1228 (comment). All in all, this looks doable, and I'm looking forward to a native build of Scalatest to commence a full-scale migration.

@olafurpg

This comment has been minimized.

Show comment
Hide comment
@olafurpg

olafurpg Jan 23, 2018

Member

scalatest 3.2 SNAP release is due out this week with native support! https://gitter.im/scalatest/scalatest?at=5a660edc517037a212dc18ee

Member

olafurpg commented Jan 23, 2018

scalatest 3.2 SNAP release is due out this week with native support! https://gitter.im/scalatest/scalatest?at=5a660edc517037a212dc18ee

@xeno-by

This comment has been minimized.

Show comment
Hide comment
@xeno-by

xeno-by Jan 23, 2018

Member

\o/ Thank you, @bvenners!

Member

xeno-by commented Jan 23, 2018

\o/ Thank you, @bvenners!

@densh

This comment has been minimized.

Show comment
Hide comment
@densh

densh Jan 29, 2018

Member

Issue with artifact names has been fixed by @sjrd (portable-scala/sbt-crossproject#81) and is going to be part of the next release of sbt-crossproject.

Member

densh commented Jan 29, 2018

Issue with artifact names has been fixed by @sjrd (portable-scala/sbt-crossproject#81) and is going to be part of the next release of sbt-crossproject.

@xeno-by

This comment has been minimized.

Show comment
Hide comment
@xeno-by

xeno-by Jan 29, 2018

Member

@densh Thanks for the great news! Is there a gitter channel or something where you guys announce new releases?

Member

xeno-by commented Jan 29, 2018

@densh Thanks for the great news! Is there a gitter channel or something where you guys announce new releases?

@sjrd

This comment has been minimized.

Show comment
Hide comment
@sjrd

sjrd Jan 29, 2018

I announced the release on Twitter a few hours ago ;)

sjrd commented Jan 29, 2018

I announced the release on Twitter a few hours ago ;)

@xeno-by

This comment has been minimized.

Show comment
Hide comment
@xeno-by

xeno-by Feb 1, 2018

Member

I started working on crosscompilation to native in https://github.com/xeno-by/scalameta/tree/topic/native. Here's a problem that I ran into:

> metapNative/run /Users/eburmako/Projects/scalameta/sandbox/META-INF/semanticdb/Library.semanticdb
[info] Compiling 1 protobuf files to /Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/.native/target/scala-2.11/src_managed/main
[info] Compiling schema /Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/semanticdb3.proto
protoc-jar: protoc version: 340, detected platform: mac os x/x86_64
protoc-jar: executing: [/var/folders/pg/v06175sd2_qb9jbg7c28xr040000gn/T/protocjar4860536607164199621/bin/protoc.exe, --plugin=protoc-gen-scala=/var/folders/pg/v06175sd2_qb9jbg7c28xr040000gn/T/protocbridge5139572597291050554, --scala_out=flat_package,grpc:/Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/.native/target/scala-2.11/src_managed/main, -I/Users/eburmako/Projects/scalameta/semanticdb/semanticdb3, -I/Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/.native/target/protobuf_external, /Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/semanticdb3.proto]
[info] Compiling protobuf
[info] Protoc target directory: /Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/.native/target/scala-2.11/src_managed/main
[info] Compiling 9 Scala sources to /Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/.native/target/scala-2.11/classes...
[info] Compiling 1 Scala source to /Users/eburmako/Projects/scalameta/semanticdb/metap/.native/target/scala-2.11/classes...
[info] Linking (1780 ms)
[error] cannot link: @scala.meta.internal.semanticdb3.SymbolInformation$Kind$CLASS$
[error] cannot link: @scala.meta.internal.semanticdb3.SymbolInformation$Kind$class$
[error] cannot link: @scala.meta.internal.semanticdb3.SymbolInformation$Kind$class$::$init$_scala.meta.internal.semanticdb3.SymbolInformation$Kind_unit
[error] unable to link
[error] (metapNative/compile:nativeLinkNIR) unable to link
[error] Total time: 14 s, completed Feb 1, 2018 11:23:32 AM

@densh, can you comment? Is this a native bug? If yes, is there a workaround that would allow me to ignore it and make progress regardless?

upd. Renaming SymbolInformation.Kind.CLASS to SymbolInformation.Kind.CLAZZ successfully works around the problem (xeno-by@b7a0691). We would still like to use the name CLASS, not CLAZZ, so it would be great if you could look into that.

upd. Submitted a ticket at scala-native/scala-native#1155.

Member

xeno-by commented Feb 1, 2018

I started working on crosscompilation to native in https://github.com/xeno-by/scalameta/tree/topic/native. Here's a problem that I ran into:

> metapNative/run /Users/eburmako/Projects/scalameta/sandbox/META-INF/semanticdb/Library.semanticdb
[info] Compiling 1 protobuf files to /Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/.native/target/scala-2.11/src_managed/main
[info] Compiling schema /Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/semanticdb3.proto
protoc-jar: protoc version: 340, detected platform: mac os x/x86_64
protoc-jar: executing: [/var/folders/pg/v06175sd2_qb9jbg7c28xr040000gn/T/protocjar4860536607164199621/bin/protoc.exe, --plugin=protoc-gen-scala=/var/folders/pg/v06175sd2_qb9jbg7c28xr040000gn/T/protocbridge5139572597291050554, --scala_out=flat_package,grpc:/Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/.native/target/scala-2.11/src_managed/main, -I/Users/eburmako/Projects/scalameta/semanticdb/semanticdb3, -I/Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/.native/target/protobuf_external, /Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/semanticdb3.proto]
[info] Compiling protobuf
[info] Protoc target directory: /Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/.native/target/scala-2.11/src_managed/main
[info] Compiling 9 Scala sources to /Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/.native/target/scala-2.11/classes...
[info] Compiling 1 Scala source to /Users/eburmako/Projects/scalameta/semanticdb/metap/.native/target/scala-2.11/classes...
[info] Linking (1780 ms)
[error] cannot link: @scala.meta.internal.semanticdb3.SymbolInformation$Kind$CLASS$
[error] cannot link: @scala.meta.internal.semanticdb3.SymbolInformation$Kind$class$
[error] cannot link: @scala.meta.internal.semanticdb3.SymbolInformation$Kind$class$::$init$_scala.meta.internal.semanticdb3.SymbolInformation$Kind_unit
[error] unable to link
[error] (metapNative/compile:nativeLinkNIR) unable to link
[error] Total time: 14 s, completed Feb 1, 2018 11:23:32 AM

@densh, can you comment? Is this a native bug? If yes, is there a workaround that would allow me to ignore it and make progress regardless?

upd. Renaming SymbolInformation.Kind.CLASS to SymbolInformation.Kind.CLAZZ successfully works around the problem (xeno-by@b7a0691). We would still like to use the name CLASS, not CLAZZ, so it would be great if you could look into that.

upd. Submitted a ticket at scala-native/scala-native#1155.

@xeno-by

This comment has been minimized.

Show comment
Hide comment
@xeno-by

xeno-by Feb 1, 2018

Member

Here's another problem that I ran into. This one may be related to ScalaTest:

> testsNative/test
[info] Compiling 1 protobuf files to /Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/.native/target/scala-2.11/src_managed/main
...
[info] Compiling 1 Scala source to /Users/eburmako/Projects/scalameta/tests/native/target/scala-2.11/test-classes...
[error] /Users/eburmako/Projects/scalameta/tests/native/target/scala-2.11/src_managed/nativetest/FrameworksMap.scala:4: type ErrorSuite is not a member of package <root>
[error]   override val tests = Map[String, AnyRef]("scala.meta.tests.contrib.StatExtractionTest" -> new _root_.scala.meta.tests.contrib.StatExtractionTest, ...
[error] one error found
[error] (testsNative/nativetest:compileIncremental) Compilation failed
[error] Total time: 129 s, completed Feb 1, 2018 11:42:59 AM

@bvenners, any idea what could be causing the issue?

upd. Moving the test suite from the root package to any other package successfully works around the problem (xeno-by@44f57a0).

Member

xeno-by commented Feb 1, 2018

Here's another problem that I ran into. This one may be related to ScalaTest:

> testsNative/test
[info] Compiling 1 protobuf files to /Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/.native/target/scala-2.11/src_managed/main
...
[info] Compiling 1 Scala source to /Users/eburmako/Projects/scalameta/tests/native/target/scala-2.11/test-classes...
[error] /Users/eburmako/Projects/scalameta/tests/native/target/scala-2.11/src_managed/nativetest/FrameworksMap.scala:4: type ErrorSuite is not a member of package <root>
[error]   override val tests = Map[String, AnyRef]("scala.meta.tests.contrib.StatExtractionTest" -> new _root_.scala.meta.tests.contrib.StatExtractionTest, ...
[error] one error found
[error] (testsNative/nativetest:compileIncremental) Compilation failed
[error] Total time: 129 s, completed Feb 1, 2018 11:42:59 AM

@bvenners, any idea what could be causing the issue?

upd. Moving the test suite from the root package to any other package successfully works around the problem (xeno-by@44f57a0).

@shadaj

This comment has been minimized.

Show comment
Hide comment
@shadaj

shadaj Feb 1, 2018

Looks like https://github.com/scala-native/scala-native/blob/17cf3c557a10b63898b2fa4251b8a298177716df/sbt-scala-native/src/main/scala/scala/scalanative/sbtplugin/TestUtilities.scala#L38 is generating the tests map with _root_ prepended to every test class, which leads to the failure since _root_ can't be used to access values or classes in the root package.

@Duhemm would it be possible to generate the TestMain into the root package so that _root_ is not needed to reference test classes/modules?

shadaj commented Feb 1, 2018

Looks like https://github.com/scala-native/scala-native/blob/17cf3c557a10b63898b2fa4251b8a298177716df/sbt-scala-native/src/main/scala/scala/scalanative/sbtplugin/TestUtilities.scala#L38 is generating the tests map with _root_ prepended to every test class, which leads to the failure since _root_ can't be used to access values or classes in the root package.

@Duhemm would it be possible to generate the TestMain into the root package so that _root_ is not needed to reference test classes/modules?

@xeno-by

This comment has been minimized.

Show comment
Hide comment
@xeno-by

xeno-by Feb 2, 2018

Member

I've just submitted a pull request with the first stab at crosscompiling Scalameta to Scala Native: #1243. I had to apply several workarounds, but I managed to get testsNative/test to green state. If you're interested, check out the pull request for details.

Member

xeno-by commented Feb 2, 2018

I've just submitted a pull request with the first stab at crosscompiling Scalameta to Scala Native: #1243. I had to apply several workarounds, but I managed to get testsNative/test to green state. If you're interested, check out the pull request for details.

@xeno-by

This comment has been minimized.

Show comment
Hide comment
@xeno-by

xeno-by Feb 6, 2018

Member

Here's another pull request that crosscompiles just semanticdb/: #1263, but without any workarounds and hacks.

Member

xeno-by commented Feb 6, 2018

Here's another pull request that crosscompiles just semanticdb/: #1263, but without any workarounds and hacks.

@xeno-by

This comment has been minimized.

Show comment
Hide comment
@xeno-by

xeno-by Feb 7, 2018

Member

Successfully merged #1263 and published a Scala Native build of semanticdb3. Already using it in Rsc: xeno-by/rsc@e1b8e2e.

Member

xeno-by commented Feb 7, 2018

Successfully merged #1263 and published a Scala Native build of semanticdb3. Already using it in Rsc: xeno-by/rsc@e1b8e2e.

@olafurpg

This comment has been minimized.

Show comment
Hide comment
@olafurpg

olafurpg May 11, 2018

Member

Fixed in #1529, native support for scalameta + contrib modules has been merged into master 🚀

Member

olafurpg commented May 11, 2018

Fixed in #1529, native support for scalameta + contrib modules has been merged into master 🚀

@olafurpg olafurpg closed this May 11, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment