Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port Scalameta to Scala Native #772

Closed
DavidDudson opened this issue Mar 29, 2017 · 25 comments
Closed

Port Scalameta to Scala Native #772

DavidDudson opened this issue Mar 29, 2017 · 25 comments
Labels

Comments

@DavidDudson
Copy link
Member

@DavidDudson DavidDudson commented Mar 29, 2017

Given that SJS support was just added, It seems like SN support should be somewhat trivial now that we are not platform specific. I could be wrong, so lets see what actually needs to be done.

Here are the scalameta dependencies

  • scalatest
  • scalapb
  • utest
  • sourcecode
  • fastparse

Further tasks:

  • Compile testsNative
  • Link testsNative
  • Remove the s/CLASS/CLAZZ/g workaround (scala-native/scala-native#1155)
  • Successfully run testsNative
  • Configure Travis to test native
@DavidDudson DavidDudson added the Build label Mar 29, 2017
@DavidDudson
Copy link
Member Author

@DavidDudson DavidDudson commented Mar 29, 2017

@densh I'm sure you can answer a lot of these questions I have.

@olafurpg
Copy link
Member

@olafurpg olafurpg commented Mar 29, 2017

This was one big motivation for porting to Scala.js/crossProject. I already tested the waters by trying to build scalafmt against scala-native, which resulted in these linking errors scala-native/scala-native#557

@xeno-by xeno-by changed the title Investigate crossbuilding with ScalaNative Port scala.meta to Scala Native Mar 30, 2017
@densh
Copy link
Member

@densh densh commented Apr 20, 2017

@olafurpg Most of the linking errors from that list are done now, we should retry and see if there is anything missing still.

@olafurpg
Copy link
Member

@olafurpg olafurpg commented Apr 22, 2017

@densh I just opened scalameta/scalafmt#898 porting scalafmt-core to Scala.js. We can work from that branch to try build scalafmt with native, I don't suspect we won't hit on any linking errors in core, but maybe we need more work in scalafmt-cli.

I will try to look into porting to native tomorrow.

@olafurpg
Copy link
Member

@olafurpg olafurpg commented Apr 27, 2017

I am super excited to confirm that scalafmt links with scala-native 0.2!

scalafmt-native

Getting it to work only required removing a single Enumeration in the scalameta parser since it seems those use runtime reflection. The rest was simple build changes.

@xeno-by
Copy link
Member

@xeno-by xeno-by commented Apr 27, 2017

@olafurpg
Copy link
Member

@olafurpg olafurpg commented Apr 27, 2017

(Moved to description)

@xeno-by
Copy link
Member

@xeno-by xeno-by commented Apr 27, 2017

@densh Can you comment on what it would take to support these libraries?

@densh
Copy link
Member

@densh densh commented Apr 28, 2017

I have no data on scalatest as they have rather complicated build that's not easily migratable to native.

utest/sourcecode/fastparse seem to have enough APIs implemented in 0.2 already, it's just a matter of cross-publishing it, build changes are trivial. We can't run the tests automatically from the sbt yet due to scala-native/scala-native#339, but @Duhemm might have time to look at it in 0.3 milestone.

Scalacheck should be either working already or require minimum changes from us to support it.

@olafurpg
Copy link
Member

@olafurpg olafurpg commented Aug 13, 2017

Fastparse 0.4.4 is out now with native support. Only remaining dependency is ScalaTest. Our options are to either

  • don't run tests in native projects (booo!)
  • write a thin FunSuite wrapper using utests.

I lean towards the second option. utest assert macro error messages are IME better than scalatest macro assert errors so I wouldn't mind switching to utest. Once lihaoyi/utest#113 is merged then utest will also have pretty console reports.

@ShaneDelmore
Copy link
Contributor

@ShaneDelmore ShaneDelmore commented Aug 13, 2017

👍 for writing a FunSuite wrapper for uTest.

@xeno-by
Copy link
Member

@xeno-by xeno-by commented Aug 14, 2017

👍

@olafurpg olafurpg changed the title Port scala.meta to Scala Native Port scalameta to Scala Native Aug 18, 2017
@xeno-by xeno-by changed the title Port scalameta to Scala Native Port Scalameta to Scala Native Jan 13, 2018
@xeno-by
Copy link
Member

@xeno-by xeno-by commented Jan 23, 2018

I tried crosscompiling a small part of Scalameta to Native, and it worked pretty well, except for a minor inconvenienced described in #1228 (comment). All in all, this looks doable, and I'm looking forward to a native build of Scalatest to commence a full-scale migration.

@olafurpg
Copy link
Member

@olafurpg olafurpg commented Jan 23, 2018

scalatest 3.2 SNAP release is due out this week with native support! https://gitter.im/scalatest/scalatest?at=5a660edc517037a212dc18ee

@xeno-by
Copy link
Member

@xeno-by xeno-by commented Jan 23, 2018

\o/ Thank you, @bvenners!

@densh
Copy link
Member

@densh densh commented Jan 29, 2018

Issue with artifact names has been fixed by @sjrd (portable-scala/sbt-crossproject#81) and is going to be part of the next release of sbt-crossproject.

@xeno-by
Copy link
Member

@xeno-by xeno-by commented Jan 29, 2018

@densh Thanks for the great news! Is there a gitter channel or something where you guys announce new releases?

@sjrd
Copy link
Contributor

@sjrd sjrd commented Jan 29, 2018

I announced the release on Twitter a few hours ago ;)

@xeno-by
Copy link
Member

@xeno-by xeno-by commented Feb 1, 2018

I started working on crosscompilation to native in https://github.com/xeno-by/scalameta/tree/topic/native. Here's a problem that I ran into:

> metapNative/run /Users/eburmako/Projects/scalameta/sandbox/META-INF/semanticdb/Library.semanticdb
[info] Compiling 1 protobuf files to /Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/.native/target/scala-2.11/src_managed/main
[info] Compiling schema /Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/semanticdb3.proto
protoc-jar: protoc version: 340, detected platform: mac os x/x86_64
protoc-jar: executing: [/var/folders/pg/v06175sd2_qb9jbg7c28xr040000gn/T/protocjar4860536607164199621/bin/protoc.exe, --plugin=protoc-gen-scala=/var/folders/pg/v06175sd2_qb9jbg7c28xr040000gn/T/protocbridge5139572597291050554, --scala_out=flat_package,grpc:/Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/.native/target/scala-2.11/src_managed/main, -I/Users/eburmako/Projects/scalameta/semanticdb/semanticdb3, -I/Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/.native/target/protobuf_external, /Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/semanticdb3.proto]
[info] Compiling protobuf
[info] Protoc target directory: /Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/.native/target/scala-2.11/src_managed/main
[info] Compiling 9 Scala sources to /Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/.native/target/scala-2.11/classes...
[info] Compiling 1 Scala source to /Users/eburmako/Projects/scalameta/semanticdb/metap/.native/target/scala-2.11/classes...
[info] Linking (1780 ms)
[error] cannot link: @scala.meta.internal.semanticdb3.SymbolInformation$Kind$CLASS$
[error] cannot link: @scala.meta.internal.semanticdb3.SymbolInformation$Kind$class$
[error] cannot link: @scala.meta.internal.semanticdb3.SymbolInformation$Kind$class$::$init$_scala.meta.internal.semanticdb3.SymbolInformation$Kind_unit
[error] unable to link
[error] (metapNative/compile:nativeLinkNIR) unable to link
[error] Total time: 14 s, completed Feb 1, 2018 11:23:32 AM

@densh, can you comment? Is this a native bug? If yes, is there a workaround that would allow me to ignore it and make progress regardless?

upd. Renaming SymbolInformation.Kind.CLASS to SymbolInformation.Kind.CLAZZ successfully works around the problem (xeno-by@b7a0691). We would still like to use the name CLASS, not CLAZZ, so it would be great if you could look into that.

upd. Submitted a ticket at scala-native/scala-native#1155.

@xeno-by
Copy link
Member

@xeno-by xeno-by commented Feb 1, 2018

Here's another problem that I ran into. This one may be related to ScalaTest:

> testsNative/test
[info] Compiling 1 protobuf files to /Users/eburmako/Projects/scalameta/semanticdb/semanticdb3/.native/target/scala-2.11/src_managed/main
...
[info] Compiling 1 Scala source to /Users/eburmako/Projects/scalameta/tests/native/target/scala-2.11/test-classes...
[error] /Users/eburmako/Projects/scalameta/tests/native/target/scala-2.11/src_managed/nativetest/FrameworksMap.scala:4: type ErrorSuite is not a member of package <root>
[error]   override val tests = Map[String, AnyRef]("scala.meta.tests.contrib.StatExtractionTest" -> new _root_.scala.meta.tests.contrib.StatExtractionTest, ...
[error] one error found
[error] (testsNative/nativetest:compileIncremental) Compilation failed
[error] Total time: 129 s, completed Feb 1, 2018 11:42:59 AM

@bvenners, any idea what could be causing the issue?

upd. Moving the test suite from the root package to any other package successfully works around the problem (xeno-by@44f57a0).

@shadaj
Copy link

@shadaj shadaj commented Feb 1, 2018

Looks like https://github.com/scala-native/scala-native/blob/17cf3c557a10b63898b2fa4251b8a298177716df/sbt-scala-native/src/main/scala/scala/scalanative/sbtplugin/TestUtilities.scala#L38 is generating the tests map with _root_ prepended to every test class, which leads to the failure since _root_ can't be used to access values or classes in the root package.

@Duhemm would it be possible to generate the TestMain into the root package so that _root_ is not needed to reference test classes/modules?

@xeno-by
Copy link
Member

@xeno-by xeno-by commented Feb 2, 2018

I've just submitted a pull request with the first stab at crosscompiling Scalameta to Scala Native: #1243. I had to apply several workarounds, but I managed to get testsNative/test to green state. If you're interested, check out the pull request for details.

@xeno-by
Copy link
Member

@xeno-by xeno-by commented Feb 6, 2018

Here's another pull request that crosscompiles just semanticdb/: #1263, but without any workarounds and hacks.

@xeno-by
Copy link
Member

@xeno-by xeno-by commented Feb 7, 2018

Successfully merged #1263 and published a Scala Native build of semanticdb3. Already using it in Rsc: xeno-by/rsc@e1b8e2e.

@olafurpg
Copy link
Member

@olafurpg olafurpg commented May 11, 2018

Fixed in #1529, native support for scalameta + contrib modules has been merged into master 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.