Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP Don't merge. Basic Scala.js compilation #1412

Closed
wants to merge 4,956 commits into from
Closed

Conversation

cquiroz
Copy link
Member

@cquiroz cquiroz commented Sep 19, 2017

Here is a first stab at getting http4s to run on scala.js. The final objective is to be able to run the http client on js but we are still several steps away from that

This PR updates the build to cross compile some of the subprojects including core, client and some dependencies

Some notes on the changes to the build

  • core and tests have been converted to full cross projects, This is needed to split the parts that contains references to classes not available in scala.js, in particular File Thus EntityEncoder/Decoder and Multipart have been split. This means the sources files have been moved from <project>/src to <project>/shared/src
  • testing, client, circe, jawn and parboiled2 have been converted to Pure cross compiled projects
  • A mock of log4s for scala.js have been added. It doesn't do anything at the moment but it lets us compile
  • Tests for client depend on java only libraries so the tests were moved to a project clientTesting

TODO:

[ ] Make the tests run on scala.js
[ ] Implement a scala.js based client
[ ] Example of a client on js

I'm posting this for comments but this certainly doesn't do anything useful at the moment. The main challenge is to come up with a client implementation on JS

build.sbt Outdated
// Ignore, tut is not supported in scala.js
sources in (Compile,doc) := Seq.empty,
// This is needed to support the TLS compiler and scala.js at the same time
// Remove the dependency on the scalajs-compiler
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused by what's going on here. Note that we reverted use of the TLS compiler.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh really, Are we on LBS now? If so I can drop this

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. We weren't using any advanced features, and the binary compatibility isn't quite yet as we thought it would be when we adopted it. I'm interested in going back at some point, but for now, it's LBS.

@@ -0,0 +1,10 @@
package org.log4s

case class Logger(name: String) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there no way to write to a console?

I still wonder whether the log4s project would be interested in cross-compilation.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we can write to console. This is only an interim to get things compiling

@@ -0,0 +1,91 @@
/** TODO fs2 port
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can drop this entire file.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is removed on master. Maybe a rebase is needed?

@@ -0,0 +1,77 @@
/** TODO fs2 port
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And drop this one, too.


def error(t: Throwable)(s: => String): Unit = {
t.printStackTrace()
println(s)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd probably flip these two lines.

println(s)

def debug(s: => String): Unit =
println(s)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These could be extremely verbose. We might want to have a gasp global threshold.

@rossabaker
Copy link
Member

See #1415 for something that will conflict with this. I took care to ensure we are publishing our fork of parboiled for scala.js.

@cquiroz
Copy link
Member Author

cquiroz commented Sep 22, 2017

@rossabaker. Thanks for including parboiled support on the fork

@cquiroz cquiroz force-pushed the scala.js branch 4 times, most recently from ba867fe to 3d9dcb8 Compare September 23, 2017 13:45
@cquiroz cquiroz force-pushed the scala.js branch 7 times, most recently from 849b8fc to 17409ce Compare October 4, 2017 16:05
@ngbinh
Copy link
Contributor

ngbinh commented Oct 9, 2017

this is awesome!

@oyvindberg
Copy link

Hey @cquiroz and everybody else :)

I have really been wanting this as well, so i took your branch and ran with it to see how far along i would get. I certainly hope you won't mind a helping hand in this effort.

So far I've rebased on master and (with a good pinch of hacks) gotten the tests to link and (mostly) run green.

My branch is located at https://github.com/oyvindberg/http4s/tree/scala.js-continue. Would you mind having a look, and update this PR as you see fitting?

Apologies in advance for the wall of text which follows as I have tried to summarize the current state of the port.

There are a bunch of issues, including some which might be of interest for jvm code.

Necessary snapshot dependencies to compile/run tests

Scala.js

Your Charset.aliases() patch is necessary, but will only be released in Scala.js 0.6.22. For that reason we need to use 0.6.22-SNAPSHOT. I built that myself for now.

Specs2

A rabbit hole. It turns out that scala-parser-combinators had a broken release for Scala.js. Specs2 (and our tests, transitively) needs it. This has been temporarily fixed. Scala.js support for Specs2 is relatively new, so I'm not sure if there is a build which both both supports Scala.js and has correct scala-parser-combinators version. For now i built it locally, but I'm sure we could nag to have 4.0.3 released.

Issues

Reflection

As noted in #956 (comment), there is one instance of reflection in the codebase. For now I kept the overall structure of everything, but inlined the result of running that code:

object  HttpHeaderParser {
  private val allParsers =  new util.concurrent.ConcurrentHashMap[CaseInsensitiveString, HeaderParser]

  addParser("ACCEPT-CHARSET".ci,            ACCEPT_CHARSET)
  addParser("ACCEPT".ci,                    ACCEPT)
  addParser("ACCEPT-ENCODING".ci,           ACCEPT_ENCODING)
  ...
  }

This obviously means that there is now a possibility of adding a header and forgetting to add it to the global mutable list. Better options should probably be explored.

Locales

In Scala.js we don't have access to Locale databases. That is mainly a problem because there are many usages of String.toLowerCase(Locale) for instance. That type signature is not really implementable without those databases. My proposed fix is to separate this into platform-specific code, and just use String.toLowerCase() and do best effort on the javascript side (not done yet).

We also have problems show up as a test failure:

[error]   x be consistent with equality for Straße and STRAẞE
[error]    false (file:1)

I have a feeling we will be stuck with whatever node/browser does here.

Unimplemented APIs in Scala.js

java.net.InetAddress and java.netInetSocketAddress

Given that they ultimately depend on native code, I strongly doubt that they are implementable in Scala.js.

For now they are implemented as simple objects and case classes on the Scala.js side, and it might not necessarily be a big issue. If we ever choose to implement a node server backend, say, we should strongly consider replacing these data types.

scala.concurrent.TrieMap

(Also noted in #956 )
This is not available, but used for Registry which backs MediaType. Hopefully a non-mutable approach can be found here too. Replaced it with mutable.HashMap to get the tests running.

URLEncoder.encode(String, String)

I imagine this should be relatively easy to add.

Platform differences: Double

[error]   x set a parameter with a float values
[error]    ?param=1.2000000476837158&param=2.0999999046325684 != ?param=1.2&param=2.1 (/home/olvind/pr/http4s/tests/js/target/scala-2.12/http4s-tests-test-fastopt.js:117838)
[error] Actual:   ?param=1.2000000476837158&param=2.0999999046325684
[error] Expected: ?param=1.2&param=2.1

This might be a bug waiting to happen on the JVM as well, though i haven't looked into it. Should be easily fixable for this particular use case.

Macro magic

TL;DR: there is a workaround.

I'll be the first to admit i don't understand how parboiled2 works. The most difficult part of this port was to figure out what was wrong when the Scala.js linker complained about missing references in macro generated code.

An example which causes such a linking error:

  private trait TransferCodingParser { self: PbParser =>
    def codingRule = rule {
      "chunked" ~ push(chunked) |
        "compress" ~ push(compress) |
        "deflate" ~ push(deflate) |
        "gzip" ~ push(gzip) |
        "identity" ~ push(identity)
    }
  }
Loong linking error
[info] Fast optimizing /home/olvind/pr/http4s/tests/js/target/scala-2.12/http4s-tests-test-fastopt.js
[error] Referring to non-existent method org.http4s.TransferCoding$$anon$3.H16()org.http4s.internal.parboiled2.Rule
[error]   called from org.http4s.parser.IpParser.rec$20(scala.Int,scala.Long,scala.Int,scala.Int)scala.Boolean
[error]   called from org.http4s.parser.SimpleHeaders$$anon$15.rec$20(scala.Int,scala.Long,scala.Int,scala.Int)scala.Boolean
[error]   called from org.http4s.parser.IpParser.IpV6Address()org.http4s.internal.parboiled2.Rule
[error]   called from org.http4s.parser.Http4sHeaderParser.IpV6Address()org.http4s.internal.parboiled2.Rule
[error]   called from org.http4s.parser.Rfc3986Parser.liftedTree24$1(scala.Int)scala.Boolean
[error]   called from org.http4s.parser.SimpleHeaders$$anon$1.liftedTree24$1(scala.Int)scala.Boolean
[error]   called from org.http4s.parser.Rfc3986Parser.wrapped$16()scala.Boolean
[error]   called from org.http4s.parser.SimpleHeaders$$anon$1.wrapped$16()scala.Boolean
[error]   called from org.http4s.parser.Rfc3986Parser.IpLiteral()org.http4s.internal.parboiled2.Rule
[error]   called from org.http4s.parser.SimpleHeaders$$anon$1.IpLiteral()org.http4s.internal.parboiled2.Rule
[error]   called from org.http4s.parser.SimpleHeaders$$anon$1.entry()org.http4s.internal.parboiled2.Rule
[error]   called from org.http4s.parser.Http4sHeaderParser.$$anonfun$parse$1(org.http4s.parser.Http4sHeaderParser)org.http4s.internal.parboiled2.Rule
[error]   called from org.http4s.parser.Http4sHeaderParser.parse()scala.util.Either
[error]   called from org.http4s.parser.SimpleHeaders.IF$undNONE$undMATCH(java.lang.String)scala.util.Either
[error]   called from org.http4s.parser.HttpHeaderParser$.IF$undNONE$undMATCH(java.lang.String)scala.util.Either
[error]   called from org.http4s.parser.HttpHeaderParser$.$$anonfun$new$22(java.lang.String)scala.util.Either
[error]   called from org.http4s.parser.HttpHeaderParser$.()
[error]   called from org.http4s.Header$Raw.parsed()org.http4s.Header
[error]   called from org.http4s.Headers.$$anonfun$$plus$plus$1(scala.collection.immutable.List,scala.collection.mutable.ListBuffer,org.http4s.Header)java.lang.Object
[error]   called from org.http4s.Headers.$$plus$plus(scala.collection.GenTraversableOnce,scala.collection.generic.CanBuildFrom)java.lang.Object
[error]   called from org.specs2.matcher.describe.OrderedCollectionDifferent.render(java.lang.String)java.lang.String
[error]   called from org.specs2.matcher.describe.OrderedCollectionDifferent.render()java.lang.String
[error]   called from org.specs2.matcher.describe.TryDifferent.render()java.lang.String
[error]   called from org.specs2.matcher.describe.ThrowableDifferentMessage.render()java.lang.String
[error]   called from org.specs2.matcher.describe.OptionIdentical.$$anonfun$render$1(org.specs2.matcher.describe.ComparisonResult)java.lang.String
[error]   called from org.specs2.matcher.describe.OptionIdentical.render()java.lang.String
[error]   called from org.specs2.matcher.describe.StackElementDifferent.$$anonfun$renderSourceLocation$1(org.specs2.matcher.describe.ComparisonResult)java.lang.String
[error]   called from org.specs2.matcher.describe.StackElementDifferent.renderSourceLocation()java.lang.String
[error]   called from org.specs2.matcher.describe.StackElementDifferent.render()java.lang.String
[error]   called from org.specs2.matcher.describe.ThrowableDifferentStackTrace.render()java.lang.String
[error]   called from org.specs2.matcher.EqualityMatcher.$$anonfun$apply$4(org.specs2.matcher.Expectable,org.specs2.matcher.describe.ComparisonResult)java.lang.String
[error]   called from org.specs2.matcher.EqualityMatcher.apply(org.specs2.matcher.Expectable)org.specs2.matcher.MatchResult
[error]   called from org.specs2.matcher.Expectable.applyMatcher(scala.Function0)org.specs2.matcher.MatchResult
[error]   called from org.specs2.matcher.MustThrownExpectationsCreation$$anon$4.applyMatcher(scala.Function0)org.specs2.matcher.MatchResult
[error]   called from org.specs2.matcher.MustExpectable.must$und$eq$eq(scala.Function0)org.specs2.matcher.MatchResult
[error]   called from org.http4s.ContentCodingSpec.$$anonfun$new$3(org.http4s.ContentCoding,org.http4s.ContentCoding)org.specs2.matcher.MatchResult
[error]   called from org.http4s.ContentCodingSpec.$$anonfun$new$2()org.specs2.scalacheck.ScalaCheckFunction2
[error]   called from org.http4s.ContentCodingSpec.$$anonfun$new$1()org.specs2.specification.core.Fragment
[error]   called from org.http4s.ContentCodingSpec.()
[error]   called from org.http4s.ContentCodingSpec.()
[error]   called from core module analyzer
[error] involving instantiated classes:
[error]   org.http4s.parser.SimpleHeaders$$anon$15
[error]   org.http4s.parser.RefererHeader$RefererParser
[error]   org.http4s.parser.LocationHeader$LocationParser
[error]   org.http4s.parser.SimpleHeaders$$anon$1
[error]   org.http4s.parser.SimpleHeaders$$anon$17
[error]   org.http4s.parser.SimpleHeaders$$anon$10
[error]   org.http4s.parser.SimpleHeaders$$anon$14
[error]   org.http4s.parser.AcceptCharsetHeader$AcceptCharsetParser
[error]   org.http4s.parser.SimpleHeaders$$anon$5
[error]   org.http4s.parser.WwwAuthenticateHeader$WWWAuthenticateParser
[error]   org.http4s.parser.AcceptEncodingHeader$AcceptEncodingParser
[error]   org.http4s.parser.ContentTypeHeader$ContentTypeParser
[error]   org.http4s.parser.SimpleHeaders$$anon$4
[error]   org.http4s.parser.StrictTransportSecurityHeader$StrictTransportSecurityParser
[error]   org.http4s.parser.SimpleHeaders$$anon$2
[error]   org.http4s.parser.RangeParser$$anon$1
[error]   org.http4s.parser.ZipkinHeader$$anon$5
[error]   org.http4s.parser.AcceptHeader$AcceptParser
[error]   org.http4s.parser.RangeParser$$anon$2
[error]   org.http4s.parser.SimpleHeaders$$anon$12
[error]   org.http4s.parser.ZipkinHeader$$anon$1
[error]   org.http4s.parser.AuthorizationHeader$AuthorizationParser
[error]   org.http4s.parser.CookieHeader$SetCookieParser
[error]   org.http4s.parser.SimpleHeaders$$anon$6
[error]   org.http4s.parser.ProxyAuthenticateHeader$ProxyAuthenticateParser
[error]   org.http4s.parser.SimpleHeaders$$anon$13
[error]   org.http4s.parser.ZipkinHeader$$anon$3
[error]   org.http4s.parser.RangeParser$AcceptRangesParser
[error]   org.http4s.parser.SimpleHeaders$$anon$7
[error]   org.http4s.parser.SimpleHeaders$$anon$11
[error]   org.http4s.parser.ZipkinHeader$$anon$2
[error]   org.http4s.parser.SimpleHeaders$$anon$3
[error]   org.http4s.parser.SimpleHeaders$$anon$8
[error]   org.http4s.parser.SimpleHeaders$$anon$16
[error]   org.http4s.parser.SimpleHeaders$$anon$9
[error]   org.http4s.parser.CacheControlHeader$CacheControlParser
[error]   org.http4s.parser.ZipkinHeader$$anon$4
[error]   org.http4s.parser.AcceptLanguageHeader$AcceptLanguageParser
[error]   org.http4s.parser.CookieHeader$CookieParser
[error]   org.http4s.parser.HttpHeaderParser$
[error]   org.http4s.Header$Raw
[error]   org.http4s.Headers
[error]   org.specs2.matcher.describe.ArrayDifference
[error]   org.specs2.matcher.describe.TryDifferent
[error]   org.specs2.matcher.describe.ThrowableDifferentMessage
[error]   org.specs2.matcher.describe.OptionIdentical
[error]   org.specs2.matcher.describe.StackElementDifferent
[error]   org.specs2.matcher.describe.ThrowableDifferentStackTrace
[error]   org.specs2.matcher.EqualityMatcher$$anon$1
[error]   org.specs2.matcher.EqualityMatcher
[error]   org.specs2.matcher.BeEqualTo
[error]   org.specs2.matcher.MustExpectationsCreation$$anon$2
[error]   org.specs2.matcher.Expectable$$anon$2
[error]   org.specs2.matcher.ExpectationsCreation$$anon$1
[error]   org.specs2.matcher.Expectable
[error]   org.specs2.matcher.MustThrownExpectationsCreation$$anon$4
[error]   org.http4s.ContentCodingSpec
[error] Referring to non-existent method org.http4s.TransferCoding$$anon$3.DecOctet()org.http4s.internal.parboiled2.Rule
[error]   called from org.http4s.parser.IpParser.rec$3(scala.Int,scala.Long,scala.Int,scala.Int)scala.Boolean
[error]   called from org.http4s.parser.SimpleHeaders$$anon$15.rec$3(scala.Int,scala.Long,scala.Int,scala.Int)scala.Boolean
[error]   called from org.http4s.parser.IpParser.wrapped$3()scala.Boolean
[error]   called from org.http4s.parser.SimpleHeaders$$anon$15.wrapped$3()scala.Boolean
[error]   called from org.http4s.parser.IpParser.IpV4Address()org.http4s.internal.parboiled2.Rule
[error]   called from org.http4s.parser.Http4sHeaderParser.IpV4Address()org.http4s.internal.parboiled2.Rule
[error]   called from org.http4s.parser.IpParser.wrapped$4()scala.Boolean
[error]   called from org.http4s.parser.SimpleHeaders$$anon$15.wrapped$4()scala.Boolean
[error]   called from org.http4s.parser.IpParser.LS32()org.http4s.internal.parboiled2.Rule
[error]   called from org.http4s.parser.Http4sHeaderParser.LS32()org.http4s.internal.parboiled2.Rule
[error]   called from org.http4s.parser.IpParser.liftedTree21$1(scala.Int)scala.Boolean
[error]   called from org.http4s.parser.SimpleHeaders$$anon$15.liftedTree21$1(scala.Int)scala.Boolean
[error]   called from org.http4s.parser.IpParser.wrapped$5()scala.Boolean
[error]   called from org.http4s.parser.SimpleHeaders$$anon$15.wrapped$5()scala.Boolean
[error]   called from org.http4s.parser.IpParser.IpV6Address()org.http4s.internal.parboiled2.Rule
[error]   called from org.http4s.parser.Http4sHeaderParser.IpV6Address()org.http4s.internal.parboiled2.Rule
[error]   called from org.http4s.parser.Rfc3986Parser.liftedTree24$1(scala.Int)scala.Boolean
[error]   called from org.http4s.parser.SimpleHeaders$$anon$1.liftedTree24$1(scala.Int)scala.Boolean
[error]   called from org.http4s.parser.Rfc3986Parser.wrapped$16()scala.Boolean
[error]   called from org.http4s.parser.SimpleHeaders$$anon$1.wrapped$16()scala.Boolean
[error]   called from org.http4s.parser.Rfc3986Parser.IpLiteral()org.http4s.internal.parboiled2.Rule
[error]   called from org.http4s.parser.SimpleHeaders$$anon$1.IpLiteral()org.http4s.internal.parboiled2.Rule
[error]   called from org.http4s.parser.SimpleHeaders$$anon$1.entry()org.http4s.internal.parboiled2.Rule
[error]   called from org.http4s.parser.Http4sHeaderParser.$$anonfun$parse$1(org.http4s.parser.Http4sHeaderParser)org.http4s.internal.parboiled2.Rule
[error]   called from org.http4s.parser.Http4sHeaderParser.parse()scala.util.Either
[error]   called from org.http4s.parser.SimpleHeaders.IF$undNONE$undMATCH(java.lang.String)scala.util.Either
[error]   called from org.http4s.parser.HttpHeaderParser$.IF$undNONE$undMATCH(java.lang.String)scala.util.Either
[error]   called from org.http4s.parser.HttpHeaderParser$.$$anonfun$new$22(java.lang.String)scala.util.Either
[error]   called from org.http4s.parser.HttpHeaderParser$.()
[error]   called from org.http4s.Header$Raw.parsed()org.http4s.Header
[error]   called from org.http4s.Headers.$$anonfun$$plus$plus$1(scala.collection.immutable.List,scala.collection.mutable.ListBuffer,org.http4s.Header)java.lang.Object
[error]   called from org.http4s.Headers.$$plus$plus(scala.collection.GenTraversableOnce,scala.collection.generic.CanBuildFrom)java.lang.Object
[error]   called from org.specs2.matcher.describe.OrderedCollectionDifferent.render(java.lang.String)java.lang.String
[error]   called from org.specs2.matcher.describe.OrderedCollectionDifferent.render()java.lang.String
[error]   called from org.specs2.matcher.describe.TryDifferent.render()java.lang.String
[error]   called from org.specs2.matcher.describe.ThrowableDifferentMessage.render()java.lang.String
[error]   called from org.specs2.matcher.describe.OptionIdentical.$$anonfun$render$1(org.specs2.matcher.describe.ComparisonResult)java.lang.String
[error]   called from org.specs2.matcher.describe.OptionIdentical.render()java.lang.String
[error]   called from org.specs2.matcher.describe.StackElementDifferent.$$anonfun$renderSourceLocation$1(org.specs2.matcher.describe.ComparisonResult)java.lang.String
[error]   called from org.specs2.matcher.describe.StackElementDifferent.renderSourceLocation()java.lang.String
[error]   called from org.specs2.matcher.describe.StackElementDifferent.render()java.lang.String
[error]   called from org.specs2.matcher.describe.ThrowableDifferentStackTrace.render()java.lang.String
[error]   called from org.specs2.matcher.EqualityMatcher.$$anonfun$apply$4(org.specs2.matcher.Expectable,org.specs2.matcher.describe.ComparisonResult)java.lang.String
[error]   called from org.specs2.matcher.EqualityMatcher.apply(org.specs2.matcher.Expectable)org.specs2.matcher.MatchResult
[error]   called from org.specs2.matcher.Expectable.applyMatcher(scala.Function0)org.specs2.matcher.MatchResult
[error]   called from org.specs2.matcher.MustThrownExpectationsCreation$$anon$4.applyMatcher(scala.Function0)org.specs2.matcher.MatchResult
[error]   called from org.specs2.matcher.MustExpectable.must$und$eq$eq(scala.Function0)org.specs2.matcher.MatchResult
[error]   called from org.http4s.ContentCodingSpec.$$anonfun$new$3(org.http4s.ContentCoding,org.http4s.ContentCoding)org.specs2.matcher.MatchResult
[error]   called from org.http4s.ContentCodingSpec.$$anonfun$new$2()org.specs2.scalacheck.ScalaCheckFunction2
[error]   called from org.http4s.ContentCodingSpec.$$anonfun$new$1()org.specs2.specification.core.Fragment
[error]   called from org.http4s.ContentCodingSpec.()
[error]   called from org.http4s.ContentCodingSpec.()
[error]   called from core module analyzer
[error] involving instantiated classes:
[error]   org.http4s.parser.SimpleHeaders$$anon$15
[error]   org.http4s.parser.SimpleHeaders$$anon$10
[error]   org.http4s.parser.SimpleHeaders$$anon$14
[error]   org.http4s.parser.AcceptCharsetHeader$AcceptCharsetParser
[error]   org.http4s.parser.SimpleHeaders$$anon$5
[error]   org.http4s.parser.RefererHeader$RefererParser
[error]   org.http4s.parser.WwwAuthenticateHeader$WWWAuthenticateParser
[error]   org.http4s.parser.AcceptEncodingHeader$AcceptEncodingParser
[error]   org.http4s.parser.ContentTypeHeader$ContentTypeParser
[error]   org.http4s.parser.SimpleHeaders$$anon$4
[error]   org.http4s.parser.SimpleHeaders$$anon$1
[error]   org.http4s.parser.LocationHeader$LocationParser
[error]   org.http4s.parser.StrictTransportSecurityHeader$StrictTransportSecurityParser
[error]   org.http4s.parser.SimpleHeaders$$anon$2
[error]   org.http4s.parser.RangeParser$$anon$1
[error]   org.http4s.parser.ZipkinHeader$$anon$5
[error]   org.http4s.parser.AcceptHeader$AcceptParser
[error]   org.http4s.parser.RangeParser$$anon$2
[error]   org.http4s.parser.SimpleHeaders$$anon$12
[error]   org.http4s.parser.ZipkinHeader$$anon$1
[error]   org.http4s.parser.AuthorizationHeader$AuthorizationParser
[error]   org.http4s.parser.CookieHeader$SetCookieParser
[error]   org.http4s.parser.SimpleHeaders$$anon$6
[error]   org.http4s.parser.ProxyAuthenticateHeader$ProxyAuthenticateParser
[error]   org.http4s.parser.SimpleHeaders$$anon$17
[error]   org.http4s.parser.SimpleHeaders$$anon$13
[error]   org.http4s.parser.ZipkinHeader$$anon$3
[error]   org.http4s.parser.RangeParser$AcceptRangesParser
[error]   org.http4s.parser.SimpleHeaders$$anon$7
[error]   org.http4s.parser.SimpleHeaders$$anon$11
[error]   org.http4s.parser.ZipkinHeader$$anon$2
[error]   org.http4s.parser.SimpleHeaders$$anon$3
[error]   org.http4s.parser.SimpleHeaders$$anon$8
[error]   org.http4s.parser.SimpleHeaders$$anon$16
[error]   org.http4s.parser.SimpleHeaders$$anon$9
[error]   org.http4s.parser.CacheControlHeader$CacheControlParser
[error]   org.http4s.parser.AcceptLanguageHeader$AcceptLanguageParser
[error]   org.http4s.parser.ZipkinHeader$$anon$4
[error]   org.http4s.parser.CookieHeader$CookieParser
[error]   org.http4s.parser.HttpHeaderParser$
[error]   org.http4s.Header$Raw
[error]   org.http4s.Headers
[error]   org.specs2.matcher.describe.ArrayDifference
[error]   org.specs2.matcher.describe.TryDifferent
[error]   org.specs2.matcher.describe.ThrowableDifferentMessage
[error]   org.specs2.matcher.describe.OptionIdentical
[error]   org.specs2.matcher.describe.StackElementDifferent
[error]   org.specs2.matcher.describe.ThrowableDifferentStackTrace
[error]   org.specs2.matcher.EqualityMatcher$$anon$1
[error]   org.specs2.matcher.EqualityMatcher
[error]   org.specs2.matcher.BeEqualTo
[error]   org.specs2.matcher.MustExpectationsCreation$$anon$2
[error]   org.specs2.matcher.Expectable$$anon$2
[error]   org.specs2.matcher.ExpectationsCreation$$anon$1
[error]   org.specs2.matcher.Expectable
[error]   org.specs2.matcher.MustThrownExpectationsCreation$$anon$4
[error]   org.http4s.ContentCodingSpec
[error] org.scalajs.core.tools.linker.LinkingException: There were linking errors
[error]         at org.scalajs.core.tools.linker.frontend.BaseLinker.linkInternal(BaseLinker.scala:160)
[error]         at org.scalajs.core.tools.linker.frontend.BaseLinker.linkInternal(BaseLinker.scala:108)
[error]         at org.scalajs.core.tools.linker.frontend.LinkerFrontend.$anonfun$link$3(LinkerFrontend.scala:63)
[error]         at org.scalajs.core.tools.logging.Logger.time(Logger.scala:28)
[error]         at org.scalajs.core.tools.logging.Logger.time$(Logger.scala:26)
[error]         at org.scalajs.sbtplugin.Loggers$SbtLoggerWrapper.time(Loggers.scala:7)
[error]         at org.scalajs.core.tools.linker.frontend.LinkerFrontend.link(LinkerFrontend.scala:62)
[error]         at org.scalajs.core.tools.linker.Linker.$anonfun$link$1(Linker.scala:52)
[error]         at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
[error]         at org.scalajs.core.tools.linker.Linker.guard(Linker.scala:69)
[error]         at org.scalajs.core.tools.linker.Linker.link(Linker.scala:50)
[error]         at org.scalajs.core.tools.linker.ClearableLinker.$anonfun$link$1(ClearableLinker.scala:52)
[error]         at org.scalajs.core.tools.linker.ClearableLinker.$anonfun$link$1$adapted(ClearableLinker.scala:52)
[error]         at org.scalajs.core.tools.linker.ClearableLinker.linkerOp(ClearableLinker.scala:63)
[error]         at org.scalajs.core.tools.linker.ClearableLinker.link(ClearableLinker.scala:52)
[error]         at org.scalajs.sbtplugin.ScalaJSPluginInternal$.$anonfun$scalaJSStageSettings$11(ScalaJSPluginInternal.scala:313)
[error]         at sbt.util.FileFunction$.$anonfun$cached$1(FileFunction.scala:73)
[error]         at sbt.util.FileFunction$.$anonfun$cached$4(FileFunction.scala:147)
[error]         at sbt.util.Difference.apply(Tracked.scala:313)
[error]         at sbt.util.Difference.apply(Tracked.scala:293)
[error]         at sbt.util.FileFunction$.$anonfun$cached$3(FileFunction.scala:143)
[error]         at sbt.util.Difference.apply(Tracked.scala:313)
[error]         at sbt.util.Difference.apply(Tracked.scala:288)
[error]         at sbt.util.FileFunction$.$anonfun$cached$2(FileFunction.scala:142)
[error]         at org.scalajs.sbtplugin.ScalaJSPluginInternal$.$anonfun$scalaJSStageSettings$10(ScalaJSPluginInternal.scala:318)
[error]         at sbt.std.Transform$$anon$3.$anonfun$apply$2(System.scala:44)
[error]         at sbt.std.Transform$$anon$4.work(System.scala:64)
[error]         at sbt.Execute.$anonfun$submit$2(Execute.scala:257)
[error]         at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
[error]         at sbt.Execute.work(Execute.scala:266)
[error]         at sbt.Execute.$anonfun$submit$1(Execute.scala:257)
[error]         at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:167)
[error]         at sbt.CompletionService$$anon$2.call(CompletionService.scala:32)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error]         at java.lang.Thread.run(Thread.java:748)
[error] (testsJS/test:fastOptJS) org.scalajs.core.tools.linker.LinkingException: There were linking errors
[error] Total time: 59 s, completed Jan 15, 2018 1:19:41 AM

Workaround

My current, very limited, understanding is that the macros somehow generate parsers which assume that other parsers it is dependent upon are available in some current scope. Or at least the Scala.js linker interprets the generated code as if it should be available. I'm not sure. Either way, the workaround is to actually put those parsers in scope. This has the effect of either cluing the Scala.js linker in that it's there, or it adds it, potentially duplicating some parser code. Either way, tests run green, and I'm certainly going to leave it at that :)

In this particular case, the H16 parser lives in IpParser, so we tell the compiler as much:

  private trait TransferCodingParser extends IpParser { self: PbParser =>
    def codingRule = rule {
      "chunked" ~ push(chunked) |
        "compress" ~ push(compress) |
        "deflate" ~ push(deflate) |
        "gzip" ~ push(gzip) |
        "identity" ~ push(identity)
    }
  }

Specs2 / unsafeRunSync

Again, as @cquiroz noted in #956 (comment), a bunch of the tests use unsafeRunSync, which is not implementable (search for unsafeRunSync) in javascript. From a cursory look, Specs2 doesn't seem to provide a callback-based test runner like scalatest, for instance.

Rewrite or keep those tests to be jvm-only would do the trick.

The good, bad and ...

The code sharing and flexibility enabled by cross compilation of this project can be absolutely mindblowing. There is, however, a price to pay.

[error] Error: Total 990, Failed 3, Errors 7, Passed 979, Skipped 1, Pending 3
[error] Failed tests:
[error]         org.http4s.UriSpec
[error]         org.http4s.util.CaseInsensitiveStringSpec
[error] Error during tests:
[error]         org.http4s.EntityDecoderSpec
[error]         org.http4s.UriTemplateSpec
[error]         org.http4s.EntityEncoderSpec
[error] (testsJS / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 184 s, completed Jan 17, 2018 10:24:53 PM

ls -lah tests/js/target/scala-2.12/http4s-tests-test-fastopt.js
-rw-rw-r-- 1 olvind olvind 20M jan.  17 23:04 tests/js/target/scala-2.12/http4s-tests-test-fastopt.js

Just parsing 20 megabytes of code is heavy duty work for node, so tests take their time. It's all expected when you think of all that is included and all that is ran, but still... We should do some tests to see what typical compile/link performance of downstream projects to judge the practicality of this before we proceed much further.

Also, in case anybody was wondering:

//using -Xmx4g for some extra heap

[http4s] λ testsJS/test:fullOptJS
[info] Full optimizing /home/olvind/pr/http4s/tests/js/target/scala-2.12/http4s-tests-test-opt.js
[info] Closure: 0 error(s), 0 warning(s)
[success] Total time: 173 s, completed Jan 17, 2018 11:10:30 PM
[http4s] λ [olvind:~/pr/http4s] scala.js(+1/-1,+158/-27)+* 4m44s ± ls -lah tests/js/target/scala-2.12/http4s-tests-test-opt.js
-rw-rw-r-- 1 olvind olvind 4,5M jan.  17 23:10 tests/js/target/scala-2.12/http4s-tests-test-opt.js

@cquiroz
Copy link
Member Author

cquiroz commented Jan 18, 2018

@oyvindberg Wow, that's great. I've been trying to get the tests to run by removing/replacing the bits on http4s that cannot properly run on scala.js. Unfortunately I'm short on time so i'd be quite happy to accept your contributions

Some comments:

  • I have a partially done port of MediaType without Registry. I'd need to check what you did but I'd rather post a proper change to http4s core. Of course we can get by with the simpler mutable.HashMap
  • locales is available for scala.js scala-java-locales BUT String.toLowerCase can't be added that way properly see String.toLowerCase and toUpperCase does not work with current locale scala-js/scala-js#2551
  • About the missing APIs ideally we'd submit them to scala.js but they could live on http4s in the interim
  • It's absolutely great that you could find the macro issue. I was really scared of that part 😄

In summary I really love this. Let's think about ways to move this forward

@oyvindberg
Copy link

Great! I guess we'll just have to follow your path and fix the outstanding issues one by one. Once we find a reasonable solution to an issue, we extract the changeset and submit a proper PR with that.

How about I take the macro issue and the reflection issue for now?

@oyvindberg
Copy link

Alright, I was able to spend a bit more time on the macro issue, see the linked bug report in Scala.js above.

You really did quite a job porting dates/locales to Scala.js, @cquiroz - well done!

Even though we cannot currently reimplement String.toUpperCase(locale) and friends, do you have any idea if we can implement some StringUtils.toUpperCase(str, locale) in scala-java-locales? Or would that be a very big undertaking?

@cquiroz
Copy link
Member Author

cquiroz commented Jan 23, 2018

@oyvindberg It would be hard I think. The reason is that you'd need to parse the rules from cldr and then implement the algorithm. I think it would be simpler to have an http4s facade that's implemented differently for js/jvm and have the js side only support english

@oyvindberg
Copy link

@rossabaker Hey there. I see your don't panic label, and just wanted to know:
I intend to keep working on this after march 1st. Does this change whether you would accept a PR? Is somebody else actually working on it?

@cquiroz
Copy link
Member Author

cquiroz commented Feb 21, 2018

@oyvindberg I have been hard pressed for time to continue with this. I still want to keep an eye and contribute somehow but I have no issue if you want to continue progressing this. I'm not sure what's the best mechanic, perhaps build on top of my PR?

I saw you got a bug fixed on scala.js, Do you need to wait for that?

@rossabaker
Copy link
Member

I have a support request in to GitHub to fix this (and other so-labelled) PRs. I'm still happy to see this work done.

If this PR is still closed when you are ready to resume, please open a new one and link to this. If we can reopen it, we can continue here.

@rossabaker
Copy link
Member

GitHub can't help bring this ticket back, so we'll need to reopen.

In the meantime, log4s just got preliminary scala.js support

@cquiroz
Copy link
Member Author

cquiroz commented Mar 14, 2018

Do you mean send another PR from my branch?

@rossabaker
Copy link
Member

Yes, another PR from the branch will reopen this. And then just mention this ticket, because there's some good discussion here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet