Reduce the impact of MimeDB on JS size

[cquiroz]

We are testing http4s for our scala.js browser app and it seems to produce a fairly big increase on the scala.js produced by scala.js. One of the contributors is the very large MimeDB which maybe not needed on the typical frontend app

Should we split MimeDB into jvm and js versions, with the latter having a minimal or empty set of mime types?

[cquiroz]

/cc @rpiaggio

[armanbilge]

Thanks for raising this issue and for testing this stuff! @ChristopherDavenport also identified this problem while testing the initial PR.

Should we split MimeDB into jvm and js versions, with the latter having a minimal or empty set of mime types?

Before we resort to this, I'd like to understand why the Scala.js static linker is unable to dead-code-eliminate the MimeDB if it's not being used. If we can identify what causes it to become statically linked, maybe we can fix it there?

Do you have any ideas about this? I think I remember seeing similar issues on some of your SJS projects which you resolved by e.g. using lazy vals, etc.

[cquiroz]

Right lazy val is an option but it is mostly a heuristic, all this is quite labor intensive as it takes a while to figure out what the optimizer is doing. We could try that but at the moment in full optimized scala.js we endup with stuff like:

this.Lorg_http4s_MimeDB$audio$__f_vnd$u002E3gpp$u002Eiufp = null,

Note this is with ES enabled which disables GCC

[armanbilge]

Note this is with ES enabled which disables GCC

Can you explain more about this? Or provide a pointer to a reference, not sure what this is or why you'd want to do this.

Right lazy val is an option but it is mostly a heuristic, all this is quite labor intensive as it takes a while to figure out what the optimizer is doing.

This comment in Scala.js explains how the DCE works I think: https://github.com/scala-js/scala-js/blob/master/javalanglib/src/main/scala/java/lang/System.scala#L24-L35

[armanbilge]

Ok, I looked into this a bit. There are a few pathways that can cause the entire MimeDB to become statically linked, but here's an easy one:

Add Accept header to request -> Accept.headerInstance -> Accept.parse -> Accept.parser -> MediaRange.parser -> MediaRange.getMediaRange -> MediaType.all -> MimeDB.allMediaTypes

I don't think there's any way around this. So, the only solution as @cquiroz proposes would be to reduce MimeDB for JS.

However, while this makes sense for the browser, I would prefer that this does not affect the Node.js runtime which should ideally have feature-parity with the JVM. I have no idea how to do this without publishing separate artifacts.

[cquiroz]

I suppose it maybe worthwhile to check if making the vals in MimeDB lazy helps. Maybe there are other parts as well that could be optimized that way

[armanbilge]

I suppose it maybe worthwhile to check if making the vals in MimeDB lazy helps

I'm pretty sure it won't. Even if lazyness defers initialization at runtime, the initialization code will be statically linked in the Accept pathway for example.

Maybe there are other parts as well that could be optimized that way

This is definitely possible.

[cquiroz]

I guess a related question is how do we get to Accept. Our client code is fairly simple, I'm not sure we're using Accept. Maybe headers should be set directly with text to avoid this link

[armanbilge]

I guess a related question is how do we get to Accept.

It used in some of the methods in DefaultClient. Also, it's not the only pathway that leads MimeDB to be statically linked, just one example that seems really easy to hit in ordinary use.

Maybe headers should be set directly with text to avoid this link

That could work 🤔

[cquiroz]

At any rate, MimeDb is the most obvious case of a large clase not being used, I guess there would be a bunch more that are not directly
Needed that are still linked via some pathway

I recall investigating this same sort of problems with scala-java-time would take me to check the jar files to look for call trees, again this is quite laborious work and soemtimes improvements would be minimal

[armanbilge]

@cquiroz what is your target JS size?

[cquiroz]

Our largest application is a whopping 16MB with scala.js fullOpt. Adding http4s client takes it to 19MB

[armanbilge]

phew! 😅 That number looks about right, I was fullOptJSing a tiny app with just the fetch client and after GCC was getting a size of around 2-2.5 mb, can't remember exactly. Then removing the MimeDB was getting it down to 1.6 mb I think.

[cquiroz]

Yeah. I noted this on my very old attempt to do scala.js where the test suite was around 25M in size which was even bigger than scala-java-time which is the largest scala.js lib I know

[armanbilge]

Btw, scala-java-time is currently a dependency of core. I know many projects try to avoid this, and also could it be contributing the the JS size?

[cquiroz]

We alrready have scala-java-time so it should be accounted for in the orginal 16M

[armanbilge]

A basic fetch/circe app generates a 1.8 mb JS in #29 (comment).

I'm hoping a strategy like http4s/http4s#5273 may help with this.

[armanbilge]

See https://armanbilge.github.io/http4s-dom/sme.html for a helpful visual.

[cquiroz]

Cool, how did you generate that graph?

[armanbilge]

@cquiroz https://www.npmjs.com/package/source-map-explorer

Will you use it for your 18 mb app? 😆

[keynmol]

If I'm reading this right, it takes ~3 seconds to even parse the output of fastOptJS:

Which makes it sort of unusable for development, as both the re-linking and reloading in the browser take a long time

Vanilla app can be dropped down to 2.5MB with fullOptJS, but it has zero to none http4s logic. But fastOptJS produces a 14MB on an app that has very little apart from http4s itself

[armanbilge]

Yeah, thanks for this. The stupid large MimeDB that we get from http4s is the source of lots of problems including #119.

So, yeah, time to fix this.

[armanbilge]

I've PRed a fix upstream in http4s/http4s#6211. Please try the snapshot (instructions in PR :)