Optimize the generated site #100

olafurpg · 2018-09-27T22:06:52Z

This commit implements two optimizations to make the generate site
contain fewer files and be smaller in bytes.

gzip all generated files: this required changing the JavaScript client
to inflate responses with the library pako (http://nodeca.github.io/pako/#inflate).
I originally hoped the browser would automatically inflate the
reponses us by using the *.gz file extension. The response headers
for the webpack + browsersync + GitHub pages all return
Content-Encoding: gzip for filenames with the *.gz extension.
Nevertheless, the fetch response body contained deflated bytes
so I resorted to inflating in JavaScript.
group symbol indexes by package. This primarily reduces the number of
generated files. Most symbol indexes were only a few bytes but
occupied the minimum block-size on disk which is 4kb on my macbook.
For a Scalameta + Scalafix combined metadoc site, this change
made the total symbol index go from taking 40mb on disk to 3.4mb!

To measure these optimizations I ran metadoc against three different
corpora.

SemanticDB index:

Corpus	Before	After
paiges	332kb	112kb
scalameta + scalafix	24mb	5.9mb
megasite	1.25gb	209mb

Symbol index:

Corpus	Before	After
paiges	776kb	16kb
scalameta + scalafix	45mb	888kb
megasite	n/a	44mb

The "megasite" corpus is a combination of 55 projects with a
total of 3.5 million lines of code. Generating a metadoc site for the
megasite takes ~50 seconds on my laptop.

olafurpg · 2018-09-27T22:12:42Z

Grouping by top-level package is problematic for widely referenced packages, for example the org.scalatest symbol index in the megasite is 11mb. I was thinking of chunking down those files in ~500kb sizes and the client needs to "guess" the first package index and from there we can link to a secondary package index. That would mean users you never download more than ~1mb to resolve a symbol (and we can lower 1mb at the cost of producing more files).

Either way, I believe even 11mb is an acceptable solution considering the megasite is for a corpus of 3.5 million loc corpus. We can further improve the situation in followup PRs.

olafurpg · 2018-09-27T22:27:10Z

Here is a live demo with the combined scalameta + scalafix site: https://olafurpg.github.io/metadoc/#/scalameta/scalameta/io/shared/src/main/scala/scala/meta/io/AbsolutePath.scala#L14C28-L14C40

Try "Find all references" on AbsolutePath

jonas

💯 Looks great.

I agree that an 11M package index is acceptable for the megasite, especially considering that this corpus size was not really manageable before.

jonas · 2018-09-28T03:53:52Z

build.sbt

@@ -262,7 +263,7 @@ lazy val noPublish = allSettings ++ Seq(
  publishArtifact := false
 )

-addCommandAlias("ci-test", ";compile ;test")
+addCommandAlias("ci-test", ";compile ; metadoc-site ; test")


Needed for the JS test to inflate index.workspace.gz with pako

jonas · 2018-09-28T03:58:53Z

metadoc-js/src/main/scala/metadoc/MetadocApp.scala

      _ = require(response.status == 200, s"${response.status} != 200")
      buffer <- response.arrayBuffer().toFuture
+      _ = println(buffer.byteLength)


jonas · 2018-09-28T03:59:01Z

metadoc-js/src/main/scala/metadoc/MetadocApp.scala

    for {
      response <- dom.experimental.Fetch.fetch(url).toFuture
+      _ = println(JSON.stringify(response))


jonas · 2018-09-28T04:00:40Z

metadoc-js/src/main/scala/metadoc/MetadocFetch.scala


 object MetadocFetch {

  def symbol(symbolId: String): Future[Option[d.SymbolIndex]] = {
-    val url = "symbol/" + JSSha512.sha512(symbolId)


Does this mean we can remove the js-sha512 NPM dependency and the Scala.js binding?

Good catch, I've removed the dependency now 👍

jonas · 2018-09-28T04:04:27Z

metadoc-tests/src/test/scala/metadoc/tests/GeneratedSiteEnrichments.scala

+import scalapb.Message
+
+object GeneratedSiteEnrichments {
+  implicit class XTensionGeneratedMessageCompanion[


The naming convention is to prefix with Xtension...

jonas · 2018-09-28T04:05:40Z

metadoc-tests/src/test/scala/metadoc/tests/JsonSuite.scala

@@ -42,9 +44,9 @@ class JsonSuite extends FunSuite with DiffAssertions with BeforeAndAfterAll {
    val symbols = FileIO
      .listAllFilesRecursively(AbsolutePath(out).resolve("symbol"))
      .toList
-    assert(symbols.length == 2)
+    assert(symbols.length == 1)


One file per package. 👍

The symbol index is a lot nicer to browse locally now

out/symbol ├── C │ └── package.symbolindexes.gz ├── _empty_ │ └── package.symbolindexes.gz ├── advanced │ └── package.symbolindexes.gz ├── annot │ └── package.symbolindexes.gz ├── banana │ └── rule │ └── package.symbolindexes.gz ├── classes │ └── package.symbolindexes.gz ├── com │ └── geirsson │ ├── mutable │ │ ├── package.symbolindexes.gz │ │ └── unsafe │ │ └── package.symbolindexes.gz │ └── package.symbolindexes.gz ├── docs │ └── package.symbolindexes.gz ├── example │ └── package.symbolindexes.gz ├── fix │ └── package.symbolindexes.gz ├── flags │ └── p │ └── package.symbolindexes.gz ├── locals │ └── package.symbolindexes.gz ├── objects │ └── package.symbolindexes.gz ├── org │ └── scalameta │ ├── adt │ │ └── package.symbolindexes.gz │ ├── collections │ │ └── package.symbolindexes.gz │ ├── data │ │ └── package.symbolindexes.gz │ ├── explore │ │ └── package.symbolindexes.gz │ ├── internal │ │ └── package.symbolindexes.gz │ ├── invariants │ │ └── package.symbolindexes.gz │ ├── overload │ │ └── package.symbolindexes.gz │ ├── package.symbolindexes.gz │ └── tests │ └── package.symbolindexes.gz ├── prefixes │ └── package.symbolindexes.gz ├── scala │ ├── meta │ │ ├── classifiers │ │ │ └── package.symbolindexes.gz │ │ ├── cli │ │ │ └── package.symbolindexes.gz │ │ ├── common │ │ │ └── package.symbolindexes.gz │ │ ├── contrib │ │ │ ├── equality │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── implicits │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── instances │ │ │ │ └── package.symbolindexes.gz │ │ │ └── package.symbolindexes.gz │ │ ├── dialects │ │ │ └── package.symbolindexes.gz │ │ ├── inputs │ │ │ └── package.symbolindexes.gz │ │ ├── interactive │ │ │ └── package.symbolindexes.gz │ │ ├── internal │ │ │ ├── classifiers │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── classpath │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── cli │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── dialects │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── inputs │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── io │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── javacp │ │ │ │ ├── asm │ │ │ │ │ └── package.symbolindexes.gz │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── metacp │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── metai │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── metap │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── package.symbolindexes.gz │ │ │ ├── parsers │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── platform │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── prettyprinters │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── quasiquotes │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── scalacp │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── scalafix │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── semanticdb │ │ │ │ ├── javac │ │ │ │ │ ├── package.symbolindexes.gz │ │ │ │ │ └── semantics │ │ │ │ │ └── package.symbolindexes.gz │ │ │ │ ├── package.symbolindexes.gz │ │ │ │ └── scalac │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── semanticidx │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── symtab │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── tokenizers │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── tokens │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── transversers │ │ │ │ └── package.symbolindexes.gz │ │ │ └── trees │ │ │ └── package.symbolindexes.gz │ │ ├── io │ │ │ └── package.symbolindexes.gz │ │ ├── metacp │ │ │ └── package.symbolindexes.gz │ │ ├── metai │ │ │ └── package.symbolindexes.gz │ │ ├── metap │ │ │ └── package.symbolindexes.gz │ │ ├── package.symbolindexes.gz │ │ ├── parsers │ │ │ └── package.symbolindexes.gz │ │ ├── prettyprinters │ │ │ └── package.symbolindexes.gz │ │ ├── quasiquotes │ │ │ └── package.symbolindexes.gz │ │ ├── testkit │ │ │ └── package.symbolindexes.gz │ │ ├── tests │ │ │ ├── contrib │ │ │ │ └── package.symbolindexes.gz │ │ │ └── prettyprinters │ │ │ └── package.symbolindexes.gz │ │ ├── tokenizers │ │ │ └── package.symbolindexes.gz │ │ ├── tokens │ │ │ └── package.symbolindexes.gz │ │ ├── transversers │ │ │ └── package.symbolindexes.gz │ │ └── trees │ │ └── package.symbolindexes.gz │ └── tools │ └── nsc │ └── typechecker │ └── package.symbolindexes.gz ├── scalafix │ ├── cli │ │ └── package.symbolindexes.gz │ ├── config │ │ └── package.symbolindexes.gz │ ├── docs │ │ └── package.symbolindexes.gz │ ├── internal │ │ ├── config │ │ │ └── package.symbolindexes.gz │ │ ├── diff │ │ │ └── package.symbolindexes.gz │ │ ├── interfaces │ │ │ └── package.symbolindexes.gz │ │ ├── jgit │ │ │ └── package.symbolindexes.gz │ │ ├── patch │ │ │ └── package.symbolindexes.gz │ │ ├── reflect │ │ │ └── package.symbolindexes.gz │ │ ├── rule │ │ │ └── package.symbolindexes.gz │ │ ├── testkit │ │ │ └── package.symbolindexes.gz │ │ ├── tests │ │ │ └── utils │ │ │ └── package.symbolindexes.gz │ │ ├── util │ │ │ └── package.symbolindexes.gz │ │ ├── v0 │ │ │ └── package.symbolindexes.gz │ │ └── v1 │ │ └── package.symbolindexes.gz │ ├── lint │ │ └── package.symbolindexes.gz │ ├── package.symbolindexes.gz │ ├── patch │ │ └── package.symbolindexes.gz │ ├── reflect │ │ └── package.symbolindexes.gz │ ├── rule │ │ └── package.symbolindexes.gz │ ├── syntax │ │ └── package.symbolindexes.gz │ ├── test │ │ └── package.symbolindexes.gz │ ├── testkit │ │ └── package.symbolindexes.gz │ ├── tests │ │ ├── cli │ │ │ └── package.symbolindexes.gz │ │ ├── config │ │ │ └── package.symbolindexes.gz │ │ ├── core │ │ │ ├── package.symbolindexes.gz │ │ │ └── util │ │ │ └── package.symbolindexes.gz │ │ ├── package.symbolindexes.gz │ │ ├── reflect │ │ │ └── package.symbolindexes.gz │ │ ├── rule │ │ │ └── package.symbolindexes.gz │ │ ├── testkit │ │ │ └── package.symbolindexes.gz │ │ ├── util │ │ │ └── package.symbolindexes.gz │ │ ├── v0 │ │ │ └── package.symbolindexes.gz │ │ └── v1 │ │ └── package.symbolindexes.gz │ ├── util │ │ └── package.symbolindexes.gz │ ├── v0 │ │ └── package.symbolindexes.gz │ ├── v1 │ │ └── package.symbolindexes.gz │ └── website │ └── package.symbolindexes.gz ├── selfs │ └── package.symbolindexes.gz ├── test │ ├── disableSyntax │ │ └── package.symbolindexes.gz │ ├── escapeHatch │ │ └── package.symbolindexes.gz │ ├── explicitResultTypes │ │ └── package.symbolindexes.gz │ ├── package.symbolindexes.gz │ └── pkg │ └── package.symbolindexes.gz ├── test2 │ └── package.symbolindexes.gz ├── traits │ └── package.symbolindexes.gz └── types └── package.symbolindexes.gz 136 directories, 126 files

jonas · 2018-09-28T04:06:33Z

metadoc-tests/src/test/scala/metadoc/tests/BaseMetadocCliSuite.scala

+import metadoc.MetadocEnrichments._
+import GeneratedSiteEnrichments._
+
+class BaseMetadocCliSuite


Nice cleanup

olafurpg

Thanks for the review!

olafurpg · 2018-09-28T07:34:17Z

build.sbt

@@ -262,7 +263,7 @@ lazy val noPublish = allSettings ++ Seq(
  publishArtifact := false
 )

-addCommandAlias("ci-test", ";compile ;test")
+addCommandAlias("ci-test", ";compile ; metadoc-site ; test")


Needed for the JS test to inflate index.workspace.gz with pako

olafurpg · 2018-09-28T07:34:56Z

metadoc-js/src/main/scala/metadoc/MetadocApp.scala

@@ -43,6 +45,7 @@ object MetadocApp {
      _ <- loadMonaco()
      workspace <- MetadocFetch.workspace()
    } {
+      println(workspace.toProtoString)


olafurpg · 2018-09-28T07:35:02Z

metadoc-js/src/main/scala/metadoc/MetadocApp.scala

    for {
      response <- dom.experimental.Fetch.fetch(url).toFuture
+      _ = println(JSON.stringify(response))


olafurpg · 2018-09-28T07:35:08Z

metadoc-js/src/main/scala/metadoc/MetadocApp.scala

      _ = require(response.status == 200, s"${response.status} != 200")
      buffer <- response.arrayBuffer().toFuture
+      _ = println(buffer.byteLength)


olafurpg · 2018-09-28T07:50:25Z

metadoc-js/src/main/scala/metadoc/MetadocFetch.scala


 object MetadocFetch {

  def symbol(symbolId: String): Future[Option[d.SymbolIndex]] = {
-    val url = "symbol/" + JSSha512.sha512(symbolId)


Good catch, I've removed the dependency now 👍

olafurpg · 2018-09-28T07:51:19Z

metadoc-tests/src/test/scala/metadoc/tests/GeneratedSiteEnrichments.scala

+import scalapb.Message
+
+object GeneratedSiteEnrichments {
+  implicit class XTensionGeneratedMessageCompanion[


olafurpg · 2018-09-28T07:52:34Z

metadoc-tests/src/test/scala/metadoc/tests/JsonSuite.scala

@@ -42,9 +44,9 @@ class JsonSuite extends FunSuite with DiffAssertions with BeforeAndAfterAll {
    val symbols = FileIO
      .listAllFilesRecursively(AbsolutePath(out).resolve("symbol"))
      .toList
-    assert(symbols.length == 2)
+    assert(symbols.length == 1)


The symbol index is a lot nicer to browse locally now

out/symbol ├── C │ └── package.symbolindexes.gz ├── _empty_ │ └── package.symbolindexes.gz ├── advanced │ └── package.symbolindexes.gz ├── annot │ └── package.symbolindexes.gz ├── banana │ └── rule │ └── package.symbolindexes.gz ├── classes │ └── package.symbolindexes.gz ├── com │ └── geirsson │ ├── mutable │ │ ├── package.symbolindexes.gz │ │ └── unsafe │ │ └── package.symbolindexes.gz │ └── package.symbolindexes.gz ├── docs │ └── package.symbolindexes.gz ├── example │ └── package.symbolindexes.gz ├── fix │ └── package.symbolindexes.gz ├── flags │ └── p │ └── package.symbolindexes.gz ├── locals │ └── package.symbolindexes.gz ├── objects │ └── package.symbolindexes.gz ├── org │ └── scalameta │ ├── adt │ │ └── package.symbolindexes.gz │ ├── collections │ │ └── package.symbolindexes.gz │ ├── data │ │ └── package.symbolindexes.gz │ ├── explore │ │ └── package.symbolindexes.gz │ ├── internal │ │ └── package.symbolindexes.gz │ ├── invariants │ │ └── package.symbolindexes.gz │ ├── overload │ │ └── package.symbolindexes.gz │ ├── package.symbolindexes.gz │ └── tests │ └── package.symbolindexes.gz ├── prefixes │ └── package.symbolindexes.gz ├── scala │ ├── meta │ │ ├── classifiers │ │ │ └── package.symbolindexes.gz │ │ ├── cli │ │ │ └── package.symbolindexes.gz │ │ ├── common │ │ │ └── package.symbolindexes.gz │ │ ├── contrib │ │ │ ├── equality │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── implicits │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── instances │ │ │ │ └── package.symbolindexes.gz │ │ │ └── package.symbolindexes.gz │ │ ├── dialects │ │ │ └── package.symbolindexes.gz │ │ ├── inputs │ │ │ └── package.symbolindexes.gz │ │ ├── interactive │ │ │ └── package.symbolindexes.gz │ │ ├── internal │ │ │ ├── classifiers │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── classpath │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── cli │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── dialects │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── inputs │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── io │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── javacp │ │ │ │ ├── asm │ │ │ │ │ └── package.symbolindexes.gz │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── metacp │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── metai │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── metap │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── package.symbolindexes.gz │ │ │ ├── parsers │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── platform │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── prettyprinters │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── quasiquotes │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── scalacp │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── scalafix │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── semanticdb │ │ │ │ ├── javac │ │ │ │ │ ├── package.symbolindexes.gz │ │ │ │ │ └── semantics │ │ │ │ │ └── package.symbolindexes.gz │ │ │ │ ├── package.symbolindexes.gz │ │ │ │ └── scalac │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── semanticidx │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── symtab │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── tokenizers │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── tokens │ │ │ │ └── package.symbolindexes.gz │ │ │ ├── transversers │ │ │ │ └── package.symbolindexes.gz │ │ │ └── trees │ │ │ └── package.symbolindexes.gz │ │ ├── io │ │ │ └── package.symbolindexes.gz │ │ ├── metacp │ │ │ └── package.symbolindexes.gz │ │ ├── metai │ │ │ └── package.symbolindexes.gz │ │ ├── metap │ │ │ └── package.symbolindexes.gz │ │ ├── package.symbolindexes.gz │ │ ├── parsers │ │ │ └── package.symbolindexes.gz │ │ ├── prettyprinters │ │ │ └── package.symbolindexes.gz │ │ ├── quasiquotes │ │ │ └── package.symbolindexes.gz │ │ ├── testkit │ │ │ └── package.symbolindexes.gz │ │ ├── tests │ │ │ ├── contrib │ │ │ │ └── package.symbolindexes.gz │ │ │ └── prettyprinters │ │ │ └── package.symbolindexes.gz │ │ ├── tokenizers │ │ │ └── package.symbolindexes.gz │ │ ├── tokens │ │ │ └── package.symbolindexes.gz │ │ ├── transversers │ │ │ └── package.symbolindexes.gz │ │ └── trees │ │ └── package.symbolindexes.gz │ └── tools │ └── nsc │ └── typechecker │ └── package.symbolindexes.gz ├── scalafix │ ├── cli │ │ └── package.symbolindexes.gz │ ├── config │ │ └── package.symbolindexes.gz │ ├── docs │ │ └── package.symbolindexes.gz │ ├── internal │ │ ├── config │ │ │ └── package.symbolindexes.gz │ │ ├── diff │ │ │ └── package.symbolindexes.gz │ │ ├── interfaces │ │ │ └── package.symbolindexes.gz │ │ ├── jgit │ │ │ └── package.symbolindexes.gz │ │ ├── patch │ │ │ └── package.symbolindexes.gz │ │ ├── reflect │ │ │ └── package.symbolindexes.gz │ │ ├── rule │ │ │ └── package.symbolindexes.gz │ │ ├── testkit │ │ │ └── package.symbolindexes.gz │ │ ├── tests │ │ │ └── utils │ │ │ └── package.symbolindexes.gz │ │ ├── util │ │ │ └── package.symbolindexes.gz │ │ ├── v0 │ │ │ └── package.symbolindexes.gz │ │ └── v1 │ │ └── package.symbolindexes.gz │ ├── lint │ │ └── package.symbolindexes.gz │ ├── package.symbolindexes.gz │ ├── patch │ │ └── package.symbolindexes.gz │ ├── reflect │ │ └── package.symbolindexes.gz │ ├── rule │ │ └── package.symbolindexes.gz │ ├── syntax │ │ └── package.symbolindexes.gz │ ├── test │ │ └── package.symbolindexes.gz │ ├── testkit │ │ └── package.symbolindexes.gz │ ├── tests │ │ ├── cli │ │ │ └── package.symbolindexes.gz │ │ ├── config │ │ │ └── package.symbolindexes.gz │ │ ├── core │ │ │ ├── package.symbolindexes.gz │ │ │ └── util │ │ │ └── package.symbolindexes.gz │ │ ├── package.symbolindexes.gz │ │ ├── reflect │ │ │ └── package.symbolindexes.gz │ │ ├── rule │ │ │ └── package.symbolindexes.gz │ │ ├── testkit │ │ │ └── package.symbolindexes.gz │ │ ├── util │ │ │ └── package.symbolindexes.gz │ │ ├── v0 │ │ │ └── package.symbolindexes.gz │ │ └── v1 │ │ └── package.symbolindexes.gz │ ├── util │ │ └── package.symbolindexes.gz │ ├── v0 │ │ └── package.symbolindexes.gz │ ├── v1 │ │ └── package.symbolindexes.gz │ └── website │ └── package.symbolindexes.gz ├── selfs │ └── package.symbolindexes.gz ├── test │ ├── disableSyntax │ │ └── package.symbolindexes.gz │ ├── escapeHatch │ │ └── package.symbolindexes.gz │ ├── explicitResultTypes │ │ └── package.symbolindexes.gz │ ├── package.symbolindexes.gz │ └── pkg │ └── package.symbolindexes.gz ├── test2 │ └── package.symbolindexes.gz ├── traits │ └── package.symbolindexes.gz └── types └── package.symbolindexes.gz 136 directories, 126 files

This commit implements two optimizations to make the generate site contain fewer files and be smaller in bytes. - gzip all generated files: this required changing the JavaScript client to inflate responses with the library pako (http://nodeca.github.io/pako/#inflate). I originally hoped the browser would automatically inflate the reponses us by using the `*.gz` file extension. The response headers for the webpack + browsersync + GitHub pages all return `Content-Encoding: gzip` for filenames with the `*.gz` extension. Nevertheless, the fetch response body contained deflated bytes so I resorted to inflating in JavaScript. - group symbol indexes by package. This primarily reduces the number of generated files. Most symbol indexes were only a few bytes but occupied the minimum block-size on disk which is 4kb on my macbook. For a Scalameta + Scalafix combined metadoc site, this change made the total symbol index go from taking 40mb on disk to 3.4mb! To measure these optimizations I ran metadoc against three different corpora. SemanticDB index: | Corpus | Before | After | | -------------------- | ------ | ----- | | paiges | 332kb | 112kb | | scalameta + scalafix | 24mb | 5.9mb | | megasite | 1.25gb | 209mb | Symbol index: | Corpus | Before | After | | -------------------- | ------ | ----- | | paiges | 776kb | 16kb | | scalameta + scalafix | 45mb | 888kb | | megasite | n/a | 44mb | The "megasite" corpus is a combination of 55 projects with a total of 3.5 million lines of code. Generating a metadoc site for the megasite takes ~50 seconds on my laptop.

olafurpg · 2018-09-28T09:00:44Z

Rebased on master and updated the demo to include scala-library https://olafurpg.github.io/metadoc/#/scala/Predef.scala#L134C8-L134C14

olafurpg · 2018-09-28T09:07:16Z

CI went from 10 minutes to 5 minutes for scripted tests on sbt 1 after rebasing on top of #101, awesome work @jonas

olafurpg · 2018-09-28T09:09:46Z

No more CI timeouts \o/

Guess not having to downloads 350mb of icons helps 😂

jonas · 2018-09-28T12:23:41Z

Oh, man, 🤦‍♂️ downloading +300Mb for one icon. What was I thinking.

olafurpg requested a review from jonas September 27, 2018 22:07

jonas approved these changes Sep 28, 2018

View reviewed changes

olafurpg mentioned this pull request Sep 28, 2018

Chunk large package indexes #102

Open

olafurpg commented Sep 28, 2018

View reviewed changes

olafurpg added 4 commits September 28, 2018 10:41

Fix scripted tests

c9f6a03

Address review feedback

9f434b5

Commit yarn lock file

9e0f248

olafurpg force-pushed the optimize branch from 52815e6 to 9e0f248 Compare September 28, 2018 09:00

olafurpg merged commit 5ebe7b0 into scalameta:master Sep 28, 2018

olafurpg deleted the optimize branch September 28, 2018 09:09

olafurpg mentioned this pull request Sep 28, 2018

Run demo on a larger corpus #22

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimize the generated site #100

Optimize the generated site #100

olafurpg commented Sep 27, 2018

olafurpg commented Sep 27, 2018

olafurpg commented Sep 27, 2018

jonas left a comment

jonas Sep 28, 2018

olafurpg Sep 28, 2018

jonas Sep 28, 2018

olafurpg Sep 28, 2018

jonas Sep 28, 2018

olafurpg Sep 28, 2018

jonas Sep 28, 2018

olafurpg Sep 28, 2018

jonas Sep 28, 2018

olafurpg Sep 28, 2018

jonas Sep 28, 2018

olafurpg Sep 28, 2018

jonas Sep 28, 2018

olafurpg left a comment

olafurpg Sep 28, 2018

olafurpg Sep 28, 2018

olafurpg Sep 28, 2018

olafurpg Sep 28, 2018

olafurpg Sep 28, 2018

olafurpg Sep 28, 2018

olafurpg Sep 28, 2018

olafurpg commented Sep 28, 2018

olafurpg commented Sep 28, 2018

olafurpg commented Sep 28, 2018

jonas commented Sep 28, 2018

Optimize the generated site #100

Optimize the generated site #100

Conversation

olafurpg commented Sep 27, 2018

olafurpg commented Sep 27, 2018

olafurpg commented Sep 27, 2018

jonas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olafurpg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olafurpg commented Sep 28, 2018

olafurpg commented Sep 28, 2018

olafurpg commented Sep 28, 2018

jonas commented Sep 28, 2018