-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimize the generated site #100
Conversation
Grouping by top-level package is problematic for widely referenced packages, for example the Either way, I believe even 11mb is an acceptable solution considering the megasite is for a corpus of 3.5 million loc corpus. We can further improve the situation in followup PRs. |
Here is a live demo with the combined scalameta + scalafix site: https://olafurpg.github.io/metadoc/#/scalameta/scalameta/io/shared/src/main/scala/scala/meta/io/AbsolutePath.scala#L14C28-L14C40 Try "Find all references" on |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
💯 Looks great.
I agree that an 11M package index is acceptable for the megasite, especially considering that this corpus size was not really manageable before.
@@ -262,7 +263,7 @@ lazy val noPublish = allSettings ++ Seq( | |||
publishArtifact := false | |||
) | |||
|
|||
addCommandAlias("ci-test", ";compile ;test") | |||
addCommandAlias("ci-test", ";compile ; metadoc-site ; test") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needed for the JS test to inflate index.workspace.gz
with pako
_ = require(response.status == 200, s"${response.status} != 200") | ||
buffer <- response.arrayBuffer().toFuture | ||
_ = println(buffer.byteLength) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Debug?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed
for { | ||
response <- dom.experimental.Fetch.fetch(url).toFuture | ||
_ = println(JSON.stringify(response)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Debug?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed
|
||
object MetadocFetch { | ||
|
||
def symbol(symbolId: String): Future[Option[d.SymbolIndex]] = { | ||
val url = "symbol/" + JSSha512.sha512(symbolId) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this mean we can remove the js-sha512 NPM dependency and the Scala.js binding?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch, I've removed the dependency now 👍
import scalapb.Message | ||
|
||
object GeneratedSiteEnrichments { | ||
implicit class XTensionGeneratedMessageCompanion[ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The naming convention is to prefix with Xtension...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed!
@@ -42,9 +44,9 @@ class JsonSuite extends FunSuite with DiffAssertions with BeforeAndAfterAll { | |||
val symbols = FileIO | |||
.listAllFilesRecursively(AbsolutePath(out).resolve("symbol")) | |||
.toList | |||
assert(symbols.length == 2) | |||
assert(symbols.length == 1) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One file per package. 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The symbol index is a lot nicer to browse locally now
out/symbol
├── C
│ └── package.symbolindexes.gz
├── _empty_
│ └── package.symbolindexes.gz
├── advanced
│ └── package.symbolindexes.gz
├── annot
│ └── package.symbolindexes.gz
├── banana
│ └── rule
│ └── package.symbolindexes.gz
├── classes
│ └── package.symbolindexes.gz
├── com
│ └── geirsson
│ ├── mutable
│ │ ├── package.symbolindexes.gz
│ │ └── unsafe
│ │ └── package.symbolindexes.gz
│ └── package.symbolindexes.gz
├── docs
│ └── package.symbolindexes.gz
├── example
│ └── package.symbolindexes.gz
├── fix
│ └── package.symbolindexes.gz
├── flags
│ └── p
│ └── package.symbolindexes.gz
├── locals
│ └── package.symbolindexes.gz
├── objects
│ └── package.symbolindexes.gz
├── org
│ └── scalameta
│ ├── adt
│ │ └── package.symbolindexes.gz
│ ├── collections
│ │ └── package.symbolindexes.gz
│ ├── data
│ │ └── package.symbolindexes.gz
│ ├── explore
│ │ └── package.symbolindexes.gz
│ ├── internal
│ │ └── package.symbolindexes.gz
│ ├── invariants
│ │ └── package.symbolindexes.gz
│ ├── overload
│ │ └── package.symbolindexes.gz
│ ├── package.symbolindexes.gz
│ └── tests
│ └── package.symbolindexes.gz
├── prefixes
│ └── package.symbolindexes.gz
├── scala
│ ├── meta
│ │ ├── classifiers
│ │ │ └── package.symbolindexes.gz
│ │ ├── cli
│ │ │ └── package.symbolindexes.gz
│ │ ├── common
│ │ │ └── package.symbolindexes.gz
│ │ ├── contrib
│ │ │ ├── equality
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── implicits
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── instances
│ │ │ │ └── package.symbolindexes.gz
│ │ │ └── package.symbolindexes.gz
│ │ ├── dialects
│ │ │ └── package.symbolindexes.gz
│ │ ├── inputs
│ │ │ └── package.symbolindexes.gz
│ │ ├── interactive
│ │ │ └── package.symbolindexes.gz
│ │ ├── internal
│ │ │ ├── classifiers
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── classpath
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── cli
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── dialects
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── inputs
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── io
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── javacp
│ │ │ │ ├── asm
│ │ │ │ │ └── package.symbolindexes.gz
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── metacp
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── metai
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── metap
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── package.symbolindexes.gz
│ │ │ ├── parsers
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── platform
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── prettyprinters
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── quasiquotes
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── scalacp
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── scalafix
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── semanticdb
│ │ │ │ ├── javac
│ │ │ │ │ ├── package.symbolindexes.gz
│ │ │ │ │ └── semantics
│ │ │ │ │ └── package.symbolindexes.gz
│ │ │ │ ├── package.symbolindexes.gz
│ │ │ │ └── scalac
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── semanticidx
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── symtab
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── tokenizers
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── tokens
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── transversers
│ │ │ │ └── package.symbolindexes.gz
│ │ │ └── trees
│ │ │ └── package.symbolindexes.gz
│ │ ├── io
│ │ │ └── package.symbolindexes.gz
│ │ ├── metacp
│ │ │ └── package.symbolindexes.gz
│ │ ├── metai
│ │ │ └── package.symbolindexes.gz
│ │ ├── metap
│ │ │ └── package.symbolindexes.gz
│ │ ├── package.symbolindexes.gz
│ │ ├── parsers
│ │ │ └── package.symbolindexes.gz
│ │ ├── prettyprinters
│ │ │ └── package.symbolindexes.gz
│ │ ├── quasiquotes
│ │ │ └── package.symbolindexes.gz
│ │ ├── testkit
│ │ │ └── package.symbolindexes.gz
│ │ ├── tests
│ │ │ ├── contrib
│ │ │ │ └── package.symbolindexes.gz
│ │ │ └── prettyprinters
│ │ │ └── package.symbolindexes.gz
│ │ ├── tokenizers
│ │ │ └── package.symbolindexes.gz
│ │ ├── tokens
│ │ │ └── package.symbolindexes.gz
│ │ ├── transversers
│ │ │ └── package.symbolindexes.gz
│ │ └── trees
│ │ └── package.symbolindexes.gz
│ └── tools
│ └── nsc
│ └── typechecker
│ └── package.symbolindexes.gz
├── scalafix
│ ├── cli
│ │ └── package.symbolindexes.gz
│ ├── config
│ │ └── package.symbolindexes.gz
│ ├── docs
│ │ └── package.symbolindexes.gz
│ ├── internal
│ │ ├── config
│ │ │ └── package.symbolindexes.gz
│ │ ├── diff
│ │ │ └── package.symbolindexes.gz
│ │ ├── interfaces
│ │ │ └── package.symbolindexes.gz
│ │ ├── jgit
│ │ │ └── package.symbolindexes.gz
│ │ ├── patch
│ │ │ └── package.symbolindexes.gz
│ │ ├── reflect
│ │ │ └── package.symbolindexes.gz
│ │ ├── rule
│ │ │ └── package.symbolindexes.gz
│ │ ├── testkit
│ │ │ └── package.symbolindexes.gz
│ │ ├── tests
│ │ │ └── utils
│ │ │ └── package.symbolindexes.gz
│ │ ├── util
│ │ │ └── package.symbolindexes.gz
│ │ ├── v0
│ │ │ └── package.symbolindexes.gz
│ │ └── v1
│ │ └── package.symbolindexes.gz
│ ├── lint
│ │ └── package.symbolindexes.gz
│ ├── package.symbolindexes.gz
│ ├── patch
│ │ └── package.symbolindexes.gz
│ ├── reflect
│ │ └── package.symbolindexes.gz
│ ├── rule
│ │ └── package.symbolindexes.gz
│ ├── syntax
│ │ └── package.symbolindexes.gz
│ ├── test
│ │ └── package.symbolindexes.gz
│ ├── testkit
│ │ └── package.symbolindexes.gz
│ ├── tests
│ │ ├── cli
│ │ │ └── package.symbolindexes.gz
│ │ ├── config
│ │ │ └── package.symbolindexes.gz
│ │ ├── core
│ │ │ ├── package.symbolindexes.gz
│ │ │ └── util
│ │ │ └── package.symbolindexes.gz
│ │ ├── package.symbolindexes.gz
│ │ ├── reflect
│ │ │ └── package.symbolindexes.gz
│ │ ├── rule
│ │ │ └── package.symbolindexes.gz
│ │ ├── testkit
│ │ │ └── package.symbolindexes.gz
│ │ ├── util
│ │ │ └── package.symbolindexes.gz
│ │ ├── v0
│ │ │ └── package.symbolindexes.gz
│ │ └── v1
│ │ └── package.symbolindexes.gz
│ ├── util
│ │ └── package.symbolindexes.gz
│ ├── v0
│ │ └── package.symbolindexes.gz
│ ├── v1
│ │ └── package.symbolindexes.gz
│ └── website
│ └── package.symbolindexes.gz
├── selfs
│ └── package.symbolindexes.gz
├── test
│ ├── disableSyntax
│ │ └── package.symbolindexes.gz
│ ├── escapeHatch
│ │ └── package.symbolindexes.gz
│ ├── explicitResultTypes
│ │ └── package.symbolindexes.gz
│ ├── package.symbolindexes.gz
│ └── pkg
│ └── package.symbolindexes.gz
├── test2
│ └── package.symbolindexes.gz
├── traits
│ └── package.symbolindexes.gz
└── types
└── package.symbolindexes.gz
136 directories, 126 files
import metadoc.MetadocEnrichments._ | ||
import GeneratedSiteEnrichments._ | ||
|
||
class BaseMetadocCliSuite |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice cleanup
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the review!
@@ -262,7 +263,7 @@ lazy val noPublish = allSettings ++ Seq( | |||
publishArtifact := false | |||
) | |||
|
|||
addCommandAlias("ci-test", ";compile ;test") | |||
addCommandAlias("ci-test", ";compile ; metadoc-site ; test") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needed for the JS test to inflate index.workspace.gz
with pako
@@ -43,6 +45,7 @@ object MetadocApp { | |||
_ <- loadMonaco() | |||
workspace <- MetadocFetch.workspace() | |||
} { | |||
println(workspace.toProtoString) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed
for { | ||
response <- dom.experimental.Fetch.fetch(url).toFuture | ||
_ = println(JSON.stringify(response)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed
_ = require(response.status == 200, s"${response.status} != 200") | ||
buffer <- response.arrayBuffer().toFuture | ||
_ = println(buffer.byteLength) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed
|
||
object MetadocFetch { | ||
|
||
def symbol(symbolId: String): Future[Option[d.SymbolIndex]] = { | ||
val url = "symbol/" + JSSha512.sha512(symbolId) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch, I've removed the dependency now 👍
import scalapb.Message | ||
|
||
object GeneratedSiteEnrichments { | ||
implicit class XTensionGeneratedMessageCompanion[ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed!
@@ -42,9 +44,9 @@ class JsonSuite extends FunSuite with DiffAssertions with BeforeAndAfterAll { | |||
val symbols = FileIO | |||
.listAllFilesRecursively(AbsolutePath(out).resolve("symbol")) | |||
.toList | |||
assert(symbols.length == 2) | |||
assert(symbols.length == 1) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The symbol index is a lot nicer to browse locally now
out/symbol
├── C
│ └── package.symbolindexes.gz
├── _empty_
│ └── package.symbolindexes.gz
├── advanced
│ └── package.symbolindexes.gz
├── annot
│ └── package.symbolindexes.gz
├── banana
│ └── rule
│ └── package.symbolindexes.gz
├── classes
│ └── package.symbolindexes.gz
├── com
│ └── geirsson
│ ├── mutable
│ │ ├── package.symbolindexes.gz
│ │ └── unsafe
│ │ └── package.symbolindexes.gz
│ └── package.symbolindexes.gz
├── docs
│ └── package.symbolindexes.gz
├── example
│ └── package.symbolindexes.gz
├── fix
│ └── package.symbolindexes.gz
├── flags
│ └── p
│ └── package.symbolindexes.gz
├── locals
│ └── package.symbolindexes.gz
├── objects
│ └── package.symbolindexes.gz
├── org
│ └── scalameta
│ ├── adt
│ │ └── package.symbolindexes.gz
│ ├── collections
│ │ └── package.symbolindexes.gz
│ ├── data
│ │ └── package.symbolindexes.gz
│ ├── explore
│ │ └── package.symbolindexes.gz
│ ├── internal
│ │ └── package.symbolindexes.gz
│ ├── invariants
│ │ └── package.symbolindexes.gz
│ ├── overload
│ │ └── package.symbolindexes.gz
│ ├── package.symbolindexes.gz
│ └── tests
│ └── package.symbolindexes.gz
├── prefixes
│ └── package.symbolindexes.gz
├── scala
│ ├── meta
│ │ ├── classifiers
│ │ │ └── package.symbolindexes.gz
│ │ ├── cli
│ │ │ └── package.symbolindexes.gz
│ │ ├── common
│ │ │ └── package.symbolindexes.gz
│ │ ├── contrib
│ │ │ ├── equality
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── implicits
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── instances
│ │ │ │ └── package.symbolindexes.gz
│ │ │ └── package.symbolindexes.gz
│ │ ├── dialects
│ │ │ └── package.symbolindexes.gz
│ │ ├── inputs
│ │ │ └── package.symbolindexes.gz
│ │ ├── interactive
│ │ │ └── package.symbolindexes.gz
│ │ ├── internal
│ │ │ ├── classifiers
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── classpath
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── cli
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── dialects
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── inputs
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── io
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── javacp
│ │ │ │ ├── asm
│ │ │ │ │ └── package.symbolindexes.gz
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── metacp
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── metai
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── metap
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── package.symbolindexes.gz
│ │ │ ├── parsers
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── platform
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── prettyprinters
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── quasiquotes
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── scalacp
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── scalafix
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── semanticdb
│ │ │ │ ├── javac
│ │ │ │ │ ├── package.symbolindexes.gz
│ │ │ │ │ └── semantics
│ │ │ │ │ └── package.symbolindexes.gz
│ │ │ │ ├── package.symbolindexes.gz
│ │ │ │ └── scalac
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── semanticidx
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── symtab
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── tokenizers
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── tokens
│ │ │ │ └── package.symbolindexes.gz
│ │ │ ├── transversers
│ │ │ │ └── package.symbolindexes.gz
│ │ │ └── trees
│ │ │ └── package.symbolindexes.gz
│ │ ├── io
│ │ │ └── package.symbolindexes.gz
│ │ ├── metacp
│ │ │ └── package.symbolindexes.gz
│ │ ├── metai
│ │ │ └── package.symbolindexes.gz
│ │ ├── metap
│ │ │ └── package.symbolindexes.gz
│ │ ├── package.symbolindexes.gz
│ │ ├── parsers
│ │ │ └── package.symbolindexes.gz
│ │ ├── prettyprinters
│ │ │ └── package.symbolindexes.gz
│ │ ├── quasiquotes
│ │ │ └── package.symbolindexes.gz
│ │ ├── testkit
│ │ │ └── package.symbolindexes.gz
│ │ ├── tests
│ │ │ ├── contrib
│ │ │ │ └── package.symbolindexes.gz
│ │ │ └── prettyprinters
│ │ │ └── package.symbolindexes.gz
│ │ ├── tokenizers
│ │ │ └── package.symbolindexes.gz
│ │ ├── tokens
│ │ │ └── package.symbolindexes.gz
│ │ ├── transversers
│ │ │ └── package.symbolindexes.gz
│ │ └── trees
│ │ └── package.symbolindexes.gz
│ └── tools
│ └── nsc
│ └── typechecker
│ └── package.symbolindexes.gz
├── scalafix
│ ├── cli
│ │ └── package.symbolindexes.gz
│ ├── config
│ │ └── package.symbolindexes.gz
│ ├── docs
│ │ └── package.symbolindexes.gz
│ ├── internal
│ │ ├── config
│ │ │ └── package.symbolindexes.gz
│ │ ├── diff
│ │ │ └── package.symbolindexes.gz
│ │ ├── interfaces
│ │ │ └── package.symbolindexes.gz
│ │ ├── jgit
│ │ │ └── package.symbolindexes.gz
│ │ ├── patch
│ │ │ └── package.symbolindexes.gz
│ │ ├── reflect
│ │ │ └── package.symbolindexes.gz
│ │ ├── rule
│ │ │ └── package.symbolindexes.gz
│ │ ├── testkit
│ │ │ └── package.symbolindexes.gz
│ │ ├── tests
│ │ │ └── utils
│ │ │ └── package.symbolindexes.gz
│ │ ├── util
│ │ │ └── package.symbolindexes.gz
│ │ ├── v0
│ │ │ └── package.symbolindexes.gz
│ │ └── v1
│ │ └── package.symbolindexes.gz
│ ├── lint
│ │ └── package.symbolindexes.gz
│ ├── package.symbolindexes.gz
│ ├── patch
│ │ └── package.symbolindexes.gz
│ ├── reflect
│ │ └── package.symbolindexes.gz
│ ├── rule
│ │ └── package.symbolindexes.gz
│ ├── syntax
│ │ └── package.symbolindexes.gz
│ ├── test
│ │ └── package.symbolindexes.gz
│ ├── testkit
│ │ └── package.symbolindexes.gz
│ ├── tests
│ │ ├── cli
│ │ │ └── package.symbolindexes.gz
│ │ ├── config
│ │ │ └── package.symbolindexes.gz
│ │ ├── core
│ │ │ ├── package.symbolindexes.gz
│ │ │ └── util
│ │ │ └── package.symbolindexes.gz
│ │ ├── package.symbolindexes.gz
│ │ ├── reflect
│ │ │ └── package.symbolindexes.gz
│ │ ├── rule
│ │ │ └── package.symbolindexes.gz
│ │ ├── testkit
│ │ │ └── package.symbolindexes.gz
│ │ ├── util
│ │ │ └── package.symbolindexes.gz
│ │ ├── v0
│ │ │ └── package.symbolindexes.gz
│ │ └── v1
│ │ └── package.symbolindexes.gz
│ ├── util
│ │ └── package.symbolindexes.gz
│ ├── v0
│ │ └── package.symbolindexes.gz
│ ├── v1
│ │ └── package.symbolindexes.gz
│ └── website
│ └── package.symbolindexes.gz
├── selfs
│ └── package.symbolindexes.gz
├── test
│ ├── disableSyntax
│ │ └── package.symbolindexes.gz
│ ├── escapeHatch
│ │ └── package.symbolindexes.gz
│ ├── explicitResultTypes
│ │ └── package.symbolindexes.gz
│ ├── package.symbolindexes.gz
│ └── pkg
│ └── package.symbolindexes.gz
├── test2
│ └── package.symbolindexes.gz
├── traits
│ └── package.symbolindexes.gz
└── types
└── package.symbolindexes.gz
136 directories, 126 files
This commit implements two optimizations to make the generate site contain fewer files and be smaller in bytes. - gzip all generated files: this required changing the JavaScript client to inflate responses with the library pako (http://nodeca.github.io/pako/#inflate). I originally hoped the browser would automatically inflate the reponses us by using the `*.gz` file extension. The response headers for the webpack + browsersync + GitHub pages all return `Content-Encoding: gzip` for filenames with the `*.gz` extension. Nevertheless, the fetch response body contained deflated bytes so I resorted to inflating in JavaScript. - group symbol indexes by package. This primarily reduces the number of generated files. Most symbol indexes were only a few bytes but occupied the minimum block-size on disk which is 4kb on my macbook. For a Scalameta + Scalafix combined metadoc site, this change made the total symbol index go from taking 40mb on disk to 3.4mb! To measure these optimizations I ran metadoc against three different corpora. SemanticDB index: | Corpus | Before | After | | -------------------- | ------ | ----- | | paiges | 332kb | 112kb | | scalameta + scalafix | 24mb | 5.9mb | | megasite | 1.25gb | 209mb | Symbol index: | Corpus | Before | After | | -------------------- | ------ | ----- | | paiges | 776kb | 16kb | | scalameta + scalafix | 45mb | 888kb | | megasite | n/a | 44mb | The "megasite" corpus is a combination of 55 projects with a total of 3.5 million lines of code. Generating a metadoc site for the megasite takes ~50 seconds on my laptop.
Rebased on master and updated the demo to include scala-library https://olafurpg.github.io/metadoc/#/scala/Predef.scala#L134C8-L134C14 |
No more CI timeouts \o/ Guess not having to downloads 350mb of icons helps 😂 |
Oh, man, 🤦♂️ downloading +300Mb for one icon. What was I thinking. |
This commit implements two optimizations to make the generate site
contain fewer files and be smaller in bytes.
to inflate responses with the library pako (http://nodeca.github.io/pako/#inflate).
I originally hoped the browser would automatically inflate the
reponses us by using the
*.gz
file extension. The response headersfor the webpack + browsersync + GitHub pages all return
Content-Encoding: gzip
for filenames with the*.gz
extension.Nevertheless, the fetch response body contained deflated bytes
so I resorted to inflating in JavaScript.
generated files. Most symbol indexes were only a few bytes but
occupied the minimum block-size on disk which is 4kb on my macbook.
For a Scalameta + Scalafix combined metadoc site, this change
made the total symbol index go from taking 40mb on disk to 3.4mb!
To measure these optimizations I ran metadoc against three different
corpora.
SemanticDB index:
Symbol index:
The "megasite" corpus is a combination of 55 projects with a
total of 3.5 million lines of code. Generating a metadoc site for the
megasite takes ~50 seconds on my laptop.