[wiki] Project Lightspeed

[SamChou19815]

Background

As noted in my open source roadmap, the current Wiki setup is far from ideal. For documentation purpose, let me list the architecture and the build system as the time the issue was created:

• Wiki is a standard Docusaurus project. It follows the standard convention of Docusaurus project layout.
• Wiki has a public build and private build.
  • The public build is self-contained in the repo. Everything need for creating a public repo is already checked in.
  • The private build is NOT self-contained. I hacked together some tools that automatically checkout my private-monorepo in CI, copying a bunch of files, doing a bunch of codegen, and then invoke docusaurus build. The build time is also significantly slowed down, because a lot of markdown files are pulled in.
• Wiki has a public deployment and private deployment.
  • Each deployment correspond to each build described above.
  • Public deployment works the same as the rest of the projects: calls the build script and upload files to Firebase hosting.
  • Private deployment deploys the code to an Google App Engine instance, which is guarded by Google Cloud IAP. The IAP puts a blanket login guard before all possible routes, and I configured in a way that only my main email can pass the check.
  • Deployment to GAE is at least two times slower than deployment to firebase hosting, even though in both cases only static files are deployed.

Pain Points

• Slow build time
• Slow deployment time
• Complex setup
• Mandates cross-repo tooling

Thoughts

When initially designing the system, I seem to limit myself to Docusaurus. To be fair, Docusaurus is quite nice, but we have to accept the fact that it's primary use cases is for static sites. To some extent, a static site does make sense for the documents currently used for a private build. However, limiting ourselves to static site solution unnecessary constrained the architecture choices.

Instead, we could accept the outcome that we lose some of Docusaurus benefits, including

• A lot of nice markdown syntax extension
• Working out-of-box docs plugin
• Painless routing
• etc

, we can design a much more scalable solution. Initial exploration in #790 shows that we can use remarkable to dynamically render markdown with the same feature support that will likely to be used by myself.

Therefore, the wiki project will go dynamic. However, it doesn't mean that we have to completely drop Docusaurus for this. The static public docs can still be served by Docusaurus, and the dynamic content could just be another Docusaurus page. What I will happily give up is the ability to throw a markdown and let Docusaurus handle the rest.

New Architecture

Static public docs will remain fully managed by Docusaurus.

Private docs will no longer be pulled from a repo and built by Docusaurus. Instead, they will be stored in a firebase database, and fetched to frontend on demand. I will use remarkable to render markdown on the frontend.

On the backend, the private docs is protected by firebase security rules. Each document should have a permission tag attached to it, so that permission control can become more flexible in the future. (e.g. Currently it's all or nothing, but in the future I can grant a subset of document to some people, without exposing the rest)

[SamChou19815]

TODO

• Implement markdown editor with preview as a proof-of-concept.
• Introduce firebase to the repo again as a login and database system
• Support rendering a list of documents to be read according to permission and tags
• Support editing documents on the frontend (only for @SamChou19815)
• Support granting document access to specific people