Scalability issues migrating from v1 to v2

[allardy]

Hello ! We are in the process of migrating from v1 to v2, and we ran into performance issues while building and using the site.

Basically, our documentation must follow the version of our application, even minor or patches. We had no problem with docusaurus 1 because the build step generated a simple versioned docs folder, but now the build process takes forever.

We currently have 131 versions, with around 50k files, the registry weights 5mb, the build time & mem consumption is high :/ Couldn't we just load the latest versions in memory, and have a dynamic routing which would try to resolve older ones on-demand?

I can't seem to find the best way to go forward, we could "segment" the versions into "groups" to make the build quicker, but it wouldn't be very friendly (and may impact seo), plus having to manage said groups.. Otherwise, being able to "archive" a version somehow and remove it from the react site (while keeping it available) could be interesting...

Our documentation is open source: https://botpress.com/versions
And the migration to v2: https://github.com/botpress/docs/tree/DOC-68-Migrate-Docusaurus-from-V1-to-V2

There was a previous discussion with similar scaling issues (#3132) but I would like to have some inputs on our situations.

Thanks

[armano2]

related to #4765

[allardy]

I've read that proposal, still didn't get an acceptable build time. I don't see why we couldn't just stop processing already older, versioned docs instead.. I mean, once they're converted as react components, we could just upload the resulting to the repository to avoid re-processing old ones

[slorber]

There's a docs option onlyIncludeVersions to filter versions to not include.

I mean, once they're converted as react components,

Webpack 5 persistent caching will cache the process of converting to React components

we could just upload the resulting to the repository to avoid re-processing old ones

The output is static HTML + JS for the SPA, not just compiled React components.

Webpack always has to assemble the app as a single-page application in the end.

[Josh-Cena]

You should build each version as a separate site and use external links to link to them. Docusaurus uses this trick for any older version:

docusaurus/website/docusaurus.config.js

Lines 382 to 398 in 813300b

	dropdownItemsAfter: [
	...ArchivedVersionsDropdownItems.map(
	([versionName, versionUrl]) => ({
	label: versionName,
	href: versionUrl,
	}),
	),
	{
	href: 'https://v1.docusaurus.io',
	label: '1.x.x',
	},
	{
	to: '/versions',
	label: 'All versions',
	},
	],
	},

For example, if you use Netlify / Vercel, you can just pick an immutable deployment URL, and remove the version's docs from your site source. Keep your active versions at around 5.

[allardy]

So basically, what you suggest is that we keep only the current version on the documentation repository (master branch), automatically deploy it to netlify on changes, and when we're ready to deploy the next, newer version, "freeze it" on netlify then start a new one for the current version?

Or... Maybe we could have a branch for each version deployed on netlify, and keep master for the current version..?

[slorber]

Not necessarily just one version on master/main, but just the versions you are actively working on.

Having active versions on main branch is convenient particularly if you need to backport features and docs to older versions. Once a version/doc becomes unmaintained anymore, it seems convenient to archive it and reduce build time.

Or... Maybe we could have a branch for each version deployed on netlify, and keep master for the current version..?

That's what versioned tools do traditionally, and it remains a good approach. You can setup one distinct Netlify deployment per branch with its own subdomain.

The problem is that it makes it harder to backport features/docs as you now have to create one PR for each version or things like git cherry-pick

[slorber]

Docusaurus 2 builds a real SPA while v1 only used React on the frontend. Indeed our Webpack/Babel/MDX setup is heavier and can it can take more time to build your site.

I'd like to write some doc about build time scalability, but yes ultimately v2 is going to be slower than v1 for the same amount of pages.

As @Josh-Cena mentioned you can archive older versions to standalone Jamstack immutable deployments, we do this on the Docusaurus site itself.

nb of docs * versions * i18n locales can lead to large build times. All versions/locales do not need to be in the same SPA, and it's probably ok to have a regular HTML navigation when switching version/locale, so you shouldn't be afraid to "shard" your site.

Also we have Webpack 5 persistent caching, if you setup your CI to cache node_modules/.cache/webpack across builds, it will be faster on rebuilds (the easiest is to try this locally first)

[allardy]

Looks like sharding or individual site deployment are the most appropriate solutions. It just involves a bit more steps than I thought. I understand that Docusaurus is going away from its initial documentation mission, so I guess we shouldn't expect some type of "dynamic" loading of older versions when a page is requested.

I think = we'll leave the currently versioned docs to docusaurus 1 and see how we can put in place an effective deployment workflow with v2 that will scale with releases

[slorber]

Yes 👍

We'll definitively try to improve Docusaurus build times in the future.
It's not the main focus atm although we try to implement the low-hanging fruits.

The ecosystem is moving to a new generation of Rust/Go/Zig bundler/tools, faster than Webpack/Babel. We should adopt one of them once we know for sure which tool is best for our use-case.