Runtime content rendering

[matthewp]

Body

Summary

It should be possible to render content, that is markdown and perhaps other formats (MDX, markdoc) at runtime rather than bundled with the larger app.

Background & Motivation

There are 2 primary reasons to consider doing this:

1. Performance - Compiling markdown and other content formats to JavaScript and then bundling it with the actually-dynamic parts of your app doesn't scale well. Rollup in particular, but really any bundler, uses a lot of memory the more stuff is added. Some formats do better or worse. MDX compiles to JSX, so that results in more AST nodes so it scales worse.
2. CMS - When using a CMS you want to be able to add content to your site without checking it in to a git repository and then rebuilding your site. You want to save it to a database and then pull it back out at runtime.

Goals

• Include a minimal markdown parser in Astro that can be used at runtime.
• Make it possible to mark a content collection as not reading from the filesystem. An API to allow non-FS based collections is needed.
• Use the runtime markdown compiler on non-FS collections.
• Have an opt-in way to use the runtime compiler on FS collections.

Example

Unknown at this time.

[bholmesdev]

Thanks for starting this discussion Matthew! I'd love to explore this idea. I think this split could improve our memory usage when building large Markdown and MDX sites as well. Instead of doing all the work during our SSR bundling step, we can defer that work to route generation for static builds (or to your server in SSR).

[bluwy]

Would this mean the user has to build the pages on-the-fly to get the benefits? e.g. SSR? IIUC if they're rendering all the pages as static HTML files at once, it would still consume a large amount of memory.

I do like that there would be a way to offload the build burden though.

[bholmesdev]

There's a chance the memory load is offset but avoiding the huge getStaticPaths bundling step though. I'd be curious to investigate

[matthewp]

It can still consume a lot of memory in SSG, that's true. There is no silver bullet, but we can tune this better and avoid Rollup being a single point of failure. For example we can:

• Inject the actual markdown into the bundle during the generateBundle step so that Rollup never has to parse it at all.
• Use (or build) a markdown compiler that has a low memory footprint.
• I'm hoping that by compiling it during the render phase that if you are build 10k pages, GC will occur periodically while this is happening, since pages are built one at a time. maybe this is a wrong assumption though.

[bluwy]

From the recent Astro docs build discussion, Rollup builds was taking around 2 out of a 6 min build, so I think markdown compilation was still fairly fast during the build phase, and the main blocker is in the rendering phase. I haven't dig too deep yet, but at the moment this RFC might not help yet till we figure out what's blocking up the rendering phase 🤔

[matthewp]

Wow, that's great news actually.