Markdoc support in content collections

[bholmesdev]

Body

Summary

This is a proposal to add Markdoc support to content collections.

Background & Motivation

We've received multiple user requests for Markdoc support since its public launch. In fact, we've seen early community projects bringing Markdoc to robust project themes.

Markdoc is also designed to solve existing limitations of MDX in Astro.

1. Performance suffers at scale. Unlike plain Markdown that outputs a string of HTML, MDX outputs a JavaScript module of JSX components. This requires Astro and Babel preprocessing for even the simplest MDX documents. Notably, this required a bump to our maximum memory usage when building / deploying docs.astro.build after migrating to MDX.
2. Your content is tied to your UI. MDX can import styles and components directly, which is convenient from a developer standpoint. However, this causes issues when content needs to be reused in multiple contexts. A common example is RSS, where you may want a component-rich version from your blog and a simplified HTML output for your RSS feed.

Markdoc is built to solve (2) by separating content from the components, styles, and assets you choose to render. You can use an Astro component renderer when using on your site, Markdoc's own html renderer for RSS, and even write your own renderer to traverse Markdoc pages yourself. (1) Is something we're excited to test, requiring a thorough performance benchmark.

The content collections API was built generically to support this future, choosing format-agnostic naming like data instead of frontmatter and body instead of rawContent. Because of this, introducing new authoring formats is possible without breaking changes.

Goals

• Create an @astrojs/markdoc integration that adds .mdoc support to content collections.
• Support Astro components and server-rendered UI components (React, Vue, Svelte, etc) within Markdoc files. Note this excludes client-rendered UI components (see non-goals).
• Benchmark Markdoc performance against MDX at 1000+ documents. This addresses problem (1) from the previous section. Metrics to compare: SSG build time, SSG and SSR build memory usage, SSR response speed.

Non-goals

• ESM import and src/pages/ support for Markdoc files. See discussion for context.
• Allowing the .md extension. This would mean overriding Astro's .md renderer, which is tightly coupled to remark and your markdown configuration options today. We agree using .md for Markdoc is a common use case, and deserves a separate proposal to make Astro's Markdown rendering flexible.
• A solution for client-rendered UI components. Unlike MDX, Markdoc doesn't have a concept of directives, and our compiler doesn't have a clear way to dynamically render client-side components (see challenges). We will recommend users wrap their components in an Astro component to apply the client: directive.
• Full alignment with Markdown and MDX rendered result. Namely, the computed headings property (which can be tackled in future releases) and frontmatter manipulation via remark (since remark is incompatible with Markdoc).

Example implementation

Markdoc will be introduced as an integration. To standardize our process for adding new collection teams, we may experiment with a (private) integration helper internally. This example shows an addContentEntryType hook to setup the .mdoc extension, and attach logic for parsing the data and body properties:

// @astrojs/markdoc/index.ts
export const markdoc: AstroIntegration = () => ({
		'astro:config:setup'({ addContentEntryType }) {
			addContentEntryType({
				extensions: ['.mdoc'],
				parser: '@astrojs/markdoc/contentEntryParser',
			});
		}
	}
});

// @astrojs/markdoc/contentEntryParser.ts
import parseFrontmatter from 'gray-matter';
export default {
	getEntryInfo({ contents }) {
		const parsed = parseFrontmatter(contents);
		return {
			// The unparsed data object that can be passed to a Zod schema.
			data: parsed.data,
			// The body of the data file. This should be the raw file contents with metadata (i.e. frontmatter block) stripped
			body: parsed.content,
			// (Optional) The untouched frontmatter block with newlines and formatting preserved. Used for computing error line hints.
			rawData: parsed.matter,
		}
	}
}

// astro.config.mjs
import markdoc from '@astrojs/markdoc';

export default {
	integrations: [markdoc()],
}

Example Usage

Say you've authored a collection of blog posts using Markdoc. You can store these entries as a blog collection, identically to Markdown or MDX:

src/content/
	blog/
		# Could also use `.md`
		post-1.mdoc
		post-2.mdoc
		post-3.mdoc
...

Then, you can query entry frontmatter with the same getCollection() and getEntryBySlug() APIs:

---
import { getCollection, getEntryBySlug } from 'astro:content';

const blog = await getCollection('blog');
const firstEntry = await getEntryBySlug('blog', 'post-1');
---

Users should also be free to render Markdoc contents using a Content component. This will be exposed from the render() result, and feature two props:

• components?: Record<string, ComponentRenderer>: A mapping from Markdoc tags or elements to Astro components.
• config?: import('@markdoc/markdoc').Config: An (optional) Markdoc config to be used during the transformation step.

---
import Title from '../components/Title.astro';
import Marquee from '../components/Marquee.astro';
import { getEntryBySlug } from 'astro:content';

const mdocEntry = await getEntryBySlug('blog', 'test');
const { Content } = await mdocEntry.render();
---

<html lang="en">
	<body>
		<Content
			config={{
				variables: { underlineTitle: true },
			}}
			components={{
				h1: Title,
				marquee: Marquee,
			}}
		/>
	</body>
</html>

Sharing config

This solution is flexible, but we expect users to reuse config and components across their project. For this, we will recommend creating a utility component to encapsulate that config. Here is one example that can render any blog collection entry with an {% aside /%} shortcode:

---
// src/components/BlogContent.astro
import Aside from './Aside.astro';
import type { CollectionEntry } from 'astro:content';

type Props = {
	entry: CollectionEntry<'blog'>;
};

const { entry } = Astro.props;
const { Content } = await entry.render();
---

<Content
	config={{
		tags: {
			aside: {
				render: 'Aside',
				attributes: {
					type: { type: String },
					title: { type: String },
				},
			},
		},
	}}
	components={{ Aside }}
/>

Now, you can pass any blog collection entry to render the result with this config:

---
import { getEntryBySlug } from 'astro:content';
import BlogContent from '../components/BlogContent.astro';

const mdocEntry = await getEntryBySlug('blog', 'test');
---

<h1>{intro.data.title}</h1>
<BlogContent entry={mdocEntry} />

See this example video for more.

Advanced use case: component prop mapping

Component renderers can also include a props() function to map Markdoc attributes and AST entries to component props. This is useful when:

• computing props based on the Markdoc AST
• mapping Markdoc's generated attributes to prop names

This example maps Markdoc's generated data-language attribute for code blocks to the lang prop used by Astro's Code component, and stringifies the contents to HTML for use with Shiki:

---
import { Code } from 'astro/components';
import { Title } from '../components/Title.astro';
import Markdoc from '@markdoc/markdoc';
...
---

...
<Content
	components={{
		h1: Title,
		pre: {
			component: Code,
			props({ attributes, getTreeNode }) {
				return {
					lang: attributes['data-language'],
					code: Markdoc.renderers.html(getTreeNode().children),
				};
			},
		},
	}}
/>

[FredKSchott]

Love this! Some thoughts:

• "Support for using the .md extension for Markdoc." Not sure if this has already been discussed and I missed it, but I think there's a good argument for markdoc to become the default .md renderer eventually, with support to opt-in to remark for those who want it. Something like mdDefaultParser: 'markdoc' | 'remark' could work, idk. BUT, I can see why you would want to treat that as out of scope, since it would be the same amount of work but one less thing to worry about here.
• I have a lot of thoughts about the components story here, more than I can lay out in a GH comment. If you're open to it, I'd love to grab some time to walk you through that 1:1.

[bholmesdev]

I agree with each of these! To address individually:

1. Yes, I think more users will expect Markdoc to support .md compared to MDX users. Markdoc's own documentation mentions using the .md extension as well. I think adding a way to override Astro's Markdown parser from an integration would be a good addition to this proposal 👍
2. We discussed offline! To summarize:

• I think we'll stick to Markdoc config and components as <Content /> component props. For Markdoc properties like variables, it makes sense to allow configuration per-entry instead of globally or per-collection. I feel similarly about components to ensure content is reusable in multiple UI contexts. What's more, keeping config at the Content component level will let users experiment with global config patterns so we can refine in the future. See the last segment of the demo video for one example.
• There is a compelling use case for auto-wiring components to Markdoc tags. This could mean a reserved directory of components Astro can make available in your content, like a src/embeds/ directory. Ex. use an {% embed:Aside /%} tag in Markdoc, and render a src/embeds/Aside.astro on the other end. This saves the user from configuring the tag and adding the component by hand (which can get complicated!) This feels like a future enhancement we can introduce.

[bholmesdev]

Update: added .md support to the goals section and Example Usage. I won't outline an example implementation just yet so we have room to experiment. This is only Stage 1!

[matthewp]

I think having that as an explicit goal while having support for src/pages and imports as a non-goal are conflicting. I can't imagine us allowing .md to mean different things depending where the files exist in the folder structure. I would recommend either dropping this as a goal or dropping src/pages and imports as a non-goal. That way you can figure out during implementation how hard it will be to support Markdoc everywhere, or whether that's a feature to defer for later.

[bholmesdev]

Why not support src/pages/?

Some users may expect Markdoc support in src/pages/. I've left this as a non-goal for a couple reasons:

• Pages support is more work for what we'd consider a secondary use case (scope creep)
• Pages perpetuates an old pattern we may move away from: the layout prop. This is a necessary evil when supporting content types that can't import a layout explicitly. As we've seen, Markdown layouts require a special content prop convention, losing out on type safety due to indirect prop passing. Content collections negates the need for this pattern.

Given this, Markdoc may be a good chance to experiment with content only formats that do not exist as standalone pages.

Why not support ESM imports?

This was mainly excluded because supporting ESM and not src/pages/ feels inconsistent; these two features are a package deal in most users' minds given Markdown and MDX support. There's also the issue of astro/client types, where we'd need to add a MarkdocInstance type to Astro core for supporting *.mdoc files. If we support .md files via Markdoc as well, this gets much more difficult to reconcile (i.e. if I import .md, is that Astro's .md or Markdoc's?).

[matthewp]

I think having .md mean one thing in the src/content/ folder and mean something different in src/pages/ is pretty confusing. I agree completely with the above reasons for not supporting Markdoc outside of content. So we might be able to get away with this by disallowing .md files outside of src/content/ as one possible solution.

[bholmesdev]

Understood. Discussing with core, we decided moving to non-goals is the best idea. Add a note for future exploration in the non-goals as well!

[Princesseuh]

As a editor tooling author, I'm super biased against formats using other formats file extensions. If ex: a VS Code extension were to be created for Markdoc, every single project would need to configure their editor to recognize .md files as Markdoc. I think it's a terrible experience and so I'm definitely +1 on not supporting it. (I don't agree with them recommending to do that..)

Apart from that, not much to comment on, I love the pattern with the wrapper component for using a custom config! Very Astro-y!

[matthewp]

Mostly super excited about this. Love that this proposal has very explicit goals and non-goals that allow this to be introduced at a smaller scope to start (such as only being available within content collections).

One question,

A solution for client-rendered UI components.

Would this not work?

<Content
  Aside={((props) => {
    return (
      <Aside client:load {...props} />
    )
  })}
/> 

I think the compiler is smart enough to pick that up. If that doesn't work it's worth investigating.

Not a blocker at this point, but I do think that one of the major use-case people have for components in markdown is to add interactive components to posts. I think we can figure out how to get it to work.

[delucis]

Questions about the DX of custom components

How does a developer know they are passing the right components to <Content />?

I noticed there was no autocomplete, and I can imagine it’s hard or impossible to statically analyse the components each Markdoc document is using to generate those types (or maybe I’m wrong and their compiler extracts that somehow!)

So that made we wonder:

• If a component is not provided, is an error thrown?
  • If no, how would I know if at some point I removed a component one of my documents was using?
  • If yes, when?
    • I know there’s been some discussion around runtime rendering so if this error happens only at runtime when one specific document is rendered/requested that would jar with the rest of our type-safe content story.

How does a developer/author know they are using the right shortcodes in a document?

Is there editor tooling etc. that allow an author to know that Aside is a valid component shortcode, but Adise is not? This seems like something that the local import model of MDX/AFMD made relatively straightforward because each component usage documented the source of the component. We’re obviously intentionally moving away from that with Markdoc but curious how that DX story translates.

[matthewp]

This proposal has been moved to stage 2: #496

Please direct any feedback / comments to that issue now, thanks!