Content Layer

[matthewp]

• Accepted Date: 2024-06-07
• Reference Issues/Discussions: Content Layer #935
• Author: @FredKSchott
• Champion(s):
• Implementation PR:

Summary

• Explore a new and improved content layer for Astro.
• Improve the current experience of loading/defining data into content collections
• Improve the current experience of querying data from content collections

Background & Motivation

Content Collections are a key primitive that brings people to Astro. Content Collections make it easy to work with local content (MD, MDX, Markdoc, etc) inside of your Astro project. They give you structure (src/content/[collection-name]/*), schema validation for frontmatter, and querying APIs.

Goals

• Explore a new and improved content layer for Astro.
• Improve the current experience of loading/defining data into content collections
• Improve the current experience of querying data from content collections

Example

// A folder full of Markdown (MDX) files
defineCollection({
    name: 'blog',
    data: glob('./content/blog/*.mdx'),
  });
// A single file containing an array of objects
defineCollection({
    name: 'authors',
    data: file('./content/authors.json'),
});
// Remote data, loaded with a custom npm package
defineCollection({
    name: 'articles',
    data: storyblokLoader({startsWith: 'articles/posts'}),
});
// Custom data, loaded from anywhere you'd like
defineCollection({
    name: 'my-custom-collection',
    data: () => { /* ... */ },
});

[florian-lefebvre]

I think it would be nice to support singletons, as suggested in #449 and #806. Many CMSs support those (eg. Keystatic, the content layer seems like it would fit well)

[lloydjatkinson]

Just wondering how this will work (if at all) for components, like in MDX? So the use case I'm thinking is when a CMS is used, they typically have a WYSIWYG rich text editor where custom components can be inserted as shown here: https://www.storyblok.com/tp/create-custom-components-in-storyblok-and-astro

Will this new API support this concept?

[xavdid]

This is cool!

I do a sort-of version of this for my review site (david.reviews), where all the data is stored in Airtable.

I load objects from the Airtable API based on a schema and cache responses in JSON locally (since loading and paging takes a while and is too slow for local development). It sort of feels like what an API-backed custom content collection could look like.

The whole thing is strongly typed, which is cool! I wrote about it in more detail here: https://xavd.id/blog/post/static-review-site-with-airtable/

[JacobNWolf]

Is this enacted somewhere yet/available as an experimental feature or on a beta version?

I've been trying to build collections from WordPress content fetched via GraphQL queries and I think this'll fix exactly what I want.

[rambleraptor]

How would this work with dependency files like images?

Right now, the src/content folder contains MDX files and can contain image files referenced from those MDX files.

I'm personally excited to separate my content folder and astro theme into separate GitHub repos, so that's where my my perspective comes from.

[ashhitch]

Coming from Gatsby and loving the data layer.

A few points that I always found hard:

• Integrations with providers not maintained by the core team
• Preview draft CMS content
• Linked content, e.g blog listing, or cross linking component not updating when new Article added (more todo with incremental updates)

[brian-montgomery]

I like the idea of being able to store content in a single file. I explored adding CSV support to the existing content collection APIs, and found there were too many assumptions around a directory of files. Providing a higher-level of abstraction around how the data is accessed/retrieved while keeping the simple APIs and excellent typing would be ideal.

In essence, separating the client usage (the query APIs, generated schemas, metadata (frontmatter) and representation) from how the source is retrieved (single file, flat-file directory structure, database, remote service) would be really helpful.

[ascorbic]

Is this enacted somewhere yet/available as an experimental feature or on a beta version?

No, still at the planning stage right now. We'll share experimental builds once they're available.

[NuroDev]

Love the initial look of this.

Will is still be possible to include a (zod) schema of some kind in the defineCollection function so if you are fetching data from a remote source it can be validated before being published?

[ascorbic]

Love the initial look of this.

Will is still be possible to include a (zod) schema of some kind in the defineCollection function so if you are fetching data from a remote source it can be validated before being published?

Yes, we'd want that to generate types too. Hopefully content sources could define this automatically in some scenarios.

[stefanprobst]

my wishlist for astro content layer:

• support for more complex collection schemas, for example multiple richtext/mdx fields per collection entry (e.g. a "summary" and "content" field). specifically i would love support for everything i can express with keystatic's collection schema.

• support for singletons (single-entry collections)

• draft mode / preview mode (should work with "hybrid" rendering, and should work with node.js adapter)

[louiss0]

Is Zod still going to be the official validator for Content Collections or can people use something else.

[reasonadmin]

Love this concept!

In the current implementation entries from a collection have a render function (and an undocumented function in Astro addContentEntryType for adding new types).

Is there a spec yet for what the data property has to be? It would be nice to remove the limitation of only having a single render function that has to return a string of fully rendered contents. Being able to manipulate data here might bring in interesting UnifiedJS style options as any data can be converted into an AST and then any Unified plugin could run against it.

For example, you could load data from a CSV into a MDAST with a table structure - and then it would render the CSV as if it had been created as a table in markdown (only the data is much more manageable in the CSV for large data sets).

[ascorbic]

@reasonadmin I think support for implementing custom renderers is a must, and it would then make sense to allow these to be more flexible than a single render() function. Perhaps allow it to accept arguments, which could specify different fields to render, or render options. For different filetypes I think it would make sense for them to be implemented as separate integrations, so something like:

defineCollection({
    name: 'accounts',
    data: csvLoader("data/accounts/**/*.csv"),
});

..and then:

import { getEntry, getEntries } from 'astro:content';

const account = await getEntry('accounts', '2024-20');

// Access the parsed CSV data
const { rows } = account;

// Render a table. Pass options to render, or maybe make them props for `<Content />`
const { Content } = account.render({ /* typesafe filter, sort options etc */ });

[reasonadmin]

@ascorbic Is it possible to join these two ideas together:

#763

For example:

defineCollection({
  name: 'my-data',
  data: async (db: DB, watcher: FileSystemWatcher) => {
    const hash = sha256(content);
    await db.addContent(hash, content);
    // Only build updates to files if the hash is different from the one in the DB
    // DB is accessible between static builds for incremental SSG rendering
  },
})

How about something like this for rendering:

defineCollection({
    name: 'accounts',
    data: Astro.autoDBWrapper(   [{page: 1},{page: 2}]   ),
    render: async(entry, options) => {
      //Return an object that has minimum fields {metadata : Object , content : String}
      //As this content will not be directly loaded via an import statement (e.g. const {Content} = import ('myfile.md');
      //We don't need a loader and therefore don't need to Stringify things as JS for the loader?
    }
});

/* -- */

import { getEntry } from 'astro:content';
const { entry, render } = await getEntry('accounts', '2024-20');

const {metadata, content} = render(entry, { 'fileType: 'HTML' });
const {metadata, content} = render(entry, { 'fileType: 'XML' });

[ArmandPhilippot]

In addition to the proposed singletons, here my thoughts.

Rich query API

I like the ideas proposed in #574 or #518.

Here another format possible:

const welcomePost = await queryEntry('posts', {where: {slug: "welcome", lang: "en"}});

const frenchPosts = await queryCollection("posts", {
  first: 10,
  where: { lang: "fr" },
  sort: {key: "publicationDate", order: "DESC"},
});

Sub-collections / Nested data-types

Idea

It would be nice to allow "sub-collections". The idea would be to improve content organization and to share some commons data-types between a same collection while sub-collections could have additional data-types.

Then we could:

• get the collection with an union of the different sub-collections types with await getCollection('collectionName')
• get a sub-collection directly with await getSubCollection('collectionName', 'subCollectionName').

Example

Maybe an example will help describe my proposal, so imagine a collection named posts. A post could have different "formats", for example: "changelog" (new features on the website), "tutorials" (some tutorials about software) and "thoughts" (for everything else).

All the formats share common data-types:

• a publication date
• a status (draft/published)

Then each format could have additional data-types:

• A "changelog" does not need anything else,
• A "tutorial" could have those additional data-types:
  • software
  • difficulty
• A "thought" could have:
  • main subject/category

The collection could be defined as follow:

const posts = defineCollection({
  type: 'content',
  schema: z.object({
    isDraft: z.boolean(),
    publicationDate: z.string().transform((str) => new Date(str)),
  }),
  subCollections: {
    changelog: {
      type: 'content',
    },
    thought: {
      type: 'content',
      schema: z.object({
        subject: z.string()
      }),
    },
    tutorial: {
      type: 'content',
      schema: z.object({
        difficulty: z.enum(["easy", "medium", "hard"]),
        software: z.string(),
      }),
    },
  }
});

export const collections = {
	posts
}

The generated types would be:

type Changelog = {
	isDraft: boolean;
	publicationDate: Date;
	subCollection: "changelog";
}

type Tutorial = {
	isDraft: boolean;
	publicationDate: Date;
	subCollection: "tutorial";
	software: string;
	difficulty: "easy" | "medium" | "hard"
}

type Thought = {
	isDraft: boolean;
	publicationDate: Date;
	subCollection: "thought";
	subject: string;	
}

type Post = Changelog | Tutorial | Thought;

When validating data-types, an error is thrown with the following examples:

• when any sub-collection is missing one of the common data-types (isDraft and/or publicationDate)
• if a "changelog" has unexpected keys like software or subject
• if a "thought" is missing the subject key

If the subCollection key is not defined, Astro will behave like it does currently.

Then it would be possible to get all the posts (with mixed formats) using await getCollection("posts") to display them in a page (like a blog page). The consumer could then use a different presentation depending on the "format".

It would also be possible to query a sub-collection directly with, for example, await getSubCollection("posts, "tutorial") and display only the tutorials.

For the organization, I don't know what would be best:

• a flat structure, the subCollection key in the frontmatter of Markdown files will be used to validate the current file
• a nested structure with sub-collection name as subdirectory name (and an optional subCollection key)
• or allow both but forbid mixed structure (the choice would be to the consumer and the compiler will check for files when there are no matching subdirectories)

src/content/
└── posts/
    ├── changelog/
    ├── thought/
    └── tutorial/

[wassfila]

I wonder if this will allow to render pure .md with an Astro component (my use case is pure markdown, e.g. existing github repo, and not .mdx that is not as tolerant as md parser). e.g. I have Heading.astro that takes props, and Code.astro,... if so how would that look like ?
If so I wonder then how would this scale to "rich cms content" like renderer, which might require recursive nodes like the AST provided by md parsers.
for info here's a link to my custom markdown renderer with Astro, https://github.com/MicroWebStacks/astro-big-doc/blob/main/src/components/markdown/AstroMarkdown.astro
I wish that becomes possible with content 2.0, as it's the only way to make md a true headless cms.

[ascorbic]

We have a preview release available, so I'd love if you can give it a try and share your feedback. Full details are in the PR, including changes in the API: withastro/astro#11334

[lloydjatkinson]

We have a preview release available, so I'd love if you can give it a try and share your feedback. Full details are in the PR, including changes in the API: withastro/astro#11334

Will the new Content Layer support loading custom components?

[ematipico]

We have a preview release available, so I'd love if you can give it a try and share your feedback. Full details are in the PR, including changes in the API: withastro/astro#11334

Will the new Content Layer support loading custom components?

What would they look like? Genuinely asking. We're considering various ways to render a new content collection, and components aren't off the table, but how would you run the schema against them?

[lloydjatkinson]

We have a preview release available, so I'd love if you can give it a try and share your feedback. Full details are in the PR, including changes in the API: withastro/astro#11334

Will the new Content Layer support loading custom components?

What would they look like? Genuinely asking. We're considering various ways to render a new content collection, and components aren't off the table, but how would you run the schema against them?

Honestly I don't know, but this is surely an important thing to think about as in the Storyblok example - content writers are likely to want to use them somehow. If a content loader doesn't support that, would that be a feature regression from what is avaliable this way? https://www.storyblok.com/tp/create-custom-components-in-storyblok-and-astro

[matthewp]

@lloydjatkinson you'll still be able to use mdx with this new API, so yes you can still use components.

[stefanprobst]

We have a preview release available, so I'd love if you can give it a try and share your feedback. Full details are in the PR, including changes in the API: withastro/astro#11334

will this support multiple markdown/richtext fields per collection? for example, would it be possible to express something like:

const schema = object({
  title: string(),
  sections: array(object({
    title: string(),
    content: mdx(),
  }))
})

[jlengstorf]

One thing that comes to mind that was a huge pain in the ass for Gatsby (and any other content aggregation abstraction I've worked with) is relationships between content.

A major motivator for using a content abstraction like this is centralizing data access. However, if there's no way to define relationships between the data, then teams are still defaulting to creating userland data merging and manipulation, which is (in my experience, at least) one of the key pain points that leads to wanting an aggregation layer in the first place.

I may have missed it in other discussion, but is there any plan or initial thoughts around how this would be managed?

Example Use Case

For example:

• Content team is writing blogs in Contentful
• Blog likes are a bespoke solution using Astro DB

Idea 1: Explicit API for creating relationships

I don't know that I like this API, but for some pseudo-code to show how this might work:

import { defineCollection, file, z, createRelationship } from 'astro:content';
import { contentful, contentfulSchema } from '../loaders/contentful';
import { likes, likesSchema } from '../loaders/likes';

const blog = defineCollection({
	type: "experimental_data",
	loader: contentful(/* some config */),
	schema: z.object({
		...contentfulSchema,
		likes: createRelationship({
			collection: 'likes',
			type: z.number(), // <-- (optional) type for the linked data — could be inferred?
			key: 'id', // <-- the Contentful schema field to link on
			foreignKey: 'blogId', // <-- the 'likes' schema field to link on
			resolver: (entry) => entry.count, // <-- (optional) how to link data in (full entry if omitted),
		}),
	}),
});

const likes = defineCollection({
	type: "experimental_data",
	loader: likes(),
	schema: z.object({
		...likesSchema,
		blog: createRelationship({ collection: 'likes', key: 'blog_id', foreignKey: 'id' }),
	}),
});

export const collections = { blog, likes };

Idea 2: Joins and an optional projection API

I like the way GraphQL and Sanity's GROQ allow you to dig into a referenced entry and get just the fields you need. Maybe something like that is possible?

import { defineCollection, file, z, reference } from 'astro:content';
import { contentful, contentfulSchema } from '../loaders/contentful';
import { comments, commentsSchema } from '../loaders/comments';
import { likes, likesSchema } from '../loaders/likes';

const blog = defineCollection({
	type: "experimental_data",
	loader: contentful(/* some config */),
	references: {
		// an optional resolver allows for custom projections of linked content
		comments: reference(contentfulSchema.id, commentsSchema.blogId, (comment) => ({
			author: comment.author.displayName,
			content: comment.content,
			date: new Date(comment.date).toLocaleString(),
		})),

		// returning a single value is also possible
		likes: reference(contentfulSchema.id, likesSchema.blogId, (entry) => entry.count),
	},
});

const comments = defineCollection({
	type: "experimental_data",
	loader: comments(),
	references: {
		// by default the full blog post entry is added as the `blog` key value
		blog: reference(commentsSchema.blogId, contentfulSchema.id),
	},
});

const likes = defineCollection({
	type: "experimental_data",
	loader: likes(),
	references: {
		blog: reference(likesSchema.blogId, contentfulSchema.id),
	},
});

export const collections = { blog, comments, likes };

---

I don't have strong opinions about the specifics, but I do think it's really important to talk through how cross-collection relationships fit into the content layer. Dropping this more to start the conversation than to try and assert a "right way" to do anything.

(Also, let me know if I should move this to a separate discussion.)