feat: add Content Layer loader #11334

ascorbic · 2024-06-25T11:08:02Z

This is the work-in-progress branch implementing the Content Layer RFC.

Testing

There is an example site in the content-layer fixture, which shows the use of a custom loader and the file() loader. Currently the API for end user looks like this:

import { defineCollection, file, z } from 'astro:content';
import { loader } from '../loaders/post-loader.js';

const blog = defineCollection({
	type: "experimental_data",
	loader: loader({ url: "https://jsonplaceholder.typicode.com/posts" }),
});

const dogs = defineCollection({
	type: "experimental_data",
	// The built in `file` loader loads multiple items from a JSON file
	loader: file("_data/dogs.json"),
	schema: z.object({
		breed: z.string(),
		id: z.string(),
		size: z.string(),
		origin: z.string(),
		lifespan: z.string(),
		temperament: z.array(z.string())
	}),
})
export const collections = { blog, dogs };

The blog collection shows a custom loader. These can be defined like this:

import type { Loader } from 'astro:content';

export interface PostLoaderConfig {
	url: string;
}

export function loader(config:PostLoaderConfig): Loader {
	return {
		name: "post-loader",
		load: async ({
			store, meta, logger
		}) => {
			logger.info('Loading posts');

			const lastSynced = meta.get('lastSynced');

			// Don't sync more than once a minute
			if (lastSynced && (Date.now() - Number(lastSynced) < 1000 * 60)) {
					logger.info('Skipping sync');
					return;
			}

			const posts = await fetch(config.url)
				.then((res) => res.json());

			store.clear();

			for (const post of posts) {
				store.set(post.id, post);
			}
			meta.set('lastSynced', String(Date.now()));
		},
	};
}

The API is currently defined like this:

export interface Loader<S extends ZodSchema = ZodSchema> {
	/** Unique name of the loader, e.g. the npm package name */
	name: string;
	/** Do the actual loading of the data */
	load: (context: LoaderContext) => Promise<void>;
	/** Optionally, define the schema of the data. Overridden by user-defined schema */
	schema?: S | Promise<S> | (() => S | Promise<S>);
	// Not yet implemented
	render?: (entry: any, options: any) => any;
}

export interface LoaderContext {
	collection: string;
	/** A database abstraction to store the actual data */
	store: ScopedDataStore;
	/**  A simple KV store, designed for things like sync tokens */
	meta: MetaStore;
	logger: AstroIntegrationLogger;

	settings: AstroSettings;

	/** Validates and parses the data according to the schema */
	parseData<T extends Record<string, unknown> = Record<string, unknown>>(
		props: ParseDataOptions
	): T;
}

export interface ParseDataOptions {
	/** The ID of the entry. Unique per collection */
	id: string;
	/** The raw, unvalidated data of the entry */
	data: Record<string, unknown>;
	/** An optional file path, where the entry represents a local file */
	filePath?: string;
}

Currently there is no support for loading md or mdx files using glob(), or for defining renderers.

Querying is currently limited to getCollection and getDataEntryById, with no filter options.

To test this, install the experimental version with npm install astro@experimental--content-layer

Docs

changeset-bot · 2024-06-25T11:08:07Z

⚠️ No Changeset found

Latest commit: 6bb89bc

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

github-actions

This PR is blocked because it contains a minor changeset. A reviewer will merge this at the next release if approved.

github-actions · 2024-06-26T16:33:22Z

Snapshots have been released for the following packages:

astro@experimental--content-layer

Publish Log

🦋  warn ===============================IMPORTANT!===============================
🦋  warn Packages will be released under the experimental--content-layer tag
🦋  warn ----------------------------------------------------------------------
🦋  info npm info astro
🦋  info npm info @astrojs/prism
🦋  info npm info @astrojs/rss
🦋  info npm info create-astro
🦋  info npm info @astrojs/db
🦋  info npm info @astrojs/alpinejs
🦋  info npm info @astrojs/lit
🦋  info npm info @astrojs/markdoc
🦋  info npm info @astrojs/mdx
🦋  info npm info @astrojs/node
🦋  info npm info @astrojs/partytown
🦋  info npm info @astrojs/preact
🦋  info npm info @astrojs/react
🦋  info npm info @astrojs/sitemap
🦋  info npm info @astrojs/solid-js
🦋  info npm info @astrojs/svelte
🦋  info npm info @astrojs/tailwind
🦋  info npm info @astrojs/vercel
🦋  info npm info @astrojs/vue
🦋  info npm info @astrojs/web-vitals
🦋  info npm info @astrojs/internal-helpers
🦋  info npm info @astrojs/markdown-remark
🦋  info npm info @astrojs/studio
🦋  info npm info @astrojs/telemetry
🦋  info npm info @astrojs/underscore-redirects
🦋  info npm info @astrojs/upgrade
🦋  info astro is being published because our local version (0.0.0-content-layer-20240626163213) has not been published on npm
🦋  warn @astrojs/prism is not being published because version 3.1.0 is already published on npm
🦋  warn @astrojs/rss is not being published because version 4.0.7 is already published on npm
🦋  warn create-astro is not being published because version 4.8.0 is already published on npm
🦋  warn @astrojs/db is not being published because version 0.11.7 is already published on npm
🦋  warn @astrojs/alpinejs is not being published because version 0.4.0 is already published on npm
🦋  warn @astrojs/lit is not being published because version 4.3.0 is already published on npm
🦋  warn @astrojs/markdoc is not being published because version 0.11.1 is already published on npm
🦋  warn @astrojs/mdx is not being published because version 3.1.2 is already published on npm
🦋  warn @astrojs/node is not being published because version 8.3.2 is already published on npm
🦋  warn @astrojs/partytown is not being published because version 2.1.1 is already published on npm
🦋  warn @astrojs/preact is not being published because version 3.5.0 is already published on npm
🦋  warn @astrojs/react is not being published because version 3.6.0 is already published on npm
🦋  warn @astrojs/sitemap is not being published because version 3.1.6 is already published on npm
🦋  warn @astrojs/solid-js is not being published because version 4.4.0 is already published on npm
🦋  warn @astrojs/svelte is not being published because version 5.6.0 is already published on npm
🦋  warn @astrojs/tailwind is not being published because version 5.1.0 is already published on npm
🦋  warn @astrojs/vercel is not being published because version 7.7.2 is already published on npm
🦋  warn @astrojs/vue is not being published because version 4.5.0 is already published on npm
🦋  warn @astrojs/web-vitals is not being published because version 0.2.1 is already published on npm
🦋  warn @astrojs/internal-helpers is not being published because version 0.4.1 is already published on npm
🦋  warn @astrojs/markdown-remark is not being published because version 5.1.1 is already published on npm
🦋  warn @astrojs/studio is not being published because version 0.1.1 is already published on npm
🦋  warn @astrojs/telemetry is not being published because version 3.1.0 is already published on npm
🦋  warn @astrojs/underscore-redirects is not being published because version 0.3.4 is already published on npm
🦋  warn @astrojs/upgrade is not being published because version 0.3.1 is already published on npm
🦋  info Publishing "astro" at "0.0.0-content-layer-20240626163213"
🦋  success packages published successfully:
🦋  astro@0.0.0-content-layer-20240626163213
🦋  Creating git tag...
🦋  New tag:  astro@0.0.0-content-layer-20240626163213

Build Log


> root@0.0.0 build /home/runner/work/astro/astro
> turbo run build --filter=astro --filter=create-astro --filter="@astrojs/*" --filter="@benchmark/*"

• Packages in scope: @astrojs/alpinejs, @astrojs/cloudflare, @astrojs/db, @astrojs/internal-helpers, @astrojs/lit, @astrojs/markdoc, @astrojs/markdown-remark, @astrojs/mdx, @astrojs/netlify, @astrojs/node, @astrojs/partytown, @astrojs/preact, @astrojs/prism, @astrojs/react, @astrojs/rss, @astrojs/sitemap, @astrojs/solid-js, @astrojs/studio, @astrojs/svelte, @astrojs/tailwind, @astrojs/telemetry, @astrojs/underscore-redirects, @astrojs/upgrade, @astrojs/vercel, @astrojs/vue, @astrojs/web-vitals, @benchmark/timer, astro, create-astro
• Running build in 29 packages
• Remote caching enabled
::group::@astrojs/upgrade:build
cache hit, suppressing logs bc4f3f091f583ec1
::endgroup::
::group::@astrojs/telemetry:build
cache hit, suppressing logs ec7499837d68ccd3
::endgroup::
::group::create-astro:build
cache hit, suppressing logs 935b58c36a25eda3
::endgroup::
::group::@astrojs/internal-helpers:build
cache hit, suppressing logs e8d97ce643c579cb
::endgroup::
::group::@astrojs/prism:build
cache hit, suppressing logs 82fa269e60a35beb
::endgroup::
::group::@astrojs/markdown-remark:build
cache hit, suppressing logs e04f3084b2a283e9
::endgroup::
::group::astro:build
cache miss, executing 31039041a22a123f

> astro@0.0.0-content-layer-20240626163213 build /home/runner/work/astro/astro/packages/astro
> pnpm run prebuild && astro-scripts build "src/**/*.{ts,js}" && tsc && pnpm run postbuild


> astro@0.0.0-content-layer-20240626163213 prebuild /home/runner/work/astro/astro/packages/astro
> astro-scripts prebuild --to-string "src/runtime/server/astro-island.ts" "src/runtime/client/{idle,load,media,only,visible}.ts"


> astro@0.0.0-content-layer-20240626163213 postbuild /home/runner/work/astro/astro/packages/astro
> astro-scripts copy "src/**/*.astro" && astro-scripts copy "src/**/*.wasm"

::endgroup::
::group::@astrojs/studio:build
cache miss, executing cde0a60f1cbd58e7

> @astrojs/studio@0.1.1 build /home/runner/work/astro/astro/packages/studio
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/lit:build
cache miss, executing 590da9547eb9758e

> @astrojs/lit@4.3.0 build /home/runner/work/astro/astro/packages/integrations/lit
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/partytown:build
cache miss, executing f563b6b14ecf2e5a

> @astrojs/partytown@2.1.1 build /home/runner/work/astro/astro/packages/integrations/partytown
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/underscore-redirects:build
cache miss, executing 5e7f7e8dd20771bf

> @astrojs/underscore-redirects@0.3.4 build /home/runner/work/astro/astro/packages/underscore-redirects
> astro-scripts build "src/**/*.ts" && tsc -p tsconfig.json


> @astrojs/underscore-redirects@0.3.4 postbuild /home/runner/work/astro/astro/packages/underscore-redirects
> astro-scripts copy "src/**/*.js"

::endgroup::
::group::@astrojs/rss:build
cache miss, executing db00d8b44f868905

> @astrojs/rss@4.0.7 build /home/runner/work/astro/astro/packages/astro-rss
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/node:build
cache miss, executing 5769374ccf98a18d

> @astrojs/node@8.3.2 build /home/runner/work/astro/astro/packages/integrations/node
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/tailwind:build
cache miss, executing d2a26a51130d540a

> @astrojs/tailwind@5.1.0 build /home/runner/work/astro/astro/packages/integrations/tailwind
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/mdx:build
cache miss, executing 58aa9788bd9506c6

> @astrojs/mdx@3.1.2 build /home/runner/work/astro/astro/packages/integrations/mdx
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/markdoc:build
cache miss, executing c530d6d95a7dce23

> @astrojs/markdoc@0.11.1 build /home/runner/work/astro/astro/packages/integrations/markdoc
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/svelte:build
cache miss, executing 3fcbe38704a147bd

> @astrojs/svelte@5.6.0 build /home/runner/work/astro/astro/packages/integrations/svelte
> astro-scripts build "src/index.ts" && astro-scripts build "src/editor.cts" --force-cjs --no-clean-dist && tsc

::endgroup::
::group::@astrojs/solid-js:build
cache miss, executing 7aa1aec0e160a401

> @astrojs/solid-js@4.4.0 build /home/runner/work/astro/astro/packages/integrations/solid
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@benchmark/timer:build
cache miss, executing 8fbb526121d666d5

> @benchmark/timer@0.0.0 build /home/runner/work/astro/astro/benchmark/packages/timer
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/react:build
cache miss, executing e04d2cbdc0664d83

> @astrojs/react@3.6.0 build /home/runner/work/astro/astro/packages/integrations/react
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/alpinejs:build
cache miss, executing a7fddf863f5ab50a

> @astrojs/alpinejs@0.4.0 build /home/runner/work/astro/astro/packages/integrations/alpinejs
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/vercel:build
cache miss, executing d885a3be7842fbad

> @astrojs/vercel@7.7.2 build /home/runner/work/astro/astro/packages/integrations/vercel
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/preact:build
cache miss, executing a6531d751e1096eb

> @astrojs/preact@3.5.0 build /home/runner/work/astro/astro/packages/integrations/preact
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/vue:build
cache miss, executing 5ae4796e2344dae1

> @astrojs/vue@4.5.0 build /home/runner/work/astro/astro/packages/integrations/vue
> astro-scripts build "src/index.ts" && astro-scripts build "src/editor.cts" --force-cjs --no-clean-dist && tsc

::endgroup::
::group::@astrojs/sitemap:build
cache miss, executing e8c6fe714a7f43f4

> @astrojs/sitemap@3.1.6 build /home/runner/work/astro/astro/packages/integrations/sitemap
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/db:build
cache miss, executing b1fa98442d6f43ce

> @astrojs/db@0.11.7 build /home/runner/work/astro/astro/packages/db
> astro-scripts build "src/**/*.ts" && tsc && pnpm types:virtual


> @astrojs/db@0.11.7 types:virtual /home/runner/work/astro/astro/packages/db
> tsc -p ./tsconfig.virtual.json

::endgroup::
::group::@astrojs/web-vitals:build
cache miss, executing 046aeb953d322139

> @astrojs/web-vitals@0.2.1 build /home/runner/work/astro/astro/packages/integrations/web-vitals
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::

 Tasks:    27 successful, 27 total
Cached:    6 cached, 27 total
  Time:    51.34s

packages/astro/src/content/data-store.ts

packages/astro/src/content/loaders.ts

packages/astro/src/content/runtime.ts

packages/astro/src/content/types-generator.ts

packages/astro/src/content/utils.ts

packages/astro/test/content-layer.test.js

ematipico

Let's get the party started!

Get out :D

ascorbic added 4 commits June 24, 2024 17:24

wip

aa1db49

wip

a7639e3

wip

63a23d2

Update demo

624c377

github-actions bot added the pkg: astro Related to the core `astro` package (scope) label Jun 25, 2024

ascorbic added 3 commits June 25, 2024 14:21

Add meta

d67ee7e

wip

631f3d2

Add file loader

4b7f8f7

ascorbic force-pushed the content-layer-loader branch from 20bee9a to 1f0affc Compare June 26, 2024 08:57

Add schema validation

14330f7

ascorbic force-pushed the content-layer-loader branch from 1f0affc to 14330f7 Compare June 26, 2024 08:57

ascorbic added 3 commits June 26, 2024 12:13

Remove log

b3b8dd4

Merge branch 'main' into content-layer-loader

04d09f0

Changeset

c932fb8

github-actions bot added the semver: minor Change triggers a `minor` release label Jun 26, 2024

github-actions bot previously requested changes Jun 26, 2024

View reviewed changes

ascorbic changed the title ~~WIP: Content layer loader~~ WIP: Content Layer Jun 26, 2024

ascorbic added 4 commits June 26, 2024 12:25

Format

941c4be

Lockfile

f997ca5

Fix type

92c3312

Merge branch 'main' into content-layer-loader

e0bd9cd

This comment was marked as outdated.

Sign in to view

ascorbic added 3 commits June 26, 2024 14:14

Handle loading for data store JSON

5cd5670

Merge branch 'main' into content-layer-loader

6ca356a

Use rollup util to import JSON

3fcdc59

ascorbic force-pushed the content-layer-loader branch from 839838e to 3fcdc59 Compare June 26, 2024 13:35

This comment was marked as outdated.

Sign in to view

ascorbic added 2 commits June 26, 2024 17:24

Fix types

14322d4

Merge branch 'main' into content-layer-loader

3e21f03

ascorbic mentioned this pull request Jun 26, 2024

Content Layer withastro/roadmap#946

Open

ascorbic changed the title ~~WIP: Content Layer~~ Content Layer Jun 26, 2024

Format

8d310af

ematipico mentioned this pull request Jun 27, 2024

feat: add typegen for loaders #11358

Merged

ascorbic changed the base branch from main to content-layer June 27, 2024 08:22

ascorbic changed the title ~~Content Layer~~ feat: add Content Layer loader Jun 27, 2024

ascorbic marked this pull request as ready for review June 27, 2024 08:23

ascorbic added 3 commits June 27, 2024 09:23

Merge branch 'content-layer' into content-layer-loader

0a89f15

Merge branch 'content-layer' into content-layer-loader

51fcda3

Add tests

9de133f

ematipico reviewed Jun 27, 2024

View reviewed changes

ascorbic force-pushed the content-layer-loader branch from 881f3c7 to e93ad3e Compare June 27, 2024 14:19

Changes from review

6bb89bc

ascorbic force-pushed the content-layer-loader branch from e93ad3e to 6bb89bc Compare June 27, 2024 14:20

ascorbic requested a review from ematipico June 27, 2024 14:24

ascorbic removed the semver: minor Change triggers a `minor` release label Jun 28, 2024

ematipico approved these changes Jun 28, 2024

View reviewed changes

ematipico merged commit 9561a07 into content-layer Jun 28, 2024
14 checks passed

ematipico deleted the content-layer-loader branch June 28, 2024 08:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add Content Layer loader #11334

feat: add Content Layer loader #11334

ascorbic commented Jun 25, 2024 •

edited

Loading

changeset-bot bot commented Jun 25, 2024 •

edited

Loading

github-actions bot left a comment

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Jun 26, 2024

ematipico left a comment

feat: add Content Layer loader #11334

feat: add Content Layer loader #11334

Conversation

ascorbic commented Jun 25, 2024 • edited Loading

Testing

Docs

changeset-bot bot commented Jun 25, 2024 • edited Loading

⚠️ No Changeset found

github-actions bot left a comment

Choose a reason for hiding this comment

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Jun 26, 2024

ematipico left a comment

Choose a reason for hiding this comment

ascorbic commented Jun 25, 2024 •

edited

Loading

changeset-bot bot commented Jun 25, 2024 •

edited

Loading