Sourcemaps generated for `astro:data-layer-content` causing slow `astro dev` reloads

[KianNH]

Astro Info

Astro                    v5.1.7
Node                     v22.12.0
System                   macOS (arm64)
Package Manager          npm
Output                   static
Adapter                  none
Integrations             @astrojs/starlight
                         @astrojs/tailwind
                         astro-live-code
                         astro-icon
                         @astrojs/sitemap
                         @astrojs/react

If this issue only occurs in one browser, which browser is a problem?

No response

Describe the Bug

Slowdowns

Whilst investigating slow page reloads in astro dev, I noticed there was always a common theme in what function was taking the longest: generateMap from magic-string. This single reload took 6.77 seconds, and generating sourcemaps seemed to dominate it:

vite-plugin-import-meta-env.js is one of two places where generateMap is used, the other being Vite's own ssrTransform.

Since it's the same function and the same sourcemaps being generated, I'll just focus on my findings from patching vite-plugin-import-meta-env.js.

Source

astro/packages/astro/src/env/vite-plugin-import-meta-env.ts

Lines 99 to 126 in 78fd73a

	transform(source, id, options) {
	if (!options?.ssr || !source.includes('import.meta.env')) {
	return;
	}
	// Find matches for *private* env and do our own replacement.
	// Env is retrieved before process.env is populated by astro:env
	// so that import.meta.env is first replaced by values, not process.env
	privateEnv ??= envLoader.getPrivateEnv();
	// In dev, we can assign the private env vars to `import.meta.env` directly for performance
	if (isDev) {
	const s = new MagicString(source);
	if (!devImportMetaEnvPrepend) {
	devImportMetaEnvPrepend = `Object.assign(import.meta.env,{`;
	for (const key in privateEnv) {
	devImportMetaEnvPrepend += `${key}:${privateEnv[key]},`;
	}
	devImportMetaEnvPrepend += '});';
	}
	s.prepend(devImportMetaEnvPrepend);
	return {
	code: s.toString(),
	map: s.generateMap({ hires: 'boundary' }),
	};
	}

Patch

diff --git a/node_modules/astro/dist/env/vite-plugin-import-meta-env.js b/node_modules/astro/dist/env/vite-plugin-import-meta-env.js
index 2246279..86edda1 100644
--- a/node_modules/astro/dist/env/vite-plugin-import-meta-env.js
+++ b/node_modules/astro/dist/env/vite-plugin-import-meta-env.js
@@ -68,9 +68,17 @@ function importMetaEnv({ envLoader }) {
       if (!options?.ssr || !source.includes("import.meta.env")) {
         return;
       }
+
+      const path = id.replace(process.cwd(), "");
+
+      console.log(path + `: source ${source.length / 1_000_000} MB`)
+
       privateEnv ??= envLoader.getPrivateEnv();
       if (isDev) {
+        console.time(path + ": new MagicString(source)");
         const s = new MagicString(source);
+        console.timeEnd(path + ": new MagicString(source)");
+
         if (!devImportMetaEnvPrepend) {
           devImportMetaEnvPrepend = `Object.assign(import.meta.env,{`;
           for (const key in privateEnv) {
@@ -78,10 +86,24 @@ function importMetaEnv({ envLoader }) {
           }
           devImportMetaEnvPrepend += "});";
         }
+
+        console.time(path + ": s.prepend(devImportMetaEnvPrepend)");
         s.prepend(devImportMetaEnvPrepend);
+        console.timeEnd(path + ": s.prepend(devImportMetaEnvPrepend)");
+
+        console.time(path + ": s.toString()");
+        const code = s.toString();
+        console.timeEnd(path + ": s.toString()");
+
+        console.time(path + ': s.generateMap({ hires: "boundary" })');
+        const map = s.generateMap({ hires: "boundary" });
+        console.timeEnd(path + ': s.generateMap({ hires: "boundary" })');
+
+        console.log(path + `: map ${map.mappings.length / 1_000_000} MB`)
+
         return {
-          code: s.toString(),
-          map: s.generateMap({ hires: "boundary" })
+          code,
+          map,
         };
       }
       if (!defaultDefines) {

Output

In the same scenario as the profile, editing an item in a content collection such as a MDX page, lots of files are transformed such as /node_modules/astro/dist/assets/services/service.js or /node_modules/astro/components/Picture.astro but these are not the culprit! As files as small as 4.77 KB, their sourcemaps are generated less than 0.205ms.

The source causing the slowdown, in my case, is astro:data-layer-content:

astro:data-layer-content: source 25.497913 MB
astro:data-layer-content: new MagicString(source): 0.037ms
astro:data-layer-content: s.prepend(devImportMetaEnvPrepend): 0.004ms
astro:data-layer-content: s.toString(): 0.003ms
astro:data-layer-content: s.generateMap({ hires: "boundary" }): 1.316s
astro:data-layer-content: map 54.23153 MB

astro:data-layer-content might look abnormally large but our .astro/data-store.json file really is 26,054,063 characters.

Content collections

Our ./src/content is around 37 MB, other than an outlier or two which we're going to move to just be a .txt file in public we could only reduce this by than 6 MB.

Wait, but this file doesn't use import.meta.env?

Well, one of our MDX files contains that string! After removing that one snippet, we're not getting any logs for astro:data-layer-content from vite-plugin-import-meta-env.js anymore and that file has disappeared from our radar:

I imagine this is also the case with Astro's own docs.

Vite itself is also generating a sourcemap for this file

As touched on earlier, ssrTransform in Vite is also using magic-string to generate a sourcemap on this file.

I'm not sure how much can be done (if anything) to avoid that, but it's definitely a place for a huge performance gain on large sites using content collections!

Back to vite-plugin-import-meta-env.js

It sounds like adding source !== "astro:data-layer-content" to vite-plugin-import-meta-env.js#transform would be a risk-free win, but I'll leave that to the folks who are more familiar with the implications.

What's the expected result?

Sourcemaps are not generated for this file (under the assumption they are not necessary).

Link to Minimal Reproducible Example

N/A

Participation

• I am willing to submit a pull request for this issue.

[ascorbic]

Thanks for tracking this down! This is definitely a bug. We don't want to be replacing env vars inside JSON files. It does do a quick check before running to see if it includes the string import.meta.env, but that's only a rough check and there will be plenty of false positives. Should be simpel enough to limit it to just source code though.