fix(build): handle invalid JSON in import.meta.env #17648

yuzheng14 · 2024-07-10T03:27:24Z

Bug Reproduction

NOTE(bluwy): the solution description below is outdated. The new solution is based on #17648 (comment)

When import.meta.env has some invalid JSON value, such as __VITE_IS_LEGACY__ imported by @vitejs/plugin-legacy, Vite will replace the import.meta.env in import.meta.env.SOMEVAR_UNDEFINED to a plain object. Then the javascript file will be an invalid javascript file which will produce a error when handled by proceeding commonjs plugin.

The minimum reproducible set is here: https://stackblitz.com/edit/stackblitz-starters-xr8lwk?file=index.html

Reason

As previous description, vite:define plugin will replace import.meta.env.SOMEVAR_UNDEFINED to {"BASE_URL": "/", "MODE": "development", "DEV": true, "PROD": false, "SSR": false, "LEGACY": __VITE_IS_LEGACY__}.SOMEVAR_UNDEFINED.

The reason is, vite:define will replace the actual import.meta.env with a marker.

vite/packages/vite/src/node/plugins/define.ts

Lines 144 to 148 in 167006e

    
           if (env && !canJsonParse(env)) { 
        
             const marker = `_${getHash(env, env.length - 2)}_` 
        
             replacementMarkers[marker] = env 
        
             define = { ...define, 'import.meta.env': marker } 
        
           }

The original value of import.meta.env is a object, while the marker is an identifier. The replace strategy of esbuild to handle this two type is different.

Replacement expressions other than arrays and objects are substituted inline, which means that they can participate in constant folding. Array and object replacement expressions are stored in a variable and then referenced using an identifier instead of being substituted inline, which avoids substituting repeated copies of the value but means that the values don't participate in constant folding.

--esbuild documentation

Consider the following example.

import.meta.env.SOMEVAR_UNDEFINED
import.meta.env.SOMEVAR_UNDEFINED

When import.meta.env is a identifier, esbuild will replace it in place.

_hashGeneratedByViteDefine_.SOMEVAR_UNDEFINED
_hashGeneratedByViteDefine_.SOMEVAR_UNDEFINED

After that, vite:define will replace the marker with the original import.meta.env

vite/packages/vite/src/node/plugins/define.ts

Lines 181 to 183 in 167006e

    
           for (const marker in replacementMarkers) { 
        
             result.code = result.code.replaceAll(marker, replacementMarkers[marker]) 
        
           }

{"BASE_URL": "/", "MODE": "development", "DEV": true, "PROD": false, "SSR": false, "LEGACY": __VITE_IS_LEGACY__}.SOMEVAR_UNDEFINED
{"BASE_URL": "/", "MODE": "development", "DEV": true, "PROD": false, "SSR": false, "LEGACY": __VITE_IS_LEGACY__}.SOMEVAR_UNDEFINED

Now it's a invalid javascript code.

Solution

Align the type of the original and marker.

I generate a marker which is a valid JSON, so that the replace strategy will be same.

Radical Solution

I found that the value of import.meta.env is spread in config.

{
  "loader": "js",
  "charset": "utf8",
  "platform": "neutral",
  "define": {
    "process.env": "{}",
    "global.process.env": "{}",
    "globalThis.process.env": "{}",
    "process.env.NODE_ENV": "\"production\"",
    "global.process.env.NODE_ENV": "\"production\"",
    "globalThis.process.env.NODE_ENV": "\"production\"",
    "import.meta.hot": "undefined",
    "import.meta.env.BASE_URL": "\"/\"",
    "import.meta.env.MODE": "\"production\"",
    "import.meta.env.DEV": "false",
    "import.meta.env.PROD": "true",
    "import.meta.env.SSR": "false",
    "import.meta.env.LEGACY": "__VITE_IS_LEGACY__",
    "import.meta.env": "_ecc578fd856ad0ab7198de04a00b9405c7a6d2549217f51adfff717ee9f7b851______________________________________________"
  },
  "sourcefile": "/Users/yuzheng14/code/repositories/legacy-conflict/node_modules/.pnpm/vue@2.7.16/node_modules/vue/dist/vue.runtime.esm.js",
  "sourcemap": false
}

And esbuild will replace it first.

Under this case, import.env.meta will be only be replace when the value of it is not defined. So why not leave a {} for import.meta.env just like process.env?

vite/packages/vite/src/node/plugins/define.ts

Lines 20 to 27 in 167006e

    
           Object.assign(processEnv, { 
        
             'process.env': `{}`, 
        
             'global.process.env': `{}`, 
        
             'globalThis.process.env': `{}`, 
        
             'process.env.NODE_ENV': JSON.stringify(nodeEnv), 
        
             'global.process.env.NODE_ENV': JSON.stringify(nodeEnv), 
        
             'globalThis.process.env.NODE_ENV': JSON.stringify(nodeEnv), 
        
           })

But I couldn't aware the side effects the this solution, so that I choose to modify the generation of marker.

stackblitz · 2024-07-10T03:27:27Z

Run & review this pull request in StackBlitz Codeflow.

yuzheng14 · 2024-07-15T01:53:01Z

@bluwy Could you please take some time to review this pull request?

yuzheng14 · 2024-07-22T01:46:07Z

@bluwy I saw that you have viewed this PR. Is it worth merging? Are there any failures?

bluwy · 2024-07-23T05:51:32Z

I have a few concerns with this trick.

It relies on the specific esbuild output, e.g. not adding/removing any spaces between the JSON object, quotes being removed, etc.
It makes the import.meta.env.SOMEVAR_UNDEFINED case where Vite should simply replace as undefined harder to fix. Before, we can probably use a regex for _hashGeneratedByViteDefine_.(.+?)\b to detect the env ahead of time. (related When referencing an environment variable that doesnt exist, vite dumps your whole environment into the bundle as an object #17710)

Maybe we need a different approach to solve this. Perhaps we can simply replace import.meta.env as __vite_import_meta_env__ and append the const __vite_import_meta_env__ = ... manually ourselves? That way we can imitate esbuild's JSON object handling without a post-replacement. We can use esbuild's banner option so the sourcemap is also handled automatically.

And we can build a regex with __vite_import_meta_env__.(.+?)\b to identify the unused env vars ahead of time, replacing import.meta.env.SOMEVAR_UNDEFINED with undefined.

Under this case, import.env.meta will be only be replace when the value of it is not defined. So why not leave a {} for import.meta.env just like process.env?

When the user does console.log(import.meta.env), it should log the entire object. It doesn't happen for process.env as it may contain sensitive information.

yuzheng14 · 2024-07-23T06:20:40Z

Ok, got it. I'll try to implement this.

…defined key

yuzheng14 · 2024-07-24T08:20:14Z

@bluwy I have finished this. Now it will replace the undefined import.meta.env key using define, and will add the manual __vite_import_meta_env__ banner only when there is a bare import.meta.env.

packages/vite/src/node/plugins/define.ts

playground/define/vite.config.js

packages/vite/src/node/plugins/define.ts

yuzheng14 · 2024-07-25T05:29:27Z

@bluwy Now it replaces import.meta.env after replaceDefine. If it needs to generate sourcemap, it replaces using magic-string to generate ; if not, it replace using regex directly.

I think that there is no need to capture the key. If there still has any keys, the keys' supposed to be undefined in import.meta.env.

bluwy · 2024-07-25T07:51:21Z

I think that there is no need to capture the key. If there still has any keys, the keys' supposed to be undefined in import.meta.env.

Good point, that's a nice optimization.

@bluwy Now it replaces import.meta.env after replaceDefine. If it needs to generate sourcemap, it replaces using magic-string to generate ; if not, it replace using regex directly.

I don't think we should do this because sourcemap handling is very expensive. I'll push some commits and test locally if we can apply the ; trick I mentioned above, which should be faster.

yuzheng14 · 2024-07-25T08:03:40Z

I don't think we should do this because sourcemap handling is very expensive. I'll push some commits and test locally if we can apply the ; trick I mentioned above, which should be faster.

It should replace the undefined key with undefined after replaceDefine. However, I have no idea about sourcemap for this (by the way, I wasn't aware of sourcemap before), so I used magic-string. Under this premise, I use magic-string to prepend the banner. I think this is an uncommon situation, so a little expense may be tolerated.

bluwy · 2024-07-25T08:25:50Z

I pushed the commits to optimize the sourcemap handling. I agree that bare import.meta.env or unknown import.meta.env.* are rare, but I think since the refactor also simplifies the code, it's worth it for now. Once the tests pass, I'll run ecosystem-ci and it should be good to go.

I also verified that the ; trick works.

bluwy · 2024-07-25T08:41:45Z

/ecosystem-ci run

vite-ecosystem-ci · 2024-07-25T08:54:42Z

📝 Ran ecosystem CI on af23b75: Open

suite	result	latest scheduled
astro	❌ failure	❌ failure
nuxt	❌ failure	❌ failure
vike	❌ failure	❌ failure
vite-plugin-react-pages	❌ failure	❌ failure
vitest	❌ failure	❌ failure

✅ analogjs, histoire, ladle, laravel, marko, previewjs, quasar, qwik, rakkas, remix, sveltekit, unocss, vite-plugin-pwa, vite-plugin-react, vite-plugin-react-swc, vite-plugin-svelte, vite-plugin-vue, vite-setup-catalogue, vitepress

yuzheng14 · 2024-07-25T09:35:21Z

I saw that there are some ecosystem-ci fail. Is there anything I should do?

bluwy · 2024-07-25T12:07:23Z

You can ignore them. The "latest scheduled" are the results from main, so this PR isn't causing those fails. I'm waiting for another review before merging this.

patak-dev

Great fix! I think we could merge it in 5.4 just to play safe.

yuzheng14 · 2024-07-25T14:28:14Z

Okay. Thanks to everyone for reviewing this pr!

silverwind · 2024-08-09T17:28:44Z

It seems that since this change, I'm running into Identifier "__vite_import_meta_env__" has already been declared errors during build when it's bundling a library built that was also built with vite.

What's also unexpected is that this __vite_import_meta_env__ variable is added to the JS output of the library even though it does not use import.meta.env at all.

yuzheng14 · 2024-08-10T15:42:17Z

It seems that since this change, I'm running into Identifier "__vite_import_meta_env__" has already been declared errors during build when it's bundling a library built that was also built with vite.

What's also unexpected is that this __vite_import_meta_env__ variable is added to the JS output of the library even though it does not use import.meta.env at all.

Please provide a issue to describe the details.

silverwind · 2024-08-11T09:32:59Z

I have a hard time creating a reproduction for this (and issues without reproduction would be rejected), but here is the full error:

error during build:
[commonjs--resolver] node_modules/lib/dist/index.js (3392:6): Identifier "__vite_import_meta_env__" has already been declared
file: node_modules/lib/dist/index.js:3392:6

3390: const create = (createState) => createState ? createImpl(createState) : createImpl;
3391: const __vite_import_meta_env__ = { "BASE_URL": "/", "DEV": false, "MODE": "production", "PROD": true, "SSR": false };
3392: function createJSONStorage(getStorage, options2) {
            ^
3393:   let storage;
3394:   try {

    at getRollupError (node_modules/vite/node_modules/rollup/dist/es/shared/parseAst.js:392:41)
    at error (node_modules/vite/node_modules/rollup/dist/es/shared/parseAst.js:388:42)
    at Module.error (node_modules/vite/node_modules/rollup/dist/es/shared/node-entry.js:13944:16)
    at ModuleScope.addDeclaration (node_modules/vite/node_modules/rollup/dist/es/shared/node-entry.js:7362:21)
    at ModuleScope.addDeclaration (node_modules/vite/node_modules/rollup/dist/es/shared/node-entry.js:12075:22)
    at Identifier.declare (node_modules/vite/node_modules/rollup/dist/es/shared/node-entry.js:6960:39)
    at VariableDeclarator.declareDeclarator (node_modules/vite/node_modules/rollup/dist/es/shared/node-entry.js:10661:17)
    at VariableDeclaration.initialise (node_modules/vite/node_modules/rollup/dist/es/shared/node-entry.js:12343:24)
    at convertNode (node_modules/vite/node_modules/rollup/dist/es/shared/node-entry.js:13167:10)
    at convertNodeList (node_modules/vite/node_modules/rollup/dist/es/shared/node-entry.js:13177:34)

The compiled output of the lib (which has minify: false, so the var names go through unmodified) has three such variables:

$ rg __vite_import_meta_env__ dist/index.js
3001:const __vite_import_meta_env__$2 = { "BASE_URL": "/", "DEV": false, "MODE": "production", "PROD": true, "SSR": false };
3020:    if ((__vite_import_meta_env__$2 ? "production" : void 0) !== "production") {
3357:const __vite_import_meta_env__$1 = { "BASE_URL": "/", "DEV": false, "MODE": "production", "PROD": true, "SSR": false };
3363:  if ((__vite_import_meta_env__$1 ? "production" : void 0) !== "production" && equalityFn && !didWarnAboutEqualityFn) {
3380:  if ((__vite_import_meta_env__$1 ? "production" : void 0) !== "production" && typeof createState !== "function") {
3391:const __vite_import_meta_env__ = { "BASE_URL": "/", "DEV": false, "MODE": "production", "PROD": true, "SSR": false };
3709:    if ((__vite_import_meta_env__ ? "production" : void 0) !== "production") {

Edit: WIP reproduction at https://github.com/silverwind/vite-import-meta, but does not reproduce yet.

yuzheng14 · 2024-08-11T14:15:32Z

Is the library has error named lib? I saw that the file is node_modules/lib/dist/index.js.

hi-ogawa · 2024-08-12T00:18:59Z

I think one scenario this can happen is when library code (or its dependencies) include both import.meta.env and process.env silverwind/vite-import-meta#1

// lib/index.js
export const foo = () => {
  // https://github.com/pmndrs/zustand/blob/a56e76db5261291dcf9a88573dac58f67edb93db/rollup.config.js#L67-L70
  console.log(import.meta.env ? import.meta.env.MODE : undefined);

  console.log(process.env.NODE_ENV);
};

// ⇓ vite library build

// lib/dist/index.js
const __vite_import_meta_env__ = { "BASE_URL": "/", "DEV": false, "MODE": "production", "PROD": true, "SSR": false };
const foo = () => {
  console.log(__vite_import_meta_env__ ? "production" : void 0);
  console.log(process.env.NODE_ENV);
};
export {
  foo
};

Then later in main app build, vite:define transforms lib/dist/index.js since it's left with process.env, which probably ends up detecting __vite_import_meta_env__ and injecting the banner __vite_import_meta_env__ again.

yuzheng14 · 2024-08-12T02:04:20Z

But is there any need to use bare import.meta.env in production output of a library?

There is no need to use conditional operation in your code. If you use vite to bundle it, it will replace import.meta.env.SOMETHING_UNDEFINED to undefined directly now or to a object including value of import.meta.env in the past version.

hi-ogawa · 2024-08-12T02:22:25Z

But is there any need to use bare import.meta.env in production output of a library?

There is no need to use conditional operation in your code. If you use vite to bundle it, it will replace import.meta.env.SOMETHING_UNDEFINED to undefined directly now or to a object including value of import.meta.env in the past version.

To clarify, this is not "my code" but just some examples in the wild (zustand in this case https://github.com/pmndrs/zustand/blob/a56e76db5261291dcf9a88573dac58f67edb93db/rollup.config.js#L67-L70). So the concern is that Vite library mode author might happen to use such dependencies (without externalized) and leave __vite_import_meta_env__ in their library output unnoticed. I think this is a rather edge case, but ideally this is desired to be handled more robustly somehow.

Another package I remembered which might causes an issue is unjs's std-env https://github.com/unjs/std-env/blob/b4ef16832baf4594ece7796a2c1805712fde70a3/src/env.ts. It uses a bare import.meta.env and process.env, so if library mode build has this dependency bundled, the same issue would probably happen.

yuzheng14 · 2024-08-12T03:27:21Z

Maybe we should create an issue to zustand because there is no need to replace import.meta.env?.MODE with (import.meta.env ? import.meta.env.MODE : undefined). esbuild can replace import.meta.env?.MODE to its value. For example:

import { transform } from 'esbuild'

const result = await transform(
  `console.log(import.meta.env?.MODE)`,
  {
    sourcemap: false,
    minify: true,
    target: ['chrome60'],
    define: {
      'import.meta.env.MODE': JSON.stringify('production'),
    },
  }
)

console.log(result.code)

It will ouput console.log("production");.

silverwind · 2024-08-12T08:05:39Z

Yes, it is zustand as well in my case as well which uses import.meta.env in their original code. Apparently, their bundler (rollup) transpiles import.meta.env?.MODE !== 'production' to process.env.NODE_ENV !== "production". So I have

app (uses import.meta.env) which imports lib
lib (does not use import.meta.env) which imports zustand
zustand which effectively uses process.env

I think it's fine for zustand to do this. They are logging some runtime deprecation warnings, but only during dev mode.

yuzheng14 · 2024-08-12T10:24:35Z

Okay, I'll try to reproduce it and make a new pr.

bluwy · 2024-08-12T10:47:50Z

I've created #17874 to track this, let's move the conversation over there

chore: repro for vitejs/vite#17648

fix(build): handle invalid JSON in import.meta.env

779f779

bluwy mentioned this pull request Jul 15, 2024

Missing .env variables are incorrectly resolved in SSR mode withastro/astro#11465

Closed

1 task

yuzheng14 force-pushed the fix/define-replace branch from bb9912c to b94ce8d Compare July 24, 2024 07:05

refactor(build): change a strategy to handle import.meta.env and un…

d7eb09b

…defined key

yuzheng14 force-pushed the fix/define-replace branch from b94ce8d to d7eb09b Compare July 24, 2024 07:05

test(define): renew test case to pass through CI

8a10814

bluwy reviewed Jul 24, 2024

View reviewed changes

packages/vite/src/node/plugins/define.ts Outdated Show resolved Hide resolved

playground/define/vite.config.js Outdated Show resolved Hide resolved

bluwy reviewed Jul 24, 2024

View reviewed changes

packages/vite/src/node/plugins/define.ts Outdated Show resolved Hide resolved

refactor(build): replace manual import.meta.env after replaceDefine

0426753

bluwy added 2 commits July 25, 2024 16:17

refactor: improve sourcemap perf

891fd0e

chore: update snapshots

c8e9034

bluwy approved these changes Jul 25, 2024

View reviewed changes

bluwy added the p3-minor-bug An edge case that only affects very specific usage (priority) label Jul 25, 2024

bluwy mentioned this pull request Jul 25, 2024

When referencing an environment variable that doesnt exist, vite dumps your whole environment into the bundle as an object #17710

Closed

7 tasks

Merge branch 'main' into pr/yuzheng14/17648

af23b75

patak-dev approved these changes Jul 25, 2024

View reviewed changes

patak-dev added this to the 5.4 milestone Jul 25, 2024

jnoordsij mentioned this pull request Jul 26, 2024

Allow for easier exposure handling on import.meta.env #17765

Open

4 tasks

bluwy merged commit 659b720 into vitejs:main Jul 30, 2024
12 checks passed

hi-ogawa added a commit to hi-ogawa/vite-import-meta that referenced this pull request Aug 12, 2024

chore: repro vitejs/vite#17648 (comment)

bca7985

bluwy mentioned this pull request Aug 12, 2024

Vite 5.4.0 env var fails when bundling Vite-bundled library #17874

Closed

7 tasks

yuzheng14 mentioned this pull request Aug 12, 2024

fix(build): avoid re-define __vite_import_meta_env__ #17876

Merged

silverwind added a commit to silverwind/vite-import-meta that referenced this pull request Aug 12, 2024

Merge pull request #1 from hi-ogawa/chore-repro

6edcde0

chore: repro for vitejs/vite#17648

clemens mentioned this pull request Sep 16, 2024

ActionCable url is not set correctly when using v3 on rail 7.2 mclintprojects/actioncable-vue#81

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(build): handle invalid JSON in import.meta.env #17648

fix(build): handle invalid JSON in import.meta.env #17648

yuzheng14 commented Jul 10, 2024 •

edited by bluwy

Loading

stackblitz bot commented Jul 10, 2024

yuzheng14 commented Jul 15, 2024

yuzheng14 commented Jul 22, 2024

bluwy commented Jul 23, 2024 •

edited

Loading

yuzheng14 commented Jul 23, 2024

yuzheng14 commented Jul 24, 2024

yuzheng14 commented Jul 25, 2024 •

edited

Loading

bluwy commented Jul 25, 2024

yuzheng14 commented Jul 25, 2024 •

edited

Loading

bluwy commented Jul 25, 2024 •

edited

Loading

bluwy commented Jul 25, 2024

vite-ecosystem-ci bot commented Jul 25, 2024

yuzheng14 commented Jul 25, 2024

bluwy commented Jul 25, 2024

patak-dev left a comment

yuzheng14 commented Jul 25, 2024

silverwind commented Aug 9, 2024 •

edited

Loading

yuzheng14 commented Aug 10, 2024

silverwind commented Aug 11, 2024 •

edited

Loading

yuzheng14 commented Aug 11, 2024

hi-ogawa commented Aug 12, 2024 •

edited

Loading

yuzheng14 commented Aug 12, 2024

hi-ogawa commented Aug 12, 2024 •

edited

Loading

yuzheng14 commented Aug 12, 2024

silverwind commented Aug 12, 2024 •

edited

Loading

yuzheng14 commented Aug 12, 2024

bluwy commented Aug 12, 2024

	if (env && !canJsonParse(env)) {
	const marker = `_${getHash(env, env.length - 2)}_`
	replacementMarkers[marker] = env
	define = { ...define, 'import.meta.env': marker }
	}

	for (const marker in replacementMarkers) {
	result.code = result.code.replaceAll(marker, replacementMarkers[marker])
	}

	Object.assign(processEnv, {
	'process.env': `{}`,
	'global.process.env': `{}`,
	'globalThis.process.env': `{}`,
	'process.env.NODE_ENV': JSON.stringify(nodeEnv),
	'global.process.env.NODE_ENV': JSON.stringify(nodeEnv),
	'globalThis.process.env.NODE_ENV': JSON.stringify(nodeEnv),
	})

fix(build): handle invalid JSON in import.meta.env #17648

fix(build): handle invalid JSON in import.meta.env #17648

Conversation

yuzheng14 commented Jul 10, 2024 • edited by bluwy Loading

Bug Reproduction

Reason

Solution

Radical Solution

stackblitz bot commented Jul 10, 2024

yuzheng14 commented Jul 15, 2024

yuzheng14 commented Jul 22, 2024

bluwy commented Jul 23, 2024 • edited Loading

yuzheng14 commented Jul 23, 2024

yuzheng14 commented Jul 24, 2024

yuzheng14 commented Jul 25, 2024 • edited Loading

bluwy commented Jul 25, 2024

yuzheng14 commented Jul 25, 2024 • edited Loading

bluwy commented Jul 25, 2024 • edited Loading

bluwy commented Jul 25, 2024

vite-ecosystem-ci bot commented Jul 25, 2024

yuzheng14 commented Jul 25, 2024

bluwy commented Jul 25, 2024

patak-dev left a comment

Choose a reason for hiding this comment

yuzheng14 commented Jul 25, 2024

silverwind commented Aug 9, 2024 • edited Loading

yuzheng14 commented Aug 10, 2024

silverwind commented Aug 11, 2024 • edited Loading

yuzheng14 commented Aug 11, 2024

hi-ogawa commented Aug 12, 2024 • edited Loading

yuzheng14 commented Aug 12, 2024

hi-ogawa commented Aug 12, 2024 • edited Loading

yuzheng14 commented Aug 12, 2024

silverwind commented Aug 12, 2024 • edited Loading

yuzheng14 commented Aug 12, 2024

bluwy commented Aug 12, 2024

yuzheng14 commented Jul 10, 2024 •

edited by bluwy

Loading

bluwy commented Jul 23, 2024 •

edited

Loading

yuzheng14 commented Jul 25, 2024 •

edited

Loading

yuzheng14 commented Jul 25, 2024 •

edited

Loading

bluwy commented Jul 25, 2024 •

edited

Loading

silverwind commented Aug 9, 2024 •

edited

Loading

silverwind commented Aug 11, 2024 •

edited

Loading

hi-ogawa commented Aug 12, 2024 •

edited

Loading

hi-ogawa commented Aug 12, 2024 •

edited

Loading

silverwind commented Aug 12, 2024 •

edited

Loading