Error occur when build with next-mdx-remote and next.js standalone option #283

EiffelFly · 2022-09-23T05:46:32Z

Context

Hi!! Thank you for building and maintaining such a great package. I am encountering some issues when using codehike + next-mdx-remote + Nextjs with the standalone option.

During development, the code goes smoothly. But after building it with the Next.js standalone option, the app can't find some file that codehike relied on and throw Internal Error.

error running remarkCodeHike [Error: ENOENT: no such file or directory, open '/app/node_modules/.pnpm/shiki@0.10.1/node_modules/shiki/languages/abap.tmLanguage.json'] {
code-hike-test-2 |   errno: -2,
code-hike-test-2 |   code: 'ENOENT',
code-hike-test-2 |   syscall: 'open',
code-hike-test-2 |   path: '/app/node_modules/.pnpm/shiki@0.10.1/node_modules/shiki/languages/abap.tmLanguage.json'
code-hike-test-2 | }

The way to reproduce

Please clone this repo https://github.com/EiffelFly/code-hike-remote-docker
install everything and use the script npm run build
Activate the standalone server with node ./.next/standalone/server.js
goto http://localhost:3000

I have set up a docker image if this could help.

npm run docker:app:develop
goto http://localhost:3000

The text was updated successfully, but these errors were encountered:

thecuvii · 2022-10-02T15:52:44Z

same with MDX-Bundler.

RequestId: eeabce19-9e2c-4811-a253-f8e2bb47d004 Error: Runtime exited with error: exit status 1
Runtime.ExitError
2022-10-02T15:50:03.592Z	ceffc43d-627d-4def-b9ef-37d29e9be86d	ERROR	error running remarkCodeHike [Error: ENOENT: no such file or directory, open '/var/task/node_modules/.pnpm/shiki@0.10.1/node_modules/shiki/languages/abap.tmLanguage.json'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/var/task/node_modules/.pnpm/shiki@0.10.1/node_modules/shiki/languages/abap.tmLanguage.json'
}
✘ [ERROR] [plugin @mdx-js/esbuild] Error: ENOENT: no such file or directory, open '/var/task/node_modules/.pnpm/shiki@0.10.1/node_modules/shiki/languages/abap.tmLanguage.json'
_mdx_bundler_entry_point-285f59bf-4135-4dab-93ee-7e00da3e41fc.mdx:0:0:
0 │ # GFM
╵ ^
2022-10-02T15:50:03.651Z	ceffc43d-627d-4def-b9ef-37d29e9be86d	ERROR	Error: Build failed with 1 error:
_mdx_bundler_entry_point-285f59bf-4135-4dab-93ee-7e00da3e41fc.mdx:0:0: ERROR: [plugin: @mdx-js/esbuild] Error: ENOENT: no such file or directory, open '/var/task/node_modules/.pnpm/shiki@0.10.1/node_modules/shiki/languages/abap.tmLanguage.json'
    at failureErrorWithLog (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:1566:15)
    at /var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:1024:28
    at runOnEndCallbacks (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:1438:61)
    at buildResponseToResult (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:1022:7)
    at /var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:1134:14
    at responseCallbacks.<computed> (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:671:9)
    at handleIncomingPacket (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:726:9)
    at Socket.readFromStdout (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:647:7)
    at Socket.emit (node:events:527:28)
    at addChunk (node:internal/streams/readable:315:12) {
  errors: [
    {
      detail: [Error],
      id: '',
      location: [Object],
      notes: [],
      pluginName: '@mdx-js/esbuild',
      text: "Error: ENOENT: no such file or directory, open '/var/task/node_modules/.pnpm/shiki@0.10.1/node_modules/shiki/languages/abap.tmLanguage.json'"
    }
  ],
  warnings: [],
  page: '/content/[category]/[name]/bundler'
}
RequestId: ceffc43d-627d-4def-b9ef-37d29e9be86d Error: Runtime exited with error: exit status 1
Runtime.ExitError

pomber · 2022-10-02T16:55:45Z

Both are using pnpm? I'll try to test it tomorrow

thecuvii · 2022-10-03T00:49:47Z

Ok, that's a problem with shiki and next.js.

Shiki import languages and themes by fs, when run next build, both languages and themes are tree-shaken by next.js (not if you manually import from your code), so not included in production build.

You can check out by build next.js with standalone, then go to .next/standalone/node_modules.

const nextConfig = {
    output: 'standalone'
}

The solution is unstable_includeFiles, see: https://nextjs.org/docs/advanced-features/output-file-tracing#caveats.

// in your next.js page 
export const config = {
  // pnpm in my case, maybe 'node_modules/**/shiki/**/*.json' for npm or yarn.
  unstable_includeFiles: ['node_modules/.pnpm/**/shiki/**/*.json'],
};

After add above lines, CodeHike with next-remote-mdx works fine for me .

thecuvii · 2022-10-03T00:50:34Z

Ok, that's a problem with shiki and next.js.

Shiki import languages and themes by fs, when run next build, both languages and themes are tree-shaken by next.js (not if you manually import from your code), so not included in production build.

You can check out by build next.js with standalone, then go to .next/standalone/node_modules.
const nextConfig = {
    output: 'standalone'
}
The solution is unstable_includeFiles, see: https://nextjs.org/docs/advanced-features/output-file-tracing#caveats.
// in your next.js page 
export const config = {
  // pnpm in my case, maybe 'node_modules/**/shiki/**/*.json' for npm or yarn.
  unstable_includeFiles: ['node_modules/.pnpm/**/shiki/**/*.json'],
};
After add above lines, CodeHike with next-remote-mdx works fine for me .

reference:

https://stackoverflow.com/questions/71422446/nextjs-force-dependency-with-outputstandalone-option

shikijs/shiki#138 (comment)

thecuvii · 2022-10-03T01:53:57Z

Above solution not works for mdx-bundler.

Basically same error with kentcdodds/mdx-bundler#158

EiffelFly · 2022-10-03T07:39:49Z

Both are using pnpm? I'll try to test it tomorrow

Yes i am using pnpm!

EiffelFly · 2022-10-03T07:43:19Z

Ok, that's a problem with shiki and next.js.
Shiki import languages and themes by fs, when run next build, both languages and themes are tree-shaken by next.js (not if you manually import from your code), so not included in production build.
You can check out by build next.js with standalone, then go to .next/standalone/node_modules.
const nextConfig = {
    output: 'standalone'
}
The solution is unstable_includeFiles, see: https://nextjs.org/docs/advanced-features/output-file-tracing#caveats.
// in your next.js page 
export const config = {
  // pnpm in my case, maybe 'node_modules/**/shiki/**/*.json' for npm or yarn.
  unstable_includeFiles: ['node_modules/.pnpm/**/shiki/**/*.json'],
};
After add above lines, CodeHike with next-remote-mdx works fine for me .
reference:

https://stackoverflow.com/questions/71422446/nextjs-force-dependency-with-outputstandalone-option

shikijs/shiki#138 (comment)

This is very interesting, thanks for your help! Let me test it further

Due to #290 and code-hike/codehike#283, In order to use code-hike in Next.js standalone server, we need to provide additional information for Next.js to not tree-shake shiki languages and themes files. This commit solves this problem by using "unstable_includeFiles" config

pomber · 2022-10-08T11:36:59Z

Above solution not works for mdx-bundler.

Basically same error with kentcdodds/mdx-bundler#158

@xWil what's the error with mdx-bundler? it may be a different issue

thecuvii · 2022-10-08T12:46:15Z

kentcdodds/mdx-bundler#158

✘ [ERROR] Could not resolve "@code-hike/mdx/dist/components.cjs.js"
_mdx_bundler_entry_point-228ef4a4-2912-45e8-b923-adb366b11779.mdx:3:109:
3 │ ...eSlot, Scrollycoding} from "@code-hike/mdx/dist/components.cjs.js";
╵                               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can mark the path "@code-hike/mdx/dist/components.cjs.js" as external to exclude it from the bundle, which will remove this error.
2022-10-08T12:44:21.409Z	6601f87c-e5b2-4983-8777-41a6e196398a	ERROR	Error: Build failed with 1 error:
_mdx_bundler_entry_point-228ef4a4-2912-45e8-b923-adb366b11779.mdx:3:109: ERROR: Could not resolve "@code-hike/mdx/dist/components.cjs.js"
    at failureErrorWithLog (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:1566:15)
    at /var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:1024:28
    at runOnEndCallbacks (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:1438:61)
    at buildResponseToResult (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:1022:7)
    at /var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:1134:14
    at responseCallbacks.<computed> (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:671:9)
    at handleIncomingPacket (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:726:9)
    at Socket.readFromStdout (/var/task/node_modules/.pnpm/esbuild@0.15.10/node_modules/esbuild/lib/main.js:647:7)
    at Socket.emit (node:events:527:28)
    at addChunk (node:internal/streams/readable:315:12) {
  errors: [
    {
      detail: undefined,
      id: '',
      location: [Object],
      notes: [Array],
      pluginName: '',
      text: 'Could not resolve "@code-hike/mdx/dist/components.cjs.js"'
    }
  ],
  warnings: [],
  page: '/content/[category]/[name]/bundler'
}
RequestId: 6601f87c-e5b2-4983-8777-41a6e196398a Error: Runtime exited with error: exit status 1
Runtime.ExitError

Local-dev works great, the error above was came from next.js server rendering.

pomber · 2022-10-08T12:54:17Z

you are using mdx-bundler inside getServerSideProps? why not getStaticProps?

thecuvii · 2022-10-08T14:06:52Z

you are using mdx-bundler inside getServerSideProps? why not getStaticProps?

Yeah, all pages are rendered inside getStaticProps, but not all of them are rendered at build stage.

https://nextjs.org/docs/api-reference/data-fetching/get-static-paths

manduks · 2022-10-13T04:31:52Z

this fixed the issue for me on getServerSideProps

// in your next.js page 
export const config = {
  unstable_includeFiles: ['node_modules/**/shiki/**/*.json'],
};

MildTomato · 2022-12-08T17:06:39Z

Also just hit this problem.
Only found the issue by looking at logs in vercel 😂

// bottom of /pages/references/javascript/[...slug].tsx
// yes.. tried them all

export const config = {
  unstable_includeFiles: [
    'node_modules/.pnpm/**/shiki/**/*.json',
    'node_modules/**/shiki/**/*.json',
    'node_modules/.npm/shiki/**/*.json',
    '/var/task/node_modules/shiki/languages/abap.tmLanguage.json',
    'node_modules/shiki/languages/abap.tmLanguage.json',
    'node_modules/shiki/**/*.json',
  ],
}

pomber · 2022-12-15T20:46:35Z

Ideally the syntax highlighting should be done at build time, but I'm going to replace shiki with something custom soon and these problems should disappear.

pomber · 2022-12-17T12:54:13Z

I'm working on PR #308. There's a canary version there if anyone wants to try it and see if it fixes the problem.

pomber · 2023-01-16T10:37:16Z

should be fixed with v0.8.0, let me know how it goes

mrboen94 · 2023-01-29T21:29:26Z

should be fixed with v0.8.0, let me know how it goes

Running v0.8.0, still running into this issue, Ill figure it out and report back though.

pomber · 2023-01-29T21:35:50Z

should be fixed with v0.8.0, let me know how it goes

Running v0.8.0, still running into this issue, Ill figure it out and report back though.

What's the error?

mrboen94 · 2023-01-29T21:38:58Z

10:33:18 PM: ./node_modules/@code-hike/lighter/dist/index.esm.mjs
10:33:18 PM: Module not found: Can't resolve 'fs' in '/opt/build/repo/node_modules/@code-hike/lighter/dist'
10:33:18 PM: Import trace for requested module:
10:33:18 PM: ./node_modules/@code-hike/lighter/dist/index.esm.mjs
10:33:18 PM: ./node_modules/@code-hike/mdx/dist/index.browser.mjs
10:33:18 PM: ./node_modules/@code-hike/mdx/dist/index.browser.mjs
10:33:18 PM: Module not found: Can't resolve 'fs' in '/opt/build/repo/node_modules/@code-hike/mdx/dist'
10:33:18 PM: Import trace for requested module:
10:33:18 PM: ./node_modules/@code-hike/mdx/dist/index.browser.mjs
10:33:18 PM: > Build failed because of webpack errors```

pomber · 2023-01-29T21:41:53Z

10:33:18 PM: ./node_modules/@code-hike/lighter/dist/index.esm.mjs
10:33:18 PM: Module not found: Can't resolve 'fs' in '/opt/build/repo/node_modules/@code-hike/lighter/dist'
10:33:18 PM: Import trace for requested module:
10:33:18 PM: ./node_modules/@code-hike/lighter/dist/index.esm.mjs
10:33:18 PM: ./node_modules/@code-hike/mdx/dist/index.browser.mjs
10:33:18 PM: ./node_modules/@code-hike/mdx/dist/index.browser.mjs
10:33:18 PM: Module not found: Can't resolve 'fs' in '/opt/build/repo/node_modules/@code-hike/mdx/dist'
10:33:18 PM: Import trace for requested module:
10:33:18 PM: ./node_modules/@code-hike/mdx/dist/index.browser.mjs
10:33:18 PM: > Build failed because of webpack errors```

That shouldn't happen, can you open a new issue with the steps to reproduce it?

mrboen94 · 2023-01-29T21:45:41Z

I can do that, but I will need to find out how to reproduce it first, im trying to debug it on my end, then I'll open the issue.

EDIT: It's late so I will have to look into creating reproducible steps and issue tomorrow, in the meantime if you are using NextJs, add this to your next.config:

...
future: {
    webpack5: true,
},
webpack(config) {
    config.resolve.fallback = {
        ...config.resolve.fallback,
        fs: false,
    }
}
...

This was referenced Oct 3, 2022

Bug: Can't use code-hike with PNPM and next-mdx-remote instill-ai/instill-core#908

Closed

feat: replace prism.js with code-hike instill-ai/console#292

Merged

pomber mentioned this issue Oct 16, 2022

not work after deployed to vercel #291

Closed

rfoel mentioned this issue Oct 19, 2022

Usage with Next.js SSR shikijs/shiki#138

Closed

FredKSchott mentioned this issue Nov 10, 2022

🐛 BUG: Code Component breaks on Vercel SSR withastro/astro#5357

Closed

pomber closed this as completed Jan 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error occur when build with next-mdx-remote and next.js standalone option #283

Error occur when build with next-mdx-remote and next.js standalone option #283

EiffelFly commented Sep 23, 2022

thecuvii commented Oct 2, 2022

pomber commented Oct 2, 2022

thecuvii commented Oct 3, 2022

thecuvii commented Oct 3, 2022

thecuvii commented Oct 3, 2022

EiffelFly commented Oct 3, 2022

EiffelFly commented Oct 3, 2022 •

edited

Loading

pomber commented Oct 8, 2022

thecuvii commented Oct 8, 2022

pomber commented Oct 8, 2022

thecuvii commented Oct 8, 2022

manduks commented Oct 13, 2022

MildTomato commented Dec 8, 2022 •

edited

Loading

pomber commented Dec 15, 2022

pomber commented Dec 17, 2022

pomber commented Jan 16, 2023

mrboen94 commented Jan 29, 2023

pomber commented Jan 29, 2023

mrboen94 commented Jan 29, 2023

pomber commented Jan 29, 2023

mrboen94 commented Jan 29, 2023 •

edited

Loading

Error occur when build with next-mdx-remote and next.js standalone option #283

Error occur when build with next-mdx-remote and next.js standalone option #283

Comments

EiffelFly commented Sep 23, 2022

Context

The way to reproduce

thecuvii commented Oct 2, 2022

pomber commented Oct 2, 2022

thecuvii commented Oct 3, 2022

thecuvii commented Oct 3, 2022

thecuvii commented Oct 3, 2022

EiffelFly commented Oct 3, 2022

EiffelFly commented Oct 3, 2022 • edited Loading

pomber commented Oct 8, 2022

thecuvii commented Oct 8, 2022

pomber commented Oct 8, 2022

thecuvii commented Oct 8, 2022

manduks commented Oct 13, 2022

MildTomato commented Dec 8, 2022 • edited Loading

pomber commented Dec 15, 2022

pomber commented Dec 17, 2022

pomber commented Jan 16, 2023

mrboen94 commented Jan 29, 2023

pomber commented Jan 29, 2023

mrboen94 commented Jan 29, 2023

pomber commented Jan 29, 2023

mrboen94 commented Jan 29, 2023 • edited Loading

EiffelFly commented Oct 3, 2022 •

edited

Loading

MildTomato commented Dec 8, 2022 •

edited

Loading

mrboen94 commented Jan 29, 2023 •

edited

Loading