Can't load shiki vscode-oniguruma in Next 13 react server component

[selenecodes]

I'm trying to use Shiki in a react server component; however, the oniguruma wasm file can't be found. I've tried setting the wasm path; however, when reading through the documentation, this isn't possible for the node.js environment.

Issue:

error - unhandledRejection: Error: ENOENT: no such file or directory, open '(sc_server)\node_modules\vscode-oniguruma\release\onig.wasm'
    at Object.openSync (node:fs:590:3)
    at Object.readFileSync (node:fs:458:35)
    at getOniguruma (webpack-internal:///(sc_server)/./node_modules/shiki/dist/index.js:2178:32)
    at Object.getHighlighter (webpack-internal:///(sc_server)/./node_modules/shiki/dist/index.js:2682:36)
    at getHighlighter (webpack-internal:///(sc_server)/./markdown.plugins.mjs:45:66)
    at Function.rehypePrettyCode (webpack-internal:///(sc_server)/./node_modules/rehype-pretty-code/dist/rehype-pretty-code.js:30114:45)
    at Function.freeze (webpack-internal:///(sc_server)/./node_modules/unified/lib/index.js:117:74)
    at Function.process (webpack-internal:///(sc_server)/./node_modules/unified/lib/index.js:308:19)
    at parseMarkdown (webpack-internal:///(sc_server)/./src/app/(website)/packages/[repository_type]/[...package_route]/page.tsx:52:31)
    at Page (webpack-internal:///(sc_server)/./src/app/(website)/packages/[repository_type]/[...package_route]/page.tsx:86:24)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  errno: -4058,
  syscall: 'open',
  code: 'ENOENT',
  path: '(sc_server)\\node_modules\\vscode-oniguruma\\release\\onig.wasm'
}

What I've tried to do to resolve it:

1. Copy onig.wasm to my src folder
2. Try to load onig.wasm in the shiki.getHighlighter options

const highlighter = await shiki.getHighlighter({
    ...options,
    paths: {
      wasm: `${getShikiPath()}/onig.wasm`, // src/lib/mdx/shiki/onig.wasm
      languages: `${getShikiPath()}/languages/`,
      themes: `${getShikiPath()}/themes/`,
    },
  })

Possible solutions:

1. Might this be solved by allowing users to specify where to load the onig.wasm from in the node.js environment. (see the shiki.getHighlighter paths -> wasm section. (this option currently only works in browser environments*)

[selenecodes]

For the minimal reproduction see github.com/selenecodes/next13-rsc-shiki-wasm-not-found

[Nedilko]

I'm expecting same issue.
Is this ready to be merged?

[ISnackable]

Another workaround is to enable the experimental serverComponentsExternalPackages option in your next.config.js to opt-out from Server Components bundling and use the native Node.js require, which will load the wasm, and the rest of shiki dependencies.

next.config.js:

/** @type {import('next').NextConfig} */
const nextConfig = {
  experimental: {
    appDir: true,
    serverComponentsExternalPackages: ['vscode-oniguruma', 'shiki'],
  },
};

module.exports = nextConfig;

lib/shiki.js:

import { getHighlighter } from 'shiki'

const highlighterPromise = getHighlighter({
  theme: 'nord',
  langs: ['javascript', 'python'],
})

export async function hightlight(code, language) {
  const highlighter = await highlighterPromise
  const output = highlighter.codeToHtml(code, { lang: language })
  return output
}

app/page.js

import { highlight } from 'lib/shiki'

const sampleCode = `import * as React from 'react';
import './App.css';
import Hello from './components/Hello';

const logo = require('./logo.svg');

function App() {
  return (
    <div className="App">
      <div className="App-header">
        <img src={logo} className="App-logo" alt="logo" />
        <h2>Welcome to React</h2>
      </div>
      <p className="App-intro">
        To get started, edit <code>src/App.tsx</code> and save to reload.
      </p>
      <Hello name="TypeScript" />
    </div>
  );
}

export default App;`

export default async function Page() {
  const html = await highlight(sampleCode, 'js')

  return <div dangerouslySetInnerHTML={{ __html: html }}/>
}

[selenecodes]

Another workaround is to enable the experimental serverComponentsExternalPackages option in your next.config.js to opt-out from Server Components bundling and use the native Node.js require, which will load the wasm, and the rest of shiki dependencies.

next.config.js:

/** @type {import('next').NextConfig} */
const nextConfig = {
  experimental: {
    appDir: true,
    serverComponentsExternalPackages: ['vscode-oniguruma', 'shiki'],
  },
};

module.exports = nextConfig;

lib/shiki.js:

import { getHighlighter } from 'shiki'

const highlighterPromise = getHighlighter({
  theme: 'nord',
  langs: ['javascript', 'python'],
})

export async function hightlight(code, language) {
  const highlighter = await highlighterPromise
  const output = highlighter.codeToHtml(code, { lang: language })
  return output
}

app/page.js

import { highlight } from 'lib/shiki'

const sampleCode = `import * as React from 'react';
import './App.css';
import Hello from './components/Hello';

const logo = require('./logo.svg');

function App() {
  return (
    <div className="App">
      <div className="App-header">
        <img src={logo} className="App-logo" alt="logo" />
        <h2>Welcome to React</h2>
      </div>
      <p className="App-intro">
        To get started, edit <code>src/App.tsx</code> and save to reload.
      </p>
      <Hello name="TypeScript" />
    </div>
  );
}

export default App;`

export default async function Page() {
  const html = await highlight(sampleCode, 'js')

  return <div dangerouslySetInnerHTML={{ __html: html }}/>
}

Can confirm this works perfectly, thanks @ISnackable

[Nedilko]

I also confirm this works

[iamnbutler]

Another workaround is to enable the experimental serverComponentsExternalPackages option in your next.config.js to opt-out from Server Components bundling and use the native Node.js require, which will load the wasm, and the rest of shiki dependencies.

next.config.js:

/** @type {import('next').NextConfig} */
const nextConfig = {
  experimental: {
    appDir: true,
    serverComponentsExternalPackages: ['vscode-oniguruma', 'shiki'],
  },
};

module.exports = nextConfig;

lib/shiki.js:

import { getHighlighter } from 'shiki'

const highlighterPromise = getHighlighter({
  theme: 'nord',
  langs: ['javascript', 'python'],
})

export async function hightlight(code, language) {
  const highlighter = await highlighterPromise
  const output = highlighter.codeToHtml(code, { lang: language })
  return output
}

app/page.js

import { highlight } from 'lib/shiki'

const sampleCode = `import * as React from 'react';
import './App.css';
import Hello from './components/Hello';

const logo = require('./logo.svg');

function App() {
  return (
    <div className="App">
      <div className="App-header">
        <img src={logo} className="App-logo" alt="logo" />
        <h2>Welcome to React</h2>
      </div>
      <p className="App-intro">
        To get started, edit <code>src/App.tsx</code> and save to reload.
      </p>
      <Hello name="TypeScript" />
    </div>
  );
}

export default App;`

export default async function Page() {
  const html = await highlight(sampleCode, 'js')

  return <div dangerouslySetInnerHTML={{ __html: html }}/>
}

Thanks so much! This worked. I've been trying to get this to work for a while.

There is a small typo in the example you shared:

// lib/shiki.js:
- export async function hightlight(code, language) {
+ export async function highlight(code, language) {

I put together a minimal app to test this as I was having some small issues getting it set up in my project, hopefully it is useful for someone :)

Code | Demo

[lokimckay]

Unfortunately the above workaround doesn't seem to work if the NextJS app is part of an NPM workspace 😢
vercel/next.js/issues/43433

Here is a workaround script I used to symlink my subfolder /root/site/node_modules to the /root/node_modules

symlink.js

const symlinkDir = require("symlink-dir");
const path = require("path");

const rootModules = path.resolve(__dirname, "../../../../node_modules");
const siteModules = path.resolve(__dirname, "../../../node_modules");

const modules = ["vscode-oniguruma", "shiki"];

modules.forEach((module) => {
  const from = path.resolve(`${rootModules}/${module}`);
  const to = path.resolve(`${siteModules}/${module}`);
  console.log(`Linking ${from} to ${to}`);
  symlinkDir(from, to).catch((err) => console.error(err));
});

package.json

{
"scripts": {
    "prepare": "npm run link-shiki",
    "link-shiki": "node ./src/lib/ci/symlink.js"
  },
}

[marcofranssen]

I'm using a component that does client-side codeblocks:

To make that work I do the following:

cd public
ln -s ../node_modules/shiki/dist dist
ln -s ../node_modules/shiki/themes themes
ln -s ../node_modules/shiki/languages languages
git add public
git commit -m "Add symlinks for shiki client-side integration"

[zm-cttae]

I did this pretty simply - zikaari/onigasm#2 (comment)