Syntax highlighting via Shiki requires additional configuration

[adamdehaven]

@farnabaz I'm working on implementing the <MDCRenderer> via our changes in #154 and am running into an issue whereby I cannot get wasm to work properly on Cloudflare Pages.

Small reproduction repo: https://github.com/adamdehaven/mdc-wasm-repro

I enabled experimental wasm support in my nuxt.config.ts nitro.experimental.wasm: true (I think this is needed?) but then when using wrangler dev preview chokes.

I think I need to somehow include the shiki/onig.wasm locally in my project based on the Shiki docs, but unsure how/where to do so.

Would you be able to provide some guidance? The markdown itself renders; however, syntax highlighting obviously is not currently working for me.

[adamdehaven]

@antfu maybe you can provide some guidance since it's shiki-related?

[farnabaz]

You don't need to enable Nitro's experimental WASM. MDC module should handle WASM configuration for you. Just remove the experimental option and try the preview.

[adamdehaven]

When I simply use the MDC module in a regular Nuxt project, I get a different error:

WebAssembly.instantiate(): Wasm code generation disallowed by embedder

[adamdehaven]

@farnabaz I've put together a small reproduction: https://github.com/adamdehaven/mdc-wasm-repro with simple instructions.

Install deps, and then if you run pnpm dev or even pnpm build && pnpm preview everything locally works fine.

However, if you run pnpm build and then pnpm wrangler pages dev dist to serve with wrangler, you can successfully load the app and the markdown renders, but the syntax highlighting is not functional and you see the following wasm error in your console:

This matches what I see when I do an actual deploy to Cloudflare pages.

[farnabaz]

I did manage to run and also deploy your reproduction is first attempt. https://a7626d8f.mdc-test-a.pages.dev/
Also, I did manage to deploy my fork directly from Cloudflare dashboard https://mdc-wasm-repro.pages.dev/

Something might be happening in your machine that cause this behavior.

• Check your node version (try with 18)
• Try force install maybe pnpm i --force

[adamdehaven]

@farnabaz if you look at either of your deploy links above, notice the syntax highlighting is not working. The deploy itself works, but the Shiki syntax highlighting isn't functional.

pnpm run dev (local)

(notice the colors in the code block)

Cloudflare Pages deploy

(notice no syntax highlighting like on local)

[adamdehaven]

For reference, the end of this line is why I originally thought I needed to enable the nitro.experimental.wasm setting:

mdc/src/module.ts

Line 149 in 373d380

useWasmAssets: !nuxt.options.dev && !!nuxt.options.nitro.experimental?.wasm

[farnabaz]

My bad, I thought it was raising error on deployment.
I'll dig this

[adamdehaven]

@farnabaz so I saw the Shiki Nuxt blog post published today which indicates this should work out of the box with Cloudflare workers/Pages.

What are we missing? 🤔 I dug in over the weekend but unfortunately didn't find anything that made this work

[adamdehaven]

@farnabaz I chatted with Sébastien on X and we determined a couple things:

1. nitropack must be updated to use at minimum version of 2.9.x. By doing this, I was able to fix my reproduction in this PR
2. nitro.experimental.wasm: true is required
3. There still appears to be an issue when utilizing in a monorepo with Layers (I think?) as shown by this separate reproduction: https://github.com/adamdehaven/mdc-wasm-monorepo
   i. For this second reproduction:
   • The repro throws warnings/errors during dev mode, but builds successfully
   • My actual project throws warnings/errors for both dev and build, so I'm trying to get this reproduced as well

It seems the issue could be related to operating within a monorepo?

[adamdehaven]

Ok I think I figured it out with help from @Atinux 🎉 but may need help figuring out the best place to supplement the documentation.

Turns out it is undocumented (I believe) that you need to add shiki/onig.wasm in your nuxt.config > vite.build.rollupOptions.external array 😅 in combination with enabling nitro.experimental.wasm.

Adding these two settings fixes syntax highlighting for the Cloudflare Pages deploy ✨ however, it breaks syntax highlighting for dev mode.

I think the easiest solution would be to only enable the experimental wasm for production, meaning your nuxt.config would look similar to this:

$production: {
  nitro: {
    // !Important: we only want to enable the wasm feature in production since it will break syntax highlighting when running the dev server
    experimental: {
      wasm: true,
    },
  },
},
vite: {
  build: {
    rollupOptions: {
      external: [
        'shiki/onig.wasm', // !Important: externalize the wasm import
      ],
    },
  },
},

Is adding the experimental wasm setting only for $production ideal here, or is there a better optimization that would be ideal?

[adamdehaven]

Here's a PR to supplement the documentation: #165

Ideally, I'd like to see if it's possible to leave the wasm enabled for $development as well

[farnabaz]

Should be fixed in #167
I just deployed your repro with @nuxtjs/mdc-edge@latest https://45865333.mdc-wasm-repro.pages.dev/