rehype plugins can't pick md inside html

[devanshj]

MDX 2 introduces parsing md inside html, but afaiu it seems to not work correctly with rehype plugins. You'll understand better with a reproduction...

import { compile } from '@mdx-js/mdx'
import rehypeShiki from "@leafac/rehype-shiki"
import shiki from "shiki"

const test = async src => String(await compile(src, {
  rehypePlugins: [
    [rehypeShiki, {
      highlighter: await shiki.getHighlighter({ theme: "github-light" })
    }]
  ]
}))

console.log("// --- TEST 1 " + "-".repeat(30))
console.log(await test(`
  \`\`\`js
  let hello = "world"
  \`\`\`
`))

console.log("\n")
console.log("// --- TEST 2 " + "-".repeat(30))
console.log(await test(`
  <div>

  \`\`\`js
  let hello = "world"
  \`\`\`

  </div>
`))

Output

(You can run this code and try it yourself in this stackblitz)

// --- TEST 1 ------------------------------
/*@jsxRuntime automatic @jsxImportSource react*/
import {Fragment as _Fragment, jsx as _jsx, jsxs as _jsxs} from "react/jsx-runtime";
function MDXContent(props = {}) {
  const {wrapper: MDXLayout} = props.components || ({});
  return MDXLayout ? _jsx(MDXLayout, Object.assign({}, props, {
    children: _jsx(_createMdxContent, {})
  })) : _createMdxContent();
  function _createMdxContent() {
    const _components = Object.assign({
      pre: "pre",
      code: "code",
      span: "span"
    }, props.components);
    return _jsx(_Fragment, {
      children: _jsx(_Fragment, {
        children: _jsx(_components.pre, {
          className: "shiki",
          style: {
            backgroundColor: "#ffffff"
          },
          children: _jsx(_components.code, {
            children: _jsxs(_components.span, {
              className: "line",
              children: [_jsx(_components.span, {
                style: {
                  color: "#CF222E"
                },
                children: "let"
              }), _jsx(_components.span, {
                style: {
                  color: "#24292F"
                },
                children: " hello "
              }), _jsx(_components.span, {
                style: {
                  color: "#CF222E"
                },
                children: "="
              }), _jsx(_components.span, {
                style: {
                  color: "#24292F"
                },
                children: " "
              }), _jsx(_components.span, {
                style: {
                  color: "#0A3069"
                },
                children: "\"world\""
              })]
            })
          })
        })
      })
    });
  }
}
export default MDXContent;

// --- TEST 2 ------------------------------
/*@jsxRuntime automatic @jsxImportSource react*/
import {jsx as _jsx} from "react/jsx-runtime";
function MDXContent(props = {}) {
  const {wrapper: MDXLayout} = props.components || ({});
  return MDXLayout ? _jsx(MDXLayout, Object.assign({}, props, {
    children: _jsx(_createMdxContent, {})
  })) : _createMdxContent();
  function _createMdxContent() {
    const _components = Object.assign({
      pre: "pre",
      code: "code"
    }, props.components);
    return _jsx("div", {
      children: _jsx(_components.pre, {
        children: _jsx(_components.code, {
          className: "language-js",
          children: "let hello = \"world\"\n"
        })
      })
    });
  }
}
export default MDXContent;

I'd expect TEST 1 and TEST 2 to produce same results (except for a difference of a div) but they don't, the code block gets transformed in TEST 1 but it's left as it is in TEST 2. Looking at the source code of @leafac/rehype-shiki both should have been transformed, so I don't think this is a problem with @leafac/rehype-shiki. (Here's my failed attempt to test @leafac/rehype-shiki with only html but it apparently doesn't transform in either case idk why)

Is this a bug? If not any pointers on how I can achieve the behavior I want would be nice.

Thanks! (Also mdx is awesome :)

[wooorm]

(Here's my failed attempt to test @leafac/rehype-shiki with only html but it apparently doesn't transform in either case idk why)

That, uhm, points at the most likely problem? If you can’t get a project to work in different examples, then maybe the problem is that project.
I verified that by trying another highlighters (rehype-highlight), which did work.

I’m guessing that this logic they have is way too strict and doesn’t account for <pre>s in <div>s.

[devanshj]

That, uhm, points at the most likely problem? If you can’t get a project to work in different examples, then maybe the problem is that project.

That would have been case if I could have made the transformation work for at least TEST 1 (as the mdx version of that works) but here it's not working for both cases, which means I'm not using the project correctly in test-rehype-shiki.mjs (I'm quite confused why it won't work for either case, I'm not doing anything fancy, could you tell me what I'm doing wrong?)

What's interesting is that, afaiu, it does work with remark-parse and remark-hype, I cloned the repo and wrapped this test case with a div and it still passes. So I thought maybe the problem is with mdx.

I’m guessing that this logic they have is way too strict and doesn’t account for <pre>s in <div>s.

But reading the logic you can tell that it's not strict to not account for pre inside div

I understand I haven't proved the problem is with mdx, so I'll take another look and see where the problem actually is.

[wooorm]

try replacing .use(rehypeParse) with .use(rehypeParse, { fragment: true }). The first test case suddenly starts working.

That plugin has few stars. It’s not downloaded a lot. It uses finicky logic. It doesn’t work where other projects do work. I’m pretty sure the problem is that plugin

[devanshj]

try replacing .use(rehypeParse) with .use(rehypeParse, { fragment: true }). The first test case suddenly starts working

Ah now that only the first test case works then it implies the problem is with the plugin not mdx.

That plugin has few stars. It’s not downloaded a lot.

Yeah but that's the only project that I found that works with shiki and can produce multiple blocks of the same code with the different themes passed (so that I can use display: none on light or dark block according to the media query).

Perhaps I'll try to fix the issue or maybe write my own plugin.

Thanks for your time, you did help to narrow down where the issue is!

[wooorm]

Hmmmm okay! Yeah, there’s a lot of shiki plugins out there.
Sounds like a complex problem. But I’d assume dark/light mode to be a problem most folks have, it can’t be solved with just CSS?

[devanshj]

Yeah it is kinda complexish. Shiki's folks have documented two approaches to this. One is generating two blocks (light and dark) and other approach uses a special "css-variables" theme to color with css variables only instead of generating two blocks (which makes it kinda a css-only solution). But the latter approach compromises with highlighting and makes it less granular hence I went with the former (which also is more straightforward to me).

There's also a discussion/issue open in the shiki repo for a more first class dark mode support, in case you're interested in reading more on this matter.

[devanshj]

An update: The problem was that the plugin traverses only root's children and not the whole tree. So it can be solved by simply traversing the whole tree, I thought it would be trivial to do that with unist-util-visit and I tried to fix the original plugin but apparently no good (opened an issue there). I ended up writing my own plugin. It's no rocket science but in case it helps anyone, it looks like the following (it's not equivalent the original but minimal enough for my needs).

import shiki from "shiki"
import { unified } from "unified"
import { visit } from "unist-util-visit"
import rehypeParse from "rehype-parse"

const themes = ["github-light", "github-dark"]

export default () => async tree => {
  let highlighter = await shiki.getHighlighter({ themes })

  visit(tree, (node, index, parent) => {
    if (!(
      node.tagName === "pre" &&
      node.children?.[0]?.tagName === "code" &&
      node.children?.[0]?.properties?.className?.[0].startsWith("language-") &&
      node.children?.[0]?.children?.[0]?.type === "text"
    )) return

    let code = node.children[0].children[0].value
    let lang = node.children[0].properties.className[0].slice("language-".length)
    parent.children.splice(
      index,
      1,
      ...themes.map(theme => {
        let n =
          unified()
          .use(rehypeParse, { fragment: true })
          .parse(highlighter.codeToHtml(code, { theme, lang }))

        n.children[0].properties.className.push(theme)
        return n
      })
    )
  })
}