Issues upgrading unist-util-visit, remark, and remark-html

[kentcdodds]

I've got a plugin I use for syntax highlighting. Upgrading to the latest version of unist-util-visit gives me TypeScript issues with this:

    visit(tree, "code", (node) => { // node is "never" 🤔
      if (
        !node.lang ||
        !node.value ||
        node.lang === "txt" ||
        node.lang === "text"
      ) {
        return;
      }
      // ...
    });

Also, I have issues with my previous approach of creating pre elements. It's hard to show a small example, so for the full code expand this:

Full code example

/*!
 * Adapted from
 * - ggoodman/nostalgie
 *   - MIT https://github.com/ggoodman/nostalgie/blob/45f3f6356684287a214dab667064ec9776def933/LICENSE
 *   - https://github.com/ggoodman/nostalgie/blob/45f3f6356684287a214dab667064ec9776def933/src/worker/mdxCompiler.ts
 */
/*!
 * Forked from @ryanflorence/md
 */
import { getHighlighter, IShikiTheme, loadTheme, setCDN } from "shiki";
import type { Highlighter } from "shiki";
import type { Element, Text } from "hast";
import LRUCache from "lru-cache";
import rangeParser from "parse-numeric-range";
import type { Plugin } from "unified";
// @ts-expect-error
import path from "path-browserify";

export { setCDN };

setCDN("https://unpkg.com/@kentcdodds/md-temp@1.0.1/");

////////////////////////////////////////////////////////////////////////////////
let highlighter: Highlighter;

let cache = new LRUCache<string, Element>({
  maxSize: 1024 * 1024 * 32, // 32 mb
  sizeCalculation(value, key) {
    return JSON.stringify(value).length + (key ? key.length : 0);
  },
});

////////////////////////////////////////////////////////////////////////////////
// cache it so the browser doesn't keep fetching it on typing
let base16Theme: IShikiTheme;

async function loadBase16() {
  if (base16Theme) return base16Theme;
  if (typeof window !== "undefined") {
    base16Theme = await loadTheme("dist/base16.json");
    return base16Theme;
  } else {
    return loadTheme(path.resolve(__dirname, "base16.json"));
  }
}

let remarkCodeBlocksShiki: Plugin = () => {
  return async function transformer(tree) {
    const [{ visit, SKIP }, { htmlEscape }] = await Promise.all([
      import("unist-util-visit"),
      import("escape-goat"),
    ]);
    let theme = await loadBase16();
    highlighter =
      highlighter ||
      (await getHighlighter({
        themes: [theme],
      }));

    let themeName = "base16";

    visit(tree, "code", (node) => {
      if (
        !node.lang ||
        !node.value ||
        node.lang === "txt" ||
        node.lang === "text"
      ) {
        return;
      }

      // TODO: figure out how this is ever an array?
      let meta = Array.isArray(node.meta) ? node.meta[0] : node.meta;

      let metaParams = new URLSearchParams();
      if (meta) {
        let linesHighlightsMeta = meta.match(/^\[(.+)\]$/);
        if (linesHighlightsMeta) {
          metaParams.set("lines", linesHighlightsMeta[1]);
        } else {
          metaParams = new URLSearchParams(meta.split(/\s+/).join("&"));
        }
      }

      let language = node.lang as string;
      let addedLines = parseLineRange(metaParams.get("add"));
      let removedLines = parseLineRange(metaParams.get("remove"));
      let highlightLines = parseLineRange(metaParams.get("lines"));
      let numbers = !metaParams.has("nonumber");

      let cacheKey = JSON.stringify([
        language,
        highlightLines,
        addedLines,
        removedLines,
        node.value,
      ]);
      let nodeValue = cache.get(cacheKey);

      if (!nodeValue) {
        let fgColor = convertFakeHexToCustomProp(
          highlighter.getForegroundColor(themeName) || ""
        );
        let bgColor = convertFakeHexToCustomProp(
          highlighter.getBackgroundColor(themeName) || ""
        );

        let tokens = highlighter.codeToThemedTokens(
          node.value as string,
          language,
          themeName
        );
        let isDiff = addedLines.length > 0 || removedLines.length > 0;
        let diffLineNumber = 0;
        let children = tokens.map(
          (lineTokens, zeroBasedLineNumber): Element => {
            let children = lineTokens.map((token): Text | Element => {
              let color = convertFakeHexToCustomProp(token.color || "");
              let content: Text = {
                type: "text",
                // Do not escape the _actual_ content
                value: token.content,
              };

              return color && color !== fgColor
                ? {
                    type: "element",
                    tagName: "span",
                    properties: {
                      style: `color: ${htmlEscape(color)}`,
                    },
                    children: [content],
                  }
                : content;
            });

            children.push({
              type: "text",
              value: "\n",
            });

            const lineNumber = zeroBasedLineNumber + 1;
            const highlightLine = highlightLines.includes(lineNumber);
            const removeLine = removedLines.includes(lineNumber);
            const addLine = addedLines.includes(lineNumber);
            if (!removeLine) {
              diffLineNumber++;
            }

            return {
              type: "element",
              tagName: "span",
              properties: {
                className: "codeblock-line",
                dataHighlight: highlightLine ? "true" : undefined,
                dataLineNumber: numbers ? lineNumber : undefined,
                dataAdd: isDiff ? addLine : undefined,
                dataRemove: isDiff ? removeLine : undefined,
                dataDiffLineNumber: isDiff ? diffLineNumber : undefined,
              },
              children,
            };
          }
        );

        let metaProps: { [key: string]: string } = {};
        metaParams.forEach((val, key) => {
          if (key === "lines") return;
          metaProps[`data-${key}`] = val;
        });

        nodeValue = {
          type: "element",
          tagName: "pre",
          properties: {
            ...metaProps,
            dataLineNumbers: numbers ? "true" : "false",
            dataLang: htmlEscape(language),
            style: `color: ${htmlEscape(
              fgColor
            )};background-color: ${htmlEscape(bgColor)}`,
          },
          children: [
            {
              type: "element",
              tagName: "code",
              children,
            },
          ],
        };

        cache.set(cacheKey, nodeValue);
      }

      let data = node.data ?? (node.data = {});

      node.type = "element";
      data.hProperties ??= {};
      data.hChildren = [nodeValue];

      return SKIP;
    });
  };
};

////////////////////////////////////////////////////////////////////////////////

let parseLineRange = (param: string | null) => {
  if (!param) return [];
  return rangeParser(param);
};

// The theme actually stores #FFFF${base-16-color-id} because vscode-textmate
// requires colors to be valid hex codes, if they aren't, it changes them to a
// default, so this is a mega hack to trick it.
function convertFakeHexToCustomProp(color: string) {
  return color.replace(/^#FFFF(.+)/, "var(--base$1)");
}

export { remarkCodeBlocksShiki };

I have a little manual test for this

Expand for test

const fs = require("fs/promises");
const { remarkCodeBlocksShiki } = require("../dist/index.js");

async function getProcessor() {
  const { remark } = await import("remark");
  const { html } = await import("remark-html");
  return remark().use(remarkCodeBlocksShiki).use(html);
}

async function processMarkdown(content) {
  let processor = await getProcessor();
  let result = await processor.process(content);
  return result.value;
}

async function build() {
  let result = await processMarkdown(`
# Cool

\`\`\`tsx lines=2,4 nonumber filename=beef.ts
function foo(arg) {
  console.log(arg[0]);
  doStuff(arg);
  return "beef";
}
\`\`\`

\`\`\`txt
text lang
\`\`\`

\`\`\`
no lang
\`\`\`

\`\`\`tsx add=3-5,7 remove=2,6,8-9
const foo = [
  'removed',
  'added',
  'added',
  'added',
  'removed',
  'added',
  'removed',
  'removed',
  'untouched',
]
\`\`\`

Yes, very nice.
`);

  await fs.writeFile(
    "./fixture/index.html",
    `
  <!doctype html>
  <html>
    <head>
      <title>Fixture</title>
      <style>
        :root {
          --base00: #2d2d2d;
          --base01: #393939;
          --base02: #515151;
          --base03: #999999;
          --base04: #b4b7b4;
          --base05: #cccccc;
          --base06: #e0e0e0;
          --base07: #ffffff;
          --base08: #f2777a;
          --base09: #f99157;
          --base0A: #ffcc66;
          --base0B: #99cc99;
          --base0C: #66cccc;
          --base0D: #6699cc;
          --base0E: #cc99cc;
          --base0F: #a3685a;
        }
        [data-add] {
          color: green;
        }
        [data-remove] {
          color: red;
        }
      </style>
    </head>
    <body>
      ${result}
    </body>
  </html>
  `
  );
}

build();

The error I get when running this is:

file:///Users/kentcdodds/Desktop/md/node_modules/mdast-util-to-markdown/lib/index.js:97
  throw new Error('Cannot handle unknown node `' + node.type + '`')
        ^

Error: Cannot handle unknown node `element`
    at Object.unknown (file:///Users/kentcdodds/Desktop/md/node_modules/mdast-util-to-markdown/lib/index.js:97:9)
    at Object.one [as handle] (file:///Users/kentcdodds/Desktop/md/node_modules/zwitch/index.js:52:17)
    at containerFlow (file:///Users/kentcdodds/Desktop/md/node_modules/mdast-util-to-markdown/lib/util/container-flow.js:34:17)
    at Object.root (file:///Users/kentcdodds/Desktop/md/node_modules/mdast-util-to-markdown/lib/handle/root.js:13:10)
    at Object.one [as handle] (file:///Users/kentcdodds/Desktop/md/node_modules/zwitch/index.js:52:17)
    at toMarkdown (file:///Users/kentcdodds/Desktop/md/node_modules/mdast-util-to-markdown/lib/index.js:51:22)
    at compiler (file:///Users/kentcdodds/Desktop/md/node_modules/remark-stringify/lib/index.js:16:12)
    at Function.stringify (file:///Users/kentcdodds/Desktop/md/node_modules/unified/lib/index.js:290:12)
    at file:///Users/kentcdodds/Desktop/md/node_modules/unified/lib/index.js:398:36
    at done (file:///Users/kentcdodds/Desktop/md/node_modules/unified/lib/index.js:337:11)

Node.js v18.4.0

This is always a big mystery to me. Is there a new way to create HTML elements these days? Using astexplorer.net shows that HTML elements are a node type html. So should I be compiling code samples to HTML strings myself now?

[ChristianMurphy]

Howdy!
In one of the recent type update was to allow for more narrow definition and stricter validation on nodes.
For example being able to better differentiate between a HAST or and MDAST node.
In general this makes code safer and easier to work with, it does require some migration.

For plugins the generic on the Plugin type can be used to define what types of node(s) it accepts.
For example it could accept mdast Root https://github.com/remarkjs/remark-validate-links/blob/5a2641a0f24049429f4ce0132e839fb1ec24ad18/lib/index.js#L54

Or if the code is being used outside a plugin context, the type can be narrowed with the techniques outlined in https://unifiedjs.com/learn/recipe/narrow-node-typescript/

[kentcdodds]

Thanks! That fixed up the never type by using Plugin<Array<{}>, MDAST.Root> for my plugin type 👍

[wooorm]

An example of a similar plugin, using starry-night instead of shiki, and not doing line highlighting or passing data props, is: https://github.com/wooorm/starry-night#example-integrate-with-unified-remark-and-rehype

[kentcdodds]

Thanks. Unfortunately I'd like to be able to do this with the Markdown, before some valuable information is lost in the translation between MD to HTML.

[wooorm]

Reading through the code:

• node.meta is never an array
• node.type = "element" should not be there I guess. It’d be better to make this a rehype plugin (which works on elements and stuff) instead of a remark plugin (which works on markdown constructs).
  Now you need to embed data on markdown constructs of how to create elements later (more on why here: https://github.com/remarkjs/remark-rehype#what-is-this, more on how “subdata” is embedded here: https://github.com/syntax-tree/mdast-util-to-hast#fields-on-nodes).
  For a quick solution, I think you could remove that line, and it might just work?

[wooorm]

Unfortunately I'd like to be able to do this with the Markdown, before some valuable information is lost in the translation between MD to HTML.

You particularly mean meta, right? In that case, you could pass just that through:

export default function myRemarkPluginPassingMetaAsData() {
  return function (tree) {
    visit(tree, 'code', (code) => {
      if (code.meta) {
        const data = code.data || (code.data = {})
        const props = data.hProperties || (data.hProperties = {})
        props.dataMeta = code.meta
        // ^-- Says to pass a `data-meta="..."` attribute through
      }
    })
  }
}

[kentcdodds]

Ok, sounds like I need to rework this plugin and switch it to a rehype plugin. Thanks for the tips!

[wooorm]

Oh btw, I forgot that data.meta is passed through to the HTML code element node.
So you won’t need the above plugins. If there was a meta string in the markdown, it will be passed through: https://github.com/syntax-tree/mdast-util-to-hast/blob/9d699fb335c4d4ffbf2282fefb9c601d56db9091/lib/handlers/code.js#L28-L30.