gatsby-plugin-mdx doesn't render "complex" Katex formulas

[nikoladev]

Description

I don't know the exact terms to describe it, but "complex" Katex formulas prevent MDX from rendering. (Feel free to change the title into something more specific)

I've figured out a workaround by passing the formula as a string to a custom component that runs the string through katex.renderToString and then passes the output to dangerouslySetInnerHTML. That might help in pinpointing where the problem occurs.

(⚠️ Note for anyone looking for a QUICKFIX: use this component to render Katex formulas that break rendering)

I have already talked to the MDX team about this, and they managed to get it working by removing gatsby-plugin-mdx from the equation (see the link for codesandboxes with examples).

Steps to reproduce

There is a minimal reproducible example here with instructions on how to make it fail as well as the workaround in action.

This is the formula in question:

$$
\begin{aligned}
  \bar{x} &= \dfrac{0+1+2+ \ldots +8}{9} = 3.56 \\
  \bar{y} &= \dfrac{23.5+25+ \ldots +37.5}{9}=30
\end{aligned}
$$

Expected result

The formula should render.

Actual result

I get this error:

error "gatsby-plugin-mdx" threw an error while running the onCreateNode lifecycle:

unknown: Expecting Unicode escape sequence \uXXXX (18:9)

  16 | {x}
  17 | {` &= \\dfrac`}
> 18 | {0+1+2+ \ldots +8}
     |          ^
  19 | {9}
  20 | {` = 3.56 `}
  21 | {`\\`}

[nikoladev]

In the discussion with the MDX team @wooorm mentions that several dependencies are out of date by several years. That might have something to do with it, though I can't say anything for certain.

I did see that several unified projects that Gatsby is using are from 3 years ago. E.g., mdast-util-to-hast is at v3, whereas we’re at v10 already

[nikoladev]

Can this issue get the not stale label?

[KyleAMathews]

Marked "not stale". Yeah, we need to do soon a major release for the whole suite of plugins with updated dependencies.

[nikoladev]

@KyleAMathews That's great to hear! I'm doing a lot of work with MDX at the moment and am willing to help test out any changes.

[nikoladev]

Just to add another edge case:

$x<3$ also doesn't work as described here by @janmaltel

[tnorlund]

@nikbelikov How were you able to get the \begin{align*}...\end{align*} to work properly? Which versions are you using?

[nikbelikov]

@nikbelikov How were you able to get the \begin{align*}...\end{align*} to work properly? Which versions are you using?

please, mention @nikoladev instead :)

[nikoladev]

@tnorlund If you look at my minimal reproducible example I describe the workaround there. I basically pass the formula as a string to a custom component and then use katex.renderToString to correctly render it.

I think it's best if you clone that repo and play with it.

[tnorlund]

@tnorlund If you look at my minimal reproducible example I describe the workaround there. I basically pass the formula as a string to a custom component and then use katex.renderToString to correctly render it.

I think it's best if you clone that repo and play with it.

Ok. Will do. I got everything working inside a math block in an MDX file. I just learned that Katex uses 'aligned' rather than 'align*'. Is this still an issue?

[KyleAMathews]

👋 hey we're working on updating the gatsby remark ecosystem to the latest versions of things which should help out with this issue. If you'll want to try testing things out to see if katex works better, that'd be really helpful! #30385

[nikoladev]

Ok. Will do. I got everything working inside a math block in an MDX file. I just learned that Katex uses 'aligned' rather than 'align*'. Is this still an issue?

I can't really say. I'm not the Latex person in our team, I just set up Gatsby for the Latex person 🤷‍♀️

Questions like this would probably be better for the actual Katex team.

[nikoladev]

@KyleAMathews Thanks!

At the moment I don't have the time. So it would be great if anyone else can clone my repo and try out the updated plugins to see if it fixes the bugs.

[nikoladev]

Update: as mentioned by @vladar in this discussion, gatsby-plugin-mdx will only be updated once MDX V2 is released. Since this issue relies on gatsby-plugin-mdx the updates mentioned in #30385 won't solve this issue.

[kimbaudi]

@nikoladev - I tried your complex math formula on a gatsby site that I am working on and it seems to render properly.

the complex math formula:

$$
\begin{aligned}
  \bar{x} &= \dfrac{0+1+2+ \ldots +8}{9} = 3.56 \\
  \bar{y} &= \dfrac{23.5+25+ \ldots +37.5}{9}=30
\end{aligned}
$$

screenshot of rendered complex formula (in dark mode):

However, I am using the latest npm packages (except for remark-math which needs to be on v3 until gatsby-plugin-mdx uses remark 13. Also, I am using rehype-katex instead of remark-html-katex (i believe rehype-katex is recommended over remark-html-katex. I didn't clone your repo, but just copy/pasted the complex math formula from your repo into my gatsby site to see there was a problem. Apparently, it seems to render correctly.

So in your gatsby-config.js, change:

    {
      resolve: 'gatsby-plugin-mdx',
      options: {
        defaultLayouts: {
          default: require.resolve(path.resolve(__dirname, 'src', 'templates', 'Page.jsx')),
        },
        remarkPlugins: [
          require('remark-math'),
          require('remark-html-katex'),
        ],
      },
    },

to:

    {
      resolve: 'gatsby-plugin-mdx',
      options: {
        defaultLayouts: {
          default: require.resolve(path.resolve(__dirname, 'src', 'templates', 'Page.jsx')),
        },
        remarkPlugins: [require('remark-math')],
        rehypePlugins: [require('rehype-katex')],
      },
    },

And make sure to install rehype-katex (and remove remark-html-katex)

[tnorlund]

@kimbaudi, why rehype-katex over remark-html-katex? I've got working with remark-html-katex just fine. From what I've heard, the blockers here is remark as a whole...

[kimbaudi]

@tnorlund according to the README in remark-math it is discouraged to use remark-html-katex and use rehype-katex or rehype-mathjax instead.

[kimbaudi]

the reason why I commented on this issue is because the "complex" katex formula rendered correctly for me using remark-math and rehype-katex (instead of remark-html-katex). I don't think there is an issue here.

From what I've heard, the blockers here is remark as a whole...

I'm assuming you mean not being able to upgrade remark-math from v3.0.1 to the latest 4.0.0, Yes, there is an issue with gatsby-plugin-mdx not being able to update to remark 13 until mdx is updated to v2 (which is out of Gatsby control). But gatsby-plugin-mdx seems to render the complex katex formula fine for me.

[nikoladev]

@kimbaudi Thanks for looking into it :)

I tried your fix, and it doesn't seem to work. You can try it out in this branch to see if I did something wrong.

Are you sure you removed the 3 backticks around the broken examples in index.mdx when testing out your changes? You can see what I mean by looking at the changes to index.mdx in this diff.

[kimbaudi]

I tried your fix, and it doesn't seem to work.

I looked into your repo and I see 2 issues in your package.json file:

1. you are not using latest packages for gatsby, gatsby-plugin-mdx, react
2. you are using v2.0.0-next.8 version of @mdx-js/mdx and @mdx-js/react

if you change your package.json from

    "gatsby": "2.26.0",
    "katex": "0.12.0",
    "react": "17.0.1",
    "react-dom": "17.0.1",
    "@mdx-js/mdx": "v2.0.0-next.8",
    "@mdx-js/react": "v2.0.0-next.8",
    "gatsby-plugin-mdx": "1.4.0",
    "gatsby-source-filesystem": "2.5.0",
    "rehype-katex": "4.0.0",
    "remark-math": "3.0.1"

to

    "gatsby": "3.2.1",
    "katex": "0.13.2",
    "react": "17.0.2",
    "react-dom": "17.0.2",
    "@mdx-js/mdx": "^1.6.22",
    "@mdx-js/react": "^1.6.22",
    "gatsby-plugin-mdx": "2.2.0",
    "gatsby-source-filesystem": "3.2.0",
    "rehype-katex": "4.0.0",
    "remark-math": "3.0.1"

then complex katex formulas work

[kimbaudi]

i think your issue is that you are trying to use v2next versions of mdx, which isn't currently supported in gatsby-plugin-mdx.

[nikoladev]

i think your issue is that you are trying to use v2next versions of mdx, which isn't currently supported in gatsby-plugin-mdx.

Yeah, the Gatsby team is still waiting for a stable release of MDX and then they'll update gatsby-plugin-mdx to use the new version of MDX as well as the new versions of remark-* packages. That should most likely solve this issue.

Thanks for thinking along :)

[janmaltel]

Hi, this comment is just to provide another example of a formula that I couldn't get to work and to provide a workaround (for this specific case), in case someone is struggling with a similar problem.

My specific (but imo quite general case) is to get some LaTeX formula into a subscript block. While simple things such as indices (e.g., $a_3$) work fine, I couldn't produce formulas such as $\mathbb{E}_{a_t \sim \pi(s_t)}$, which should render

but which suffers from the same problem as the one described in the original post, i.e.,

SyntaxError: unknown: Expecting Unicode escape sequence \uXXXX (615:6)

  613 | {E}
  614 | {`_`}
> 615 | {a_t \sim \pi(s_t)}
      |       ^
  616 | {`$`}</p>
  617 | /Users/malte/... unknown: Expecting Unicode escape sequence \uXXXX (615:6)

One quite annoying way to make this work is to use a sequence of empty \phantom{} blocks and subscript everyone of them with a single character or symbol. For the example mentioned above this would be to use

$\mathbb{E}_{a_t} \phantom{}_\sim \phantom{}_\pi \phantom{}_{(s_t)}$,

which I used to produce the image shown above. Maybe thhis helps someone until the update with MDX v2 is out :)

[kimbaudi]

@janmaltel - i tried $\mathbb{E}_{a_t \sim \pi(s_t)}$ on my gatsby app and I am able to render the math equation just fine:

BTW, I am using @mdx-js/mdx and @mdx-js/react v1.6.22. Also, I am using remark-math v3.0.1 and rehype-katex v5.0.0.

Here is the relevant snippet from my gatsby-config.js:

const remarkMath = require(`remark-math`);
const rehypeKatex = require(`rehype-katex`);

// ...
{
  resolve: `gatsby-plugin-mdx`,
  options: {
    defaultLayouts: {
      default: path.resolve(__dirname, `src/templates/mdx-template.jsx`),
    },
    remarkPlugins: [remarkMath],
    rehypePlugins: [rehypeKatex],
  },
},
// ...

Out of curiosity, are you using mdx v2? Because I've only encountered the error SyntaxError: unknown: Expecting Unicode escape sequence \uXXXX when trying to upgrade from mdx v1 to mdx v2 on a next.js project.

On a side note, there is a major overhaul in how mdx is parsed in v2 (see mdx-js/mdx#1039) and that could affect the way math equations are rendered. Also, there will be breaking changes mdx v2 so some things that worked in markdown or mdx v1 no longer will work in mdx v2 (i.e., html comments will no longer be supported in mdx v2. see mdx-js/mdx#1042). As already mentioned by the author of mdx, MD !== MDX. I want to add that MDXv1 !== MDXv2

[janmaltel]

Thank you for the info and your time @kimbaudi !

Out of curiosity, are you using mdx v2?
Yes indeed, I'm using MDX v2, similar to @nikoladev 's setup.

I tried to adapt my package.json file according to your post but ended up in some never-ending dependency hell. :(
I think I'll stay with my current setup for now and wait for MDX v2 to be properly supported by gatsby.

@nikoladev you wrote this earlier:

Yeah, the Gatsby team is still waiting for a stable release of MDX and then they'll update gatsby-plugin-mdx to use the new version of MDX as well as the new versions of remark-* packages. That should most likely solve this issue.

Do you know of an issue / thread that tracks progress on this?

[janmaltel]

Woooooow, I only now saw your KatexBlock and KatexInline components @nikoladev ! This is 1000x better and easier to use than my stupid hack. This saved my day -- thank you!!

[nikoladev]

Woooooow, I only now saw your KatexBlock and KatexInline components @nikoladev ! This is 1000x better and easier to use than my stupid hack. This saved my day -- thank you!!

You're welcome :)

Thanks for letting me know it was hard to find, I've updated the intro post for anyone who is looking for a quickfix.

Do you know of an issue / thread that tracks progress on this?

Not really. Here's a response from a Gatsby employee on them waiting for a stable version 2 of MDX. Only thing I can think of is subscribing to MDX releases (you can click on Watch at the top and then select Custom and then choose Releases to get Github notification of releases).

[vrabe]

MDX v2 was out in February. But we still need to wait Gatsby for its MDX v2 support. #34421

[LekoArts]

Hi!

We merged #35650 which means that a new major version of gatsby-plugin-mdx will be released on August 16. You can already try it out by installing gatsby-plugin-mdx@next & gatsby@next.

With this you can use MDX v2. Please open an issue with a minimal reproduction if this is still not solved yet. Thanks!