Add ignoreTokens prop to allow skipping parse of specific Markdown tokens

[bjudson]

Here's a proof of concept for the ability to disable parsing of specific tokens. I'm not excited about the level of messing with the parser prototype. And it does seem that some tokenizers just can't be disabled without throwing errors (eg a list tokenizer reference is hardcoded, so it can't be removed without breaking other things). In the test, I included the full list of tokenizers, and commented out the ones that don't seem to be cleanly removable.

I'm conflicted about this, because the ability to disable parsing of some tokens is a requirement for us, and this seems like the best way to do it. On the other hand, remark-parse clearly wasn't designed for disabling these parsers, and the documented method for doing so is a brittle hack.

If it makes sense to move forward, there are some more tests, error handling & refactoring I plan to do, but just want to get some general feedback before spending more time on it.

Eager to hear others' thoughts on this approach.

[rexxars]

Excellent work. I feel like this is really hacky, but if that's the way remark proposes this should work, I guess it has to be this way. Have you tried asking on Gitter or through GitHub issues whether there is any way to fix the list issue you mentioned?

[bjudson]

Thanks for the quick response. I didn't realize quite how brittle this approach would be until I worked on the implementation, but I agree it's worth reaching out to remark about this. I'll open an issue and see if we can make some improvements on their end.

[bjudson]

I was about to start making a remark plugin for this, and discovered that one already exists (not sure why my previous searches didn't bring it up):
https://github.com/zestedesavoir/zmarkdown/tree/master/packages/remark-disable-tokenizers

I installed this and it is working well for me so far:

import remarkDisableTokenizers from "remark-disable-tokenizers";

<ReactMarkdown
  plugins={[
    [
      remarkDisableTokenizers,
      { block: ["blockquote", "indentedCode", "fencedCode", "table"], inline: ["code"] }
    ]
  ]}>
  {markdownContent}
</ReactMarkdown>

As far as I'm concerned, we can close this PR and link to the plugin from the related issues.

[rexxars]

Cool, thanks for investigating, @bjudson !