New formats parser / Backticks in math mode

[diomidov]

What's wrong?

Backticks in math mode produce unexpected results. I think preprocessMarkdownTicks replaces backticked expression with TICK0ENDTICK, then KaTeX wraps each character in a span, and putTicksBack can't find TICK0ENDTICK anymore.

I can't think of an easy way to fix this without breaking dollar-signs-in-code-blocks 😢.

How to Reproduce

https://coauthor.csail.mit.edu/coauthor-dev/m/mjhWj7zaP6H9ydm7A

Screenshots

[edemaine]

Yuck, nice find. I don't see a clean way to fix this. The hacky way would be to process ticks and dollar signs in one pass, keeping track of being in vs. out of each type.

But I think it's finally time to switch Coauthor's formats (in particular $ processing and all \macros) into a Markdown plugin. Then these options can be processed at only the right times, in particular, not within ticks and not e.g. in the middle of an HTML tag (though I already take some pains to avoid this, I doubt it's exactly right).

For formats other than Markdown, my plan is to then build my own "non-Markdown" processor that can run these same plugins, but with just a couple of simple rules for processing everything else (namely, HTML tags and text; and for LaTeX, some typography transmuting). I might build this out of markdown-it by just disabling almost all rules / extracting the few relevant rules, or I might build it from scratch given that it doesn't need to do much. But I think it's doable, and it would correct format processing a lot.

At the same time, I could fix the argument parsing to LaTeX commands to do an actual scan with brace counting, instead of the current regular expression hack that has a limited brace depth. This was annoying to do when using String.replace, but now that we'll be matching and eating a prefix of the string, it should be much easier.

Ideally the same plugins can be used for the Markdown syntax highlighter.

Before doing this, we should consider whether to switch from markdown-it to remark/micromark. See comparison table. #212 somewhat related to this decision; apparently micromark can give position information but unclear whether remark can.

[edemaine]

Another bad example: `@edemaine` expands to <code><a href="...">@Erik Demaine</a></code>

Edit: This particular case was fixed.