Markdown mode highlighting isn't CommonMark compliant

[b-kelly]

Umbrella issue to track multiple bugs.

Reported on Meta.SE in the following posts:

• https://meta.stackexchange.com/a/360038/395497
• https://meta.stackexchange.com/a/360050/395497
• https://meta.stackexchange.com/a/360153/395497
• https://meta.stackexchange.com/a/380383/51 - formats based on escaped md symbols
• https://meta.stackexchange.com/a/380381/51 - lang-* doesnt work in RT, works on site
• https://meta.stackexchange.com/a/380339/51

[Glorfindel83]

Probably related bug when editing https://meta.stackexchange.com/a/363991/295232:

[b-kelly]

@Glorfindel83 Agreed, that looks related to me and is likely due to the mixing of * (emphasis) and ** (strong emphasis). We use highlight.js as the backing syntax highlighter, which uses regex to detect syntax. My guess is that the regex is not perfect in this scenario and is causing this issue.

Without committing to a specific timeline here (stay tuned), this is relatively high on our priority list to address.

[adunndevster]

@b-kelly - I've submitted a fix for this issue into the highlight.js. Maybe you can help promote the PR?

It was SUPER simple. Really the parser was just being over vigilant in trying to match the ending characters for bold/italics.

highlightjs/highlight.js#3456

[V2Blast]

I just encountered that bold-and-italics problem myself (well, I noticed it a while ago; I just didn't figure out the source of the problem until now). I also noticed something else: If the bolded-and-italicized text occurs within a blockquote, then the rest of the text from that point onwards is also displayed as if it’s grayed-out (i.e. as if it were part of a blockquote), as well as bolded and italicized.

(Another issue with the Stacks editor’s inline preview/"highlighting" in Markdown mode that doesn't seem to have been mentioned above: It behaves as if the syntax for italics/bold formatting works across multiple lines (e.g. with the opening * being on one line and the closing * being on a different line) – even though that's not how the post will actually be rendered when the post is saved.

[yellis]

@b-kelly given that highlightjs/highlight.js#3457 was merged into hljs, is this resolved now (at least in part)? Is stacks editor up to date with hljs?

[b-kelly]

Stacks-Editor should be up-to-date with hljs (check package.json for exact version). This issue however still needs a ton of work, likely involving a massive amount of upstream changes or switching to a different highlighting approach entirely (for markdown mode, not the rich-text or baked post).

[yellis]

@b-kelly is it possible to fully define the project somewhere and to potentially break down the work needed into smaller bits?

[yellis]

@b-kelly I just added:

• https://meta.stackexchange.com/a/380383/51 - adds formatting based on escaped md symbols
• https://meta.stackexchange.com/a/380381/51 - lang-* doesnt work for highlighting in RT (works on site)

Hopefully will also be addressed in your suggested approach.

If either isn't included here, we should split off into separate issues.