Markdown syntax highlighting bug due to a dollar sign ($) within a code block

[SultanOrazbayev]

Describe the bug

The syntax highlighting breaks down once a dollar sign is used within a code block (see the attached image). I tried using syntax highlighting (e.g. specifying bash after the triple backticks), but it doesn't help.

Screen Shot 2019-07-06 at 17.20.40.pdf

To Reproduce
Steps to reproduce the behavior:

1. Create a new Markdown file and open it with the editor in Jupyter lab.
2. Type the following code:

# Hello

Sample code:

    ```
    echo $HOME
    ```

# Next section

Note: no syntax highlighting above.

3. See error (the section title is not highlighted after the dollar sign).

Expected behavior
The syntax highlighting of the # Next section and any supporting code below.

Desktop (please complete the following information):

• OS: MacOS
• Browser: Chrome 75
• JupyterLab 1.0.1

** Potentially related cases (jupyter notebook)

#1834

#5266
#547

[SultanOrazbayev]

To add: the markdown renders and displays correctly, it's only syntax highlighting within the jupyterlab editor.

[ian-r-rose]

Thanks for the report @SultanOrazbayev! I have confirmed that the CodeMirror demo here does not display this rendering problem, so the bug is on our end (n.b., the classic notebook also shows this bug).

This is probably a mistake in our multiplexing codemirror mode here:

jupyterlab/packages/codemirror/src/codemirror-ipythongfm.ts

Lines 32 to 58 in 079115d

	return CodeMirror.multiplexingMode(
	gfmMode,
	{
	open: '$$',
	close: '$$',
	mode: texMode,
	delimStyle: 'delimit'
	},
	{
	open: '$',
	close: '$',
	mode: texMode,
	delimStyle: 'delimit'
	},
	{
	open: '\\(',
	close: '\\)',
	mode: texMode,
	delimStyle: 'delimit'
	},
	{
	open: '\\[',
	close: '\\]',
	mode: texMode,
	delimStyle: 'delimit'
	}
	// .. more multiplexed styles can follow here

which switches to LaTeX mode upon encountering a $ symbol.

I'm not sure what the fix would look like just now -- we would need to identify that we are in a code fence and not turn on LaTeX mode.

[SultanOrazbayev]

Thanks for the follow-up!

I'm not sure if this is an option, but could a keyword at the beginning of the code block be used to disable texMode? e.g. if the code block contains shell, bash, console or similar:

```shell
echo $HOME
```

```bash
echo $HOME
```

[austincauthon]

A workaround for this issue is to use the unicode $ in place of using a $. This will render a $ inside your rendered document, but not affect your syntax highlighting in your editor.

[dturaev]

I also observed problems with the bash syntax highlighting, with differing results between Jupyterlab (1.1.3) and classic Notebook.
In Jupyterlab, markdown cell (content and rendered):

Switching to Classic Notebook:

The wrong syntax coloring in the unrendered cell is slightly annoying, but wouldn't bother me much, in case the rendered result would be correct. Inserting a unicode code like $ within the code block didn't work for me, it doesn't render as a character.
The offending symbol seems to be the $ sign, the syntax highlighting is correct if it's removed.

[RA80533]

I encountered this strange quirk today. Phew, I'm glad its root cause is known.

In the interim of any changes to this behavior, it may be worth conveying this functionality to the user in some way.

[fperez]

Confirming that this remains in current JLab (3.0.12) but doesn't occur in classic, same file open side-by-side simultaneously in both:

(glad I found this open already, was just about to open a new one as I scratched my head for a while wondering if I'd messed up my own markdown)...

Not sure how hard the fix is, wish I could help :) I'm trying to dog-food hard by using JLab for everything where I'd normally use other tools, hopefully we can iron the smaller kinks that way...

[telamonian]

I wonder if this is related to #8645. Both may boil down to issues with detecting where you're trying to enter into "mathmode"

[fperez]

@telamonian agreed - I haven't looked at the logic, but basically the math mode detection shouldn't cross code boundaries, nicely highlighted by having two:

[tannerlegvold]

Just as a further note, this also occurs in JupyterLab 3.0.14 using <code> </code> blocks instead of backtick code blocks.

[telamonian]

For reference, the way the mathmode parsing works is that we take this Very Special regex:

jupyterlab/packages/rendermime/src/latex.ts

Line 16 in 32c98cc

const MATHSPLIT = /(\$\$?|\\(?:begin|end)\{[a-z]*\*?\}|\\[{}$]|[{}]|(?:\n\s*)+|@@\d+@@|\\\\(?:\(|\)|\[|\]))/i;

and use it to split markdown cells into non-math and math parts. It's likely that in order to fix this issue either MATHSPLIT or the code that directly consumes it will have to be fixed.

[RA80533]

Hm, that's interesting because, as @ian-r-rose noted, there's functionality for changing the context to TeX upon encountering the delimiter elsewhere (see codemirror-ipythongfm.ts). I suspect the regex is unnecessary for the reason of CodeMirror having built-in support for this kind of behavior (which, as aforementioned, appears to have been implemented).

[mattlegro]

I don't know if this is useful information, but even using the unicode workaround doesn't fix the rendering issue. Text between two $ also is italicised and bolded. In the following example, it renders "15 billion.....at" as bold and italicized,

* Netflix is reported to invest $15 billion on new content; Prime Video and Apple TV+ at $6 billion; Hulu, Disney+ and HBO Max are all coming in around $3 billion

[krassowski]

I looked into this one. The clean solution would be to re-write the CodeMirror.multiplexingMode to use the context of the outer mode (gfmMode) when deciding whether to proceed with switching to the inner mode, but this is a difficult thing as we would need to manage concurrent parsers (modes) consuming the same stream (as does the overlayMode) which might end up very error-prone (and might not be accepted upstream given the slow migration to CodeMirror 6). I found a simple workaround which should make everyone happy for now, hopefully without adding maintenance overhead - see #10867.

[dturaev]

Thanks a lot for looking into this! I'm still not completely happy, though (sorry about that)... With JupyterLab 3.1.9:

[krassowski]

@dturaev thanks for bringing attention to this one. I see that editor works well now but there are some rendering issues ;this is a very different part of the codebase, but I guess it could be solved together with #10861 - could you post your comment there so it is not forgotten, please?

[dturaev]

Thanks for the clarification, will do.

[krassowski]

There are further fixes to the remaining edge cases in #11479 and #11488 - hopefully both will get included in the next 3.2.x release. Please help testing (e.g. on binder) if you can. If you discover new edge cases after release please open a new issue and tag me.