Multiline math #717

Merged
merged 6 commits into from Dec 12, 2017

Conversation

Projects
None yet
3 participants
Member

mpacer commented Dec 6, 2017

This is a modification of the logic #715 (now #716) that takes the approach I suggested in #715 (comment).

In particular it uses the MathBlockLexer as a way to pass the multiline sections of math blocks through to the MathInlineLexer, which will do the actual parsing.

I don't want this to be merged until we can figure out how to simplify the test examples as they are making a set of already hard-to-read tests even more hard-to-read.

I also really want to know why our previous test with a LaTeX environment with a * in its name (the first instance of the test) didn't work to catch this.
Specifically, that test case is:

Lines 124 to 128 in 3e203ce

 "\\begin{equation*}\n" + ("\\left( \\sum_{k=1}^n a_k b_k \\right)^2 " "\\leq \\left( \\sum_{k=1}^n a_k^2 \\right) " "\\left( \\sum_{k=1}^n b_k^2 \\right)\n") + "\\end{equation*}"),

danilobellini and others added some commits Dec 4, 2017

 Add Markdown block lexer/parser for LaTeX blocks 
The block lexer/parser was splitting equations like this

$$x = 2$$

So the inline lexer/parser was never seeing the whole equation, and
it wasn't getting properly rendered. This fixes such breaking by
adding a block-level lexer/parser to the LaTeX equations written as
either $$...$$ or \$...\$

The inline "block math" parsing code was kept as is, since the above
equation could have been part of a paragraph like "$$x = 2$$" to keep
the compatibility with Jupyter Notebook rendering engine (and because
there's a test enforcing that behavior)
 1db9666 
 use the multiline math block lexer as a pass through to the inlinelexer 
 542f6de 
 add docstrings to explain the logic of the classes 
 2ab8b3f 

takluyver reviewed Dec 6, 2017

 identify math content spanning multiple lines. These are used by the MathBlockLexer. """ multi_math_str = "|".join([r"(^\$\$.*?\$\$)",

takluyver Dec 6, 2017

Member

Do you know about the re.VERBOSE option? It lets you split a regex over several lines and have comments, without needing to assemble the string yourself.

mpacer Dec 6, 2017

Author Member

Could you give an example in this case?

Because I genuinely prefer reading individual regexes joined with a pipe to indicate alternations to one long regex that is individually difficult to parse.

takluyver Dec 6, 2017

Member

Something like this:

multiline_math = re.compile(r"""(^\$\$.*?\$\$)|
(^\\\\$.*?\\\\$)|
(^\\begin\{([a-z]*\*?)\}(.*?)\\end\{\4\})""",
re.DOTALL | re.VERBOSE)

Personally I prefer that to the string-joining approach, but you're the one writing it, so you get to decide on code style.

mpacer Dec 6, 2017

Author Member

Interesting, in this case I find them equally easy to read. it's more in cases like this in mistune https://github.com/lepture/mistune/blob/92b7f32664bad1a4b3740ee81eda47e5246e780f/mistune.py#L120-L134 where individual chunks need to split multiple lines, and accordingly detecting that an alternation is happening is a lot harder.

mpacer Dec 7, 2017

Author Member

Since I find these to be equally readable, I'm going to go with your style in this case. But I also figured out a way to simplify it further (since we don't really need the groups for grabbing the content, as you pointed out).

mpacer Dec 7, 2017 • edited

Author Member

can I ask your rationale for disliking the string joining approach? I'm just curious as to why it would be dispreferred so that I can update my preferences accordingly.

mpacer Dec 7, 2017

Author Member

Ok… actually I'm going to keep using the join method, but with simpler component strings.

I ended up not finding the pipe's use for alternations and the bitwise OR to be equally readable as not needing the second flag. I had skimmed over that detail at a first glance.

takluyver reviewed Dec 6, 2017

 super(MarkdownWithMath, self).__init__(renderer, **kwargs) def output_multiline_math(self): return self.inline(self.token["text"])

takluyver Dec 6, 2017

Member

Does this mean we're parsing it again with the inline parser? Can we skip that, given that we just want to put it in the output unmodified?

mpacer Dec 6, 2017 • edited

Author Member

We're parsing it semantically for the first time in the inline parser. We actually aren't putting it in the output unmodified, because we need to escape the individual parts of it for the purposes of putting > and < in html. That occurs in the rendering step.

Member

Gotcha, thanks.

takluyver reviewed Dec 6, 2017

 """Add token to pass through mutiline math.""" self.tokens.append({ "type": "multiline_math", "text": m.group(1) or m.group(2) or m.group(3)

takluyver Dec 6, 2017

Member

Since the three groups are each the whole of their alternative, this is equivalent to m.group(0), right?

mpacer Dec 6, 2017

Author Member

Nice! Much cleaner, grazie.

mpacer added some commits Dec 6, 2017

 Use m.group(0) as it captures the entirety of the main captured group 
thx @takluyver!
 d0105e7 
 replace m.group(0) with m.string, arrange associated Grammars and Lexers 
 6a2fd6b 

takluyver reviewed Dec 7, 2017

 """Add token to pass through mutiline math.""" self.tokens.append({ "type": "multiline_math", "text": m.string

takluyver Dec 7, 2017

Member

I think this should be m.group(0), which is the full match, not m.string, which is the string it tried to match against.

 3│ re.match(r'\d+-', '12-AB').string
3> '12-AB'

4│ re.match(r'\d+-', '12-AB').group(0)
4> '12-'


I haven't checked how mistune works, but m.string could be wrong, whereas m.group(0) can't be. So let's go with the more robust option.

Member Author

mpacer commented Dec 8, 2017

 Ok simplified the tests, escaped them and fixed the m.string thing.

Member

takluyver commented Dec 9, 2017

 I'm happy with implicitly joined strings like this. But I think they should be e.g. $$x\n rather than $$x\\n. The former makes a string which ends with a newline (because Python's parser interprets \n as such), whereas the latter makes a string which ends with the characters \n, because the backslash is escaped. See e.g. line 129 in the same file for an example of what I mean.
Member Author

mpacer commented Dec 10, 2017 • edited

 Oops! That's what I get for "s/\/\\/g"ing. I will fix it in a bit.
 return m.string to m.group(0); simplify and escape new tests 
 773b403 

Member Author

mpacer commented Dec 11, 2017

 Ok… force pushed onto the last bit since this conceptually seemed like part of the immediately previous commit.

takluyver merged commit f849836 into jupyter:master Dec 12, 2017 1 check passed

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

Merged