Markdown lexer edge case with links and square brackets (rare)

[Ravlen]

Name of the lexer
Markdown

This might be too rare to bother put effort into it, but this is something I run into from time to time as some of our engineers like linking to TOML section examples, and the section names are usually [section] or [[section]].

I'm raising the issue because::

• It renders correctly with our markdown renderer (Kramdown): https://docs.gitlab.com/runner/configuration/autoscale.html#runner-global-options (see the two links in the table, which are correct).
• It does color properly with whatever code coloring VS Code uses:

Code sample

Does not work with GitHub's coloring either:

[This is a link `with backticks`](example.com)
[This is a link to a TOML section `[with brackets]` (and backticks)](example.com)
[This is a link to a TOML section `[[with double brackets]]` (and backticks)](example.com)

Does not work with Rouge:

Works fine with VS Code:

Additional context
Yeah, it's a rare edge case. Now that I saw one fix earlier this week (that was fast, btw!), I'll see if I can spot any issues with the regex (though I'm still learning).

[pyrmont]

Agreed this should parse. It works fine in CommonMark.

[pyrmont]

The problem is with how the regex uses non-greedy matching in rules like this:

rouge/lib/rouge/lexers/markdown.rb

Line 105 in 65207be

rule %r/(!?\[)(#{edot}*?)(\])/ do

The problem that I can imagine is that a more complex set of rules potentially leads you to a state where it's difficult to know on an initial pass whether you're in a link or not.

[Ravlen]

@pyrmont Would it be crazy to add the opening paren for the link, as a way to identify the full link-text section?

- rule %r/(!?\[)(#{edot}*?)(\])/ do
+ rule %r/(!?\[)(#{edot}*?)(\]\()/ do

[pyrmont]

It would need to be either a ( or a [ (since you can have inline links and reference links) and it might be preferable to do it as a lookahead so that the rest of the lexing logic stays the same.

[Ravlen]

(Sidenote, I just noticed you and jneen are both in Japan, which is funny because I'm in Kagoshima!)

[Ravlen]

@pyrmont I struggle with lookahead logic and never got a good grasp of it, but I could easily add a ( OR [:

rule %r/(!?\[)(#{edot}*?)(\]\(|\]\[)/ do

[pyrmont]

You can think of a lookahead as a special kind of 'group' (using (?=...)). So in this case: /(!?\[)(#{edot}*?)(?=\][\[(])/. It's difficult to read because of all the escaping but in plain English, that's a ] and then a character class consisting of either [ or (.

[Ravlen]

@pyrmont I read through some regex docs and I'm starting to grasp it, but with the tool I was using your example wasn't working, though a slight modification worked. Does this make sense?

rule %r/(!?\[)(#{edot}*?)(?=\]\(|\]\[)/ do

Once I got that working, I started to wonder if I should use lookbehind to skip the first bracket as well. Does this make sense?

rule %r/(!?(?<=\[))(#{edot}*?)(?=\]\(|\]\[)/ do

[Ravlen]

I got it working now, turns out the online tool was just buggy and I had to refresh and it just started working again. I'll add the lookahead to the PR.