The corner case for smart emphasis and footnotes

[Totktonada]

I meet corner case of using smart emphasis with the footnotes extension. Let’s consider example:

some text[^1]

[^1]:
    _emphasis works_

    _emphasis doesn't work_

The problem appears only with last footnote’s paragraph and when ending undescore is last symbol in the paragraph. As I know footnotes extension appends non-breaking space and a backreference at end of a last paragraph for each footnote. Smart emphasis does (?!\w) lookahead (by the way, it’s looks as equial to \b) after ending underscore, so non-breaking space, I guess, blocks regex from matching.

The example easy to fix by replacing underscores with stars or by disabling smart emphasis. However, I think, it’s good to be aware of weird corner cases. The issue is informational and has no much impact over me (I’m fallbacks to stars).

[waylan]

@Totktonada thanks for the report. This is definitely a corner case, but a bug nonetheless. Given the extreme edge that one has to go to to find this and the fact that there is an east workaround, a fix is not terribly high on my priority list. I expect this will be addressed with the 3.0 refactor. However, a PR would certainly be welcome if anyone wanted to address it before that.

[facelessuser]

This actually isn't too bad. Footnotes passes the content above with no newlines at the end of the block. Simply adding a new line causes it to pass through and emphasis is handled. I think just inserting a newline when passing any chunk back through the parser is a reasonable solution to this. It shouldn't have any affect on anything, and it resolves this issue. No need to wait for 3.0. I can through together a pull.

[facelessuser]

It works but then all footnotes have an extra new line in HTML output which isn't ideal as it changes all the tests. I'll see if I can come up with something that doesn't cause this.

[waylan]

@facelessuser yes, that has been an annoyance of mine from time to time. If you notice, none of the tests include a newline at the end of the file/ouput (since before I became involved in the project). I have wanted to change this for years, but didn't want to have to update all the tests. In the end, I've always just worked around it.

[facelessuser]

I found something interesting. Footnotes is in a list and we send the content. By setting the parser state to 'list' before parsing chunks. Everything works:

self.parser.state.set('list')
self.parser.parseChunk(li, self.footnotes[id])

I wonder if this would also affect #412.

[facelessuser]

No go for #412 but it seems it is sensitive to state. I would need to play with this more. Can't promise a clean fix yet. But I'll see what can be done.

[facelessuser]

Seems like the first block needs to be parsed as looselist and subsequent blocks can get parsed as list:

            blocks = self.footnotes[id].split('\n\n')
            if len(blocks) > 1:
                self.parser.state.set('looselist')
                self.parser.parseBlocks(li, blocks[0:1])
                self.parser.state.reset()
                self.parser.state.set('list')
                self.parser.parseBlocks(li, blocks[1:])
                self.parser.state.reset()
            else:
                self.parser.state.set('looselist')
                self.parser.parseBlocks(li, blocks)
                self.parser.state.reset()

I would need to test this more, but I think it is possible to fix this. #412 is a totally different matter and looks like it will be a headache to fix...we'll see if I get to that one.

[waylan]

Ah, yes, a "looselist" is a list which contains block level children (the name comes from the fact that there is a blank line between list items in the Markdown syntax). Whereas a "list" (which would not contain any blank lines in the Markdown syntax) does not contain any block level elements (the text is a direct child of the <li> with no wrapping <p> tag). In the case of footnotes, we always want a "loose list," even if the footnote is a single paragraph (the text should still get wrapped in a <p> tag).

[facelessuser]

In the case of footnotes, we always want a "loose list," even if the footnote is a single paragraph (the text should still get wrapped in a

tag).

Yes, but it appears you need to switch to 'list' after the first block or things like _emphasis doesn't work_ break at the end.

I borrowed this idea from the list processor:

            # parse first block differently as it gets wrapped in a p.
            li = util.etree.SubElement(lst, 'li')
            self.parser.state.set('looselist')
            firstitem = items.pop(0)
            self.parser.parseBlocks(li, [firstitem])
            self.parser.state.reset()

This still doesn't explain the quirks of #412, but it handles cases like mentioned in the OP. I have a feeling there is something more down the pipe I'm missing which is preventing me from figuring out #412. But that is not priority to address this issue.

[facelessuser]

I may dig deeper in this and find I'm wrong with the switching to list. I've only looked at this briefly. I need to double check outputs and make sure I'm not missing anything.