Incorrect smart quote conversion for input HTML with markup

[lisaah]

Version: 1.19.2.1
Command: pandoc -o output.html input.html -S

input.html

"Hello <em>world</em>..." "Hello world..."

"Hello world..." "Hello world..."

output.html

&quot;Hello <em>world</em>…&quot; “Hello world…”

"Hello world…" “Hello world…”

Expected:

“Hello <em>world</em>…” “Hello world…”

“Hello world…” “Hello world…”

Having any html tag seems to break the smart quote conversion. The ellipsis conversion seems fine. The smart quotes seem to convert correctly when a markdown version of this is used (e.g. "Hello *world*..." "Hello world..."). Am I missing something or is this a bug?

[jgm]

As a workaround try converting to markdown without --smart, then converting the result with --smart.

[lisaah]

Hah, yep, that workaround will do for now. Thanks for looking into it!

[jgm]

The issue is that in the HTML reader we apply smartPunctuation only in parsing tag contents. So it only works, as it were, between tags. The reasons for this are a bit complex: the HTML reader parses a string of tokens produced by an HTML5 tokenizer. So we can't use our existing smart punctuation code, which operates on strings, on that -- but we can use it on the tag contents.

One could simply duplicate smart punctuation parsing logic using token parsers in the HTML reader, for a better solution. (Crossed out 'simply' because it's not that simple; I guess it would require splitting tag contents so that quotes were separately recognizable tokens.)