Open HTML tag parsing depends on tag contents in strange way

[link2xt]

I just want to make Muse reader interpret <code>&foo></code> as code block with literal &foo>, but markdown is also affected:

$ pandoc -f markdown -t native
<code>foo</code>
[Para [RawInline (Format "html") "<code>",Str "foo",RawInline (Format "html") "</code>"]]
$ pandoc -f markdown -t native
<code>&foo</code>
[Para [RawInline (Format "html") "<code>&foo</code>"]]
$ pandoc -f markdown -t native
<code>&foo<</code>
[Para [RawInline (Format "html") "<code>&foo<</code>"]]
$ pandoc -f markdown -t native
<code>&foo></code>
[Para [RawInline (Format "html") "<code>&foo>",RawInline (Format "html") "</code>"]]
$ pandoc -f muse -t native
<code>&foo></code>
[Para [Code ("",[],[]) ""]]
$ pandoc -f muse -t native
<code>foo&foo></code>
[Para [Code ("",[],[]) "foo&foo>"]]

Look like from reading <code>&foo></code> instead of "&foo>" I got nothing inside code tag. I don't want to parse any HTML/XML entities inside tags, I just want to get a string of anyChars and parse them as inlines:

pandoc/src/Text/Pandoc/Readers/Muse.hs

Line 107 in 5ba890a

content <- manyTill anyChar endtag

Looks like my code has nothing treating & in a special way, so it is probably in htmlTag.

[jgm]

You could investigate this by using htmlTag in ghci to try to figure out why it's not working.

[mb21]

Is this the same issue as #4088 ?

[jgm]

> runParser (htmlTag (const True)) defaultParserState "hi" "<code>&foo>"
Right (TagOpen "code" [],"<code>&foo>")

and for #4088

> runParser (htmlTag (const True)) defaultParserState "hi" "<input type=\"checkbox\" disabled=\"\" checked=\"\">&thinsp;I'm a _checked_ task</li>"
Right (TagOpen "input" [("type","checkbox"),("disabled",""),("checked","")],"<input type=\"checkbox\" disabled=\"\" checked=\"\">&thinsp;I'm a _checked_ task</li>")

Definitely an issue with htmlTag here!

[jgm]

Here's the tokenization for <code>&foo>:

[TagPosition 1 1,TagOpen "code" [],TagPosition 1 8,TagText "&foo> "]

[jgm]

I think this is a bug in tag-soup:

Prelude Text.HTML.TagSoup> parseTagsOptions  Text.HTML.TagSoup.parseOptions{ optTagWarning = False, optTagPosition = True} "<a>hi"
[TagPosition 1 1,TagOpen "a" [],TagPosition 1 4,TagText "hi"]
Prelude Text.HTML.TagSoup> parseTagsOptions  Text.HTML.TagSoup.parseOptions{ optTagWarning = False, optTagPosition = True} "<a>&hi>"
[TagPosition 1 1,TagOpen "a" [],TagPosition 1 5,TagText "&hi>"]

I don't see why the TagPosition after the <a> tag should depend on whether it is followed by a &.

[jgm]

Even clearer:

Prelude Text.HTML.TagSoup> parseTagsOptions  Text.HTML.TagSoup.parseOptions{ optTagWarning = False, optTagPosition = True} "<a>&</a>"
[TagPosition 1 1,TagOpen "a" [],TagPosition 1 5,TagText "&",TagPosition 1 5,TagClose "a"]
Prelude Text.HTML.TagSoup> parseTagsOptions  Text.HTML.TagSoup.parseOptions{ optTagWarning = False, optTagPosition = True} "<a>x</a>"
[TagPosition 1 1,TagOpen "a" [],TagPosition 1 4,TagText "x",TagPosition 1 5,TagClose "a"]

Note that in the first case, the TagPosition before and after the & are the same!

[jgm]

This should be fixed by the new tagsoup version, once it is released.