Enabling mmd_title_block causes YAML title blocks to be mis-parsed

[zackw]

Consider

---
booktitle_url: http://www.sigsac.org/ccs/CCS2013/
...

This is unambiguously a YAML header, and the manual says YAML headers take precedence over MMD headers. But watch what happens:

$ pandoc -f markdown -t json < test.md
[{"unMeta":{"booktitle_url":{"t":"MetaInlines","c":[{"t":"Str","c":"http://www.sigsac.org/ccs/CCS2013/"}]}}},[]]

$ pandoc -f markdown+mmd_title_block -t json < test.md
[{"unMeta":{"booktitle_url":{"t":"MetaBlocks","c":[]}}},[]]

Not all values are affected. URLs consistently seem to get eaten, and strings containing no punctuation consistently seem to survive, but don't quote me on that.

I have pandoc 1.12.4.2, in case that matters.

[jgm]

I can reproduce this with latest pandoc:

% pandoc -f markdown+mmd_title_block -s -t native
---
booktitle_url: http://www.sigsac.org/ccs/CCS2013/
...
^D
Pandoc (Meta {unMeta = fromList [("booktitle_url",MetaBlocks [])]})
[]

No idea why off hand, but will have to look into it.

[lierdakil]

Well, I think I know why that happens.

echo -e "http://google.com" | pandoc -f markdown+mmd_title_block -t json

produces

[{"unMeta":{"http":{"t":"MetaBlocks","c":[{"t":"Plain","c":[{"t":"Str","c":"//google.com"}]}]}}},[]]

i.e. "http" is interpreted as a metadata key, and the rest as metadata value.

Since metadata values are parsed with the same parser... yeah. No idea on how to fix this properly though. How does multimarkdown handle this, I wonder?

[zackw]

Possibly related, then:

$ printf '%s\n%s\n%s\n' '---' 'title: This Title Has: A Colon In It' '...' | pandoc
pandoc: Could not parse YAML header: mapping values are not allowed in this context "source" (line 3, column 22)

If I'm reading the YAML spec correctly, This Title Has: A Colon In It should be interpreted as a string literal. But I'm not 100% sure about that.

[lierdakil]

@zackw, not sure about YAML spec, but it's parsed by yaml, so that's not directly related to pandoc itself.

Multimarkdown title block is parsed by pandoc, however.

Fix options would include:

1. Require space after semicolon in mmd title block
2. Disable all metadata block extensions when parsing metadata fields

In all honesty, I think both should be implemented...

[jgm]

+++ Nikolay Yakimov [Mar 27 15 16:18 ]:

Well, I think I know why that happens.

echo -e "http://google.com" | pandoc -f markdown+mmd_title_block -t json

produces

[{"unMeta":{"http":{"t":"MetaBlocks","c":[{"t":"Plain","c":[{"t":"Str","c":"//google.com"}]}]}}},[]]

i.e. "http" is interpreted as a metadata key, and the rest as metadata value.

Since metadata values are parsed with the same parser... yeah. No idea on how to fix this properly though. How does multimarkdown handle this, I wonder?

Multimarkdown does not parse metadata values at all - so you can't e.g. have italics in a metadata value. (Not the greatest feature.)

[jgm]

Yes, good point: try putting the whole URL in single quotes. Colons in YAML values normally need to be escaped.

+++ Zack Weinberg [Mar 27 15 16:24 ]:

Possibly related, then:

$ printf '%s\n%s\n%s\n' '---' 'title: This Title Has: A Colon In It' '...' | pandoc
pandoc: Could not parse YAML header: mapping values are not allowed in this context "source" (line 3, column 22)

If I'm reading the YAML spec correctly, This Title Has: A Colon In It should be interpreted as a string literal. But I'm not 100% sure about that.

---

Reply to this email directly or view it on GitHub:
#2026 (comment)

[lierdakil]

@jgm, I mean, how does mmd handle markdown that begins with URI? Does it interpret it as title block?

Anyway, I've pushed commits for my fix proposals. You can cherry-pick one or both, or I can create a PR.

[lierdakil]

Oh, and by the way, single/double quotes won't work, since those are stripped away by yaml parser.

[jgm]

+++ Nikolay Yakimov [Mar 27 15 17:01 ]:

@jgm, I mean, how does mmd handle markdown that begins with URI? Does it interpret it as title block?

It definitely requires a space after the colon.

[jgm]

+++ Nikolay Yakimov [Mar 27 15 17:01 ]:

@jgm, I mean, how does mmd handle markdown that begins with URI? Does it interpret it as title block?

Anyway, I've pushed commits for my fix proposals. You can cherry-pick one or both, or I can create a PR.

I think both fixes are needed. IF you can create a PR, that would be helpful.

(Btw, I've confirmed experimentally that multimarkdown does not allow a
metadata field with no value, or a blank value. So technically we
should both (a) skip one or more spaces, and (b) use many1 instead of
many for the value.)

[lierdakil]

Ok, I've created #2030. I'll see about requiring a value in a minute.

[lierdakil]

Skipping spaces is more convoluted than I initially thought, when considering empty value, since Parsec's space matches newline as well, but I think I managed to make it work in #2030. Please review.

[jgm]

@lierdakil, I probably would have just used Text.Pandoc.Parsing.spaceChar which is just space or tab. But I did check, and multimarkdown allowed a unicode nonbreaking space. So your solution seems more correct.

Closed by 86a4442