Dealing with ambiguous YAML frontmatter anywhere

[trieloff]

In Helix Pipeline we are processing documents that have YAML blocks in the middle of the document, as supported by the anywhere option in https://github.com/remarkjs/remark-frontmatter

Unfortunately, this has lead us to experience a couple of scenarios where we had to resolve ambiguities between YAML fences, Setext headlines, and thematic breaks (see tests)with a number of custom constraints such as:

• forbid empty lines in midmatter, but allow it in frontmatter
• expect empty lines around thematic breaks
• only accept YAML blocks that contain actual parseable YAML (only objects)

this has lead us down the path of building a custom remark plugin

Now with the switch to micromark, @tripodsan has started to re-implement our ambiguity-resolving constraints based on micromark-extension-frontmatter and co, but we wondered if it would be better to contribute it back to the official extension instead of keeping our separate fork.

As it stands, the anywhere: true and marker: '-' settings are practically incompatible, and implementing our heuristics within micromark-extension-frontmatter would either require additional configuration settings or callbacks or some other approach, but we'd like to get some feedback before we submit a big PR that may not be along the lines of your vision for the project.

[wooorm]

Have you checked w/ new remark (micromark)?

I just checked YAML anywhere vs. setext, and this behavior makes sense to me and is I believe what you want: micromark/micromark-extension-frontmatter@5e86096

What is an example of thematic breaks interfering in a weird way?

[tripodsan]

hi @wooorm, I think we want the other way around.

here some examples that should not be treated as front- / midmatter:

1. consecutive headers

Hello
---

Bar
---

2. reject any matter that has leading empty line

# Multimedia Test
---
    
![](https://hlx.blob.core.windows.net/external/20f9d6dff67514da262230822bda5f3b50ef28c6#image.png)
    
---
    
PUBLISHED ON 28-04-2020
    
---
    
### SlideShare

<https://www.slideshare.net/adobe/adobe-digital-insights-holiday-recap-2019>

3. reject invalid yaml

---
- Foo
hello: 42
---

[wooorm]

Thanks for these examples, that helped!

Can we take a step back? As you know, dashes are used a lot, so adding more constructs with dashes adds even more complexity.

Can you use different markers? E.g., YAML itself has a dashes and dots structure:

Some text

---
data
...

More markdown.

Or, how about plusses for fences?

---

A different idea: Node uses comments to add YAML data to sections: https://github.com/nodejs/node/blame/master/doc/api/fs.md#L179. The benefit of something like this is that all markdown parsers know how to deal with comments. But midmatter isn’t supported anywhere, leading to non-portable content.

[trieloff]

We probably just have to live with the complexity as we already have a couple thousand documents with --- delimited matter. The dashes and dots approach looks very appealing, though and we might want to have it as an additional option.

Comments are not feasible for our authors, between <!-- and --> there are too many things you can get wrong.

[wooorm]

We probably just have to live with the complexity as we already have a couple thousand documents with --- delimited matter. The dashes and dots approach looks very appealing, though and we might want to have it as an additional option.

The AST is mostly the same: you could parse with the old plugins, and compile with the new plugins, to “codetransform” the old to the new? Is that feasible?

Comments are not feasible for our authors, between <!-- and --> there are too many things you can get wrong.

I can somewhat see that but also, I find it a bit strong compared to the number of limitation you have in place now that folks need to remember?

---

In my experience authors (and developers such as me 😅) have a hard time with YAML. So if you’re going to add it in a lot of places, you probably want to validate it (either by checking if it is parseable, or also schema validation). The edge cases you have now, seem to instead fail silently?

More, the edge cases say to me that YAML is seen too often, as thematic breaks or setext headings are intended. Both those constructs have alternatives: ATX headings, or other markers. Can you suggest those?

I’d suggest to pick something that isn’t dashes only for frontmatter, and regardless of whether that’s possible, to check the yaml and report it to editors when it’s incorrect. This will alert them to the fact that intended YAML is incorrect, which often happens, but also for cases where they tried to make setext headings or thematic breaks, but it failed.
I would suggest to have an editorial rule to use ATX headings and different markers for thematic breaks, which could also be in the message?

[tripodsan]

I tried to implement a check in after: https://github.com/micromark/micromark-extension-frontmatter/blob/5e86096266010978411aa3427d988f2022f7ede4/lib/syntax.js#L92

something like:

    function after(code) {
      const token = effects.exit(name);
      const yamlString = self.sliceSerialize(token);
      if (!validYaml(yamlString)) {
        return nok(code);
      }
      return ok(code);
    }

but it messes up the parser. a simple example:

---
- Foo
hello: 42
---

produces without the extension:

root[3]
├─0 thematicBreak
├─1 list[1]
│   │ ordered: false
│   │ start: null
│   │ spread: false
│   └─0 listItem[1]
│       │ spread: false
│       │ checked: null
│       └─0 paragraph[1]
│           └─0 text "Foo\nhello: 42"
└─2 thematicBreak

but with the extension:

root[2]
├─0 thematicBreak
└─1 heading[1]
    │ depth: 2
    └─0 text "- Foo\nhello: 42"

It looks like the state is not reset properly, if the yaml construct fails.

any ideas?

[wooorm]

This is interesting!

In markdown, you have containers, such as listitems and blockquotes, that are parsed differently from the rest of flow, and have an weird thing:

- ~~~code
  console.log()

we’re no longer in code here, even though it wasn’t closed

The reason code is closed, is because it is what I call “concrete”. It a) can’t be interrupted by more containers, and b) runs to the end if there is no closing. HTML (flow), aka block HTML, is like that too. But paragraphs, definitions, or setext headings are not.

As frontmatter can’t be interrupted by containers:

---
- this isn’t a list
---

However, as we see here, it does expect an ending, which isn’t found in your example.

Here is an example without your changes, that I believe thus also fails in micromark-extension-frontmatter:

---
- Foo

But there is the limitation imposed by micromark: the containers are already done. That phase has passed, the data is lost, we can’t find lists/blockquotes anymore

[tripodsan]

However, as we see here, it does expect an ending, which isn’t found in your example.

yes it has.

---

thanks for the explanation...but I still don't understand. looking at the syntax, the simplified list of effects of a frontmatter block would be:

- fenceConstruct
- afterOpeningFence
- lineEnd
- *
- fenceConstruct
- after

when I add a return nok() in lineEnd, it resets the state correctly, just not in after.
it looks like returning nok() in a returnState function doesn't work. eg:

      return effects.attempt(fenceConstruct, after, lineStart);

[wooorm]

Calling nok in lineEnd signals that something isn’t frontmatter on the first line already. Calling nok in the third line is after the potential list. That second line is already dealt with. It can’t change anymore.

A similar example is saying that something isn’t a link (a span, “text” constructs): that doesn’t affect whether the block is a paragraph (a block, flow constructs). It’s already dealt with.