merge yaml no longer working in version 2.7.3+; issues working with latex templates and yaml blocks

[pedro-nonfree]

I am using a particular use case since debian 9 (pandoc 1.17.2?): Latex templates with variables defined in a yaml file.

pandoc varfile.yml -o outfile.pdf --template=templatefile.tex --pdf-engine=xelatex

Specifically for a project based on invoice-boilerplate. In 2.7.3 (installed using .deb available here) the yaml file is ignored. In the internet there are different sites pointing to use pandoc in this direction and I'm sure more people is experiencing the same and most probably this is duplicated (sorry):

• https://translate.google.com/translate?sl=auto&tl=en&u=https%3A%2F%2Fondiz.github.io%2FcursoLatex%2FContenido%2F15.Pandoc.html
• https://www.managementletters.eu/posts/publishing/

so while this is not solved, I am reverting to pandoc 2.2.1 included in debian 10

[jgm]

I don't see any obvious problems with the command line. But we can't help diagnose your problem unless you give enough information to fully reproduce it. That means we need to see varfile.yml and template.tex. If these contain information you don't want to share, you can create a minimal example that illustrates the problem.

[pedro-nonfree]

@jgm thanks! giving more info to facilitate its reproducibility

pandoc varfile.yml -o outfile.pdf --template=templatefile.tex --pdf-engine=xelatex

pandoc 1.17.2 and 2.2.1 works

pandoc 2.7.3 does not work (it fails to show the imported text variable)

varfile.yml

---
# if this comment line misses fails silently, why?

variables: &variables
  importedtext: my imported text

fontsize: 10pt
geometry: a4paper, left=43mm, right=43mm, top=51mm, bottom=17mm

<<: *variables

mytext: my text

---

templatefile.tex

\documentclass[$fontsize$, a4paper]{article}
\usepackage{geometry}
\geometry{$geometry$}

\begin{document}

\normalsize \sffamily

imported variables are not shown in 2.7.3

importedtext: $importedtext$

mytext: $mytext$

\end{document}

[jgm]

As for # if this comment line misses, pandoc uses a heuristic to distinguish YAML blocks from occurrences of --- that are horizontal rules. The content must start on the line after ---.

As for the missing element, apparently HsYAML (the library we use to parse YAML in recent versions) is not handling the <<: *variables the way the old yaml library worked.

Can you point to documentation in the YAML spec for this syntactic form: <<:? I wasn't able to find it.

[jgm]

Data.YAML> decodeNode "variables: &variables\n  importedtext: my text\n\n<<: *variables\n"
Right [Doc {getDoc = Mapping (Pos {posByteOffset = 0, posCharOffset = 0, posLine = 1, posColumn = 0}) Just "tag:yaml.org,2002:map" (fromList [(Scalar (Pos {posByteOffset = 47, posCharOffset = 47, posLine = 4, posColumn = 0}) (SStr "<<"),Mapping (Pos {posByteOffset = 22, posCharOffset = 22, posLine = 2, posColumn = 0}) Just "tag:yaml.org,2002:map" (fromList [(Scalar (Pos {posByteOffset = 24, posCharOffset = 24, posLine = 2, posColumn = 2}) (SStr "importedtext"),Scalar (Pos {posByteOffset = 38, posCharOffset = 38, posLine = 2, posColumn = 16}) (SStr "my text"))])),(Scalar (Pos {posByteOffset = 0, posCharOffset = 0, posLine = 1, posColumn = 0}) (SStr "variables"),Mapping (Pos {posByteOffset = 22, posCharOffset = 22, posLine = 2, posColumn = 0}) Just "tag:yaml.org,2002:map" (fromList [(Scalar (Pos {posByteOffset = 24, posCharOffset = 24, posLine = 2, posColumn = 2}) (SStr "importedtext"),Scalar (Pos {posByteOffset = 38, posCharOffset = 38, posLine = 2, posColumn = 16}) (SStr "my text"))]))])}]

[jgm]

Seems like HsYAML is parsing the <<: as a regular key in a mapping:

Just "tag:yaml.org,2002:map" (fromList [(Scalar (Pos {posByteOffset = 47, posCharOffset = 47, posLine = 4, posColumn = 0}) (SStr "<<")

[pedro-nonfree]

thanks @jgm

this is alias, anchor and merge concept. https://stackoverflow.com/questions/41063361/what-is-the-double-left-arrow-syntax-in-yaml-called-and-wheres-it-specced/41065222 -> https://yaml.org/type/merge.html

looks like is a draft for yaml 1.1 that was not included in the specification but lots of libraries included it: http://blogs.perl.org/users/tinita/2019/05/reusing-data-with-yaml-anchors-aliases-and-merge-keys.html

do you know an alternative way to achieve mostly the same with current pandoc's library? in my example, I had only one variable, but the real case looks like this:

https://gitlab.com/pedrolab/invoice-boilerplate/blob/c74eaa1ffd8aeec05f64f65c285808016c29b0cf/header.yml#L29-38

https://gitlab.com/pedrolab/invoice-boilerplate/blob/c74eaa1ffd8aeec05f64f65c285808016c29b0cf/example.yml#L41

[pedro-nonfree]

thanks to this conversation with @jgm I discovered that the merge feature is not part of the specification, so I used what I found here https://yaml.org/spec/1.2/spec.html#id2761803

varfile.yml

changes

-<<: *variables
+env: *variables

new file

---
# mandatory text here to let pandoc detect we are in a yaml block

variables: &variables
  importedtext: my imported text

fontsize: 10pt
geometry: a4paper, left=43mm, right=43mm, top=51mm, bottom=17mm

env: *variables

mytext: my text

---

templatefile.tex

changes

-importedtext: $importedtext$
+importedtext: $env.importedtext$

new file

\documentclass[$fontsize$, a4paper]{article}
\usepackage{geometry}
\geometry{$geometry$}

\begin{document}

\normalsize \sffamily

imported variables are not shown in 2.7.3

importedtext: $env.importedtext$

mytext: $mytext$

\end{document}

[jgm]

Okay, that helps. I suppose we could manually implement the merge syntax in pandoc, but I am reluctant to depart from official YAML spec. Glad you found a way to work around it.