Metadata vs. other data

[bpj]

I wrote this some weeks ago and mulled it over. I have now revised it in the light of discussion in #146.

The problem

In Pandoc filter configuration and some other — mainly template — configuration is conflated with metadata, and that has its definite drawbacks. The most obvious is that once a filter has run you probably don’t want its configuration around in the output metadata; I find myself including code to remove it from output metadata, as well as adding further configuration for the times when I do not want to remove it.

Proposed solution

In Pandoc defaults files and on the command line you can separate template variables/directives from metadata by putting them in separate sections. I would extend that to parser, renderer and filter configuration and document variables, each in its own section/namespace.

Proposed syntax

I now think that the different namespaces should not be smushed together into one big table. In light of the discussion in #146, #36 I now think something like this would be best:

{% Made available to the parser as a dictionary %}
:::-parser
: rerum

  ea

: dolor

  beatae

: culpa

  nam
:::

{% Made available to the render, by format or renderer name, as a
dictionary/JSON %}
:::-renderer
: html

  : sunt

    nihil

  : dolor

    eos

  : qui

    soluta
    
: iste-ducimus.ext

  : rem

    in ducimus

  : soluta

    ipsa aliquid

  : sint

    hic neque
:::

{% Made available to renderer & filters %}
:::-meta
: optio

  Omnis nemo

: quae

  Vero veniam

: recusandae

  Non velit
:::

{% Made available to renderers/filters (see below!) %}
:::-vars
: distinctio

  Quisquam laborum

: quisquam

  Eum at

: qui

  Possimus quas
:::

{% Made available to parsers/renderers/filters (see below!) %}
:::-symb
: rerum

  Dolore molestiae

: odio

  Provident labore

: recusandae

  Possimus quas
:::

{% Tells which filers to run. May pass config/data to individual filters. %}
:::-filters
: aut-quas.ext

: numquam-a.ext

: deserunt-nulla.ext

  : molestiae

    debitis laborum

  : repellendus

    ducimus porro

  : nisi

    ratione quas
:::

Namespaces

As can be seen this proposed syntax uses

:::-LABEL
LIST
:::

blocks to separate namespaces. Assuming something similar will be adopted for roles, whose labels may or may not have a punctuation prefix, I propose that data labels have a distinct prefix to separate between the roles namspace and the data namespaces. I’m not attached to anything when it comes to which punctuation are used as prefix(es) and for each.

The advantage of having separate namespaces is hopefully obvious. There are advantages of having separate blocks for each namespace rather than something like

: parser

  LIST

: renderer

  LIST

: meta

  LIST

: vars

  LIST

: symb

  LIST

: filters

  LIST

• Separation of purpose in the mind of the user. In my experience as documentation writer this is a big deal. All of the following would work with a single data block type, but conceptual separation between namespaces enhances things.

• There may be multiple blocks for each namespace, perhaps in different files, in which values under top-level keys might be combined, while identical keys on seond level for the same namespace/top level key will override each other (I think on a simple “last in linear order in the input wins” principle.)

• Different namespaces may treat their data differently, e.g. values as raw strings or as AST fragments.

• Clear separation between namespaces and names within each namespace.

Comments on the respective proposed namespaces

parser and renderer

These would strictly be configuration/directives for the parser/renderer respectively.

In the case of renderer top level keys would be format names (rendered by the djot program) or, I imagine, names of external renderer programs — these might be separated as !external-renderer-name — with separate data for each. Maybe parser data could be similarly separated by implementation (assuming that implementations might have different f3atures!) Naturally there could be an all field.

meta

This would be strictly for output document metadata but renderers and filters should get a chance to inspect/alter it.

vars

Made available to parsers/renderers/filters data in this would be plain associative/list arrays with string/number leaves, which could be used for anything.

symb

These would differ from vars in that values would be parsed as AST fragments (inline only?).

Presumably the parser could insert optional var fields holding references to these values into symb elements with coinciding names, which filters/renderers could make use of if they want to.

filters

Here top level keys would be names of filters and optional values would be config/data for each filter.

Possibly top level keys could be names of output formats/renderers, with lists as per the above paragraph as values, with an all field which would come into play if there is no specific field for a particular format/renderer.