👌 IMPROVE: Directive option parsing (#796)

Previously, directive options blocks have been parsed using a "full" YAML parser.
This is unnecessary, and problematic since only a mapping of **string** key to **string** value is required,
not any of the other YAML constructs; sequences, flow style, anchors, aliases, tags, ...,
It required addition handling of non-string values (failing or converting back to strings),
and could also cause confusion when values were not parsed as strings.

This commit, introduces a new "restricted" YAML parser (adapted from pyyaml),
which will maintain back-compatibilty with existing options blocks, but will only parse mappings of string keys to string values (including multi-line strings and comment parsing).
Parsing errors are also improved, by reporting the exact line on which the error occured.

.gitignore

@@ -133,3 +133,5 @@ _archive/
 
 .vscode/
 .DS_Store
+
+docs/apidocs

.pre-commit-config.yaml

@@ -6,7 +6,8 @@ exclude: >
       \.vscode/settings\.json|
       tests/test_commonmark/commonmark\.json|
       .*\.xml|
-      tests/.*/.*\.md
+      tests/.*/.*\.md|
+      tests/.*/.*\.yaml
     )$
 
 repos:

docs/conf.py

@@ -85,6 +85,7 @@
 nitpick_ignore_regex = [
     (r"py:.*", r"docutils\..*"),
     (r"py:.*", r"pygments\..*"),
+    (r"py:.*", r"typing\.Literal\[.*"),
 ]
 nitpick_ignore = [
     ("py:obj", "myst_parser._docs._ConfigBase"),

docs/syntax/roles-and-directives.md

@@ -23,9 +23,8 @@ It is effectively a Markdown code fence with curly brackets around the language,
 Here is the basic structure:
 
 `````{list-table}
----
-header-rows: 1
----
+:header-rows: 1
+
 * - MyST
   - reStructuredText
 * - ````md
@@ -55,16 +54,13 @@ This is my note
 ```
 :::
 
-#### Parameterizing directives
+#### Parameterizing directives (options)
 
-For directives that take parameters as input, there are two ways to parameterize them.
-In each case, the options themselves are given as `key: value` pairs. An example of
-each is shown below:
+Many directives can take key/value pairs, in an optional *option block* at the start of the directive.
 
-**Short-hand options with `:` characters**. If you only need one or two options for your
-directive and wish to save lines, you may also specify directive options as a collection
-of lines just after the first line of the directive, each preceding with `:`. Then the
-leading `:` is removed from each line, and the rest is parsed as YAML.
+The option block starts on the first line of the directive body and is defined by a set of lines prefixed with `:`.
+
+The block then follows a YAML-like mapping syntax, where the key (string) and value (string) are separated by a colon (`:`):
 
 :::{myst-example}
 ```{code-block} python
@@ -77,10 +73,28 @@ print(f'my {a}nd line')
 ```
 :::
 
-**Using YAML frontmatter**. A block of YAML front-matter just after the
-first line of the directive will be parsed as options for the directive. This needs to be
-surrounded by `---` lines. Everything in between will be parsed by YAML and
-passed as keyword arguments to your directive. For example:
+Comments, starting `#`, are also allowed in between options or at the end of values, and are ignored.
+The values can be enclosed in quotes (`"` or `'`) and span multiple lines.
+Newline behaviour can be controlled by starting the value with `|` (preserve newlines) or `>` (collapse newlines):
+
+:::{myst-example}
+```{code-block} python
+:lineno-start: 10  # this is a comment
+: # this is also a comment
+:emphasize-lines: "1, 3"
+:caption: |
+:    This is my
+:    multi-line caption. It is *pretty nifty* ;-)
+
+a = 2
+print('my 1st line')
+print(f'my {a}nd line')
+```
+:::
+
+::::{dropdown} Old-style options block
+
+Option blocks can also be enclosed by `---`, with no `:` prefix, for example:
 
 :::{myst-example}
 ```{code-block} python
@@ -97,6 +111,8 @@ print(f'my {a}nd line')
 ```
 :::
 
+::::
+
 (syntax/directives/parsing)=
 
 #### How directives parse content
@@ -209,9 +225,8 @@ Roles are similar to directives - they allow you to define arbitrary new functio
 To define an in-line role, use the following form:
 
 ````{list-table}
----
-header-rows: 1
----
+:header-rows: 1
+
 * - MyST
   - reStructuredText
 * - ````md

myst_parser/mdit_to_docutils/base.py

@@ -1741,6 +1741,7 @@ def run_directive(
                 directive_class,
                 first_line,
                 content,
+                line=position,
                 additional_options=additional_options,
             )
         except MarkupError as error:
@@ -1750,12 +1751,11 @@ def run_directive(
             )
             return [error]
 
-        if parsed.warnings:
-            _errors = ",\n".join(parsed.warnings)
+        for warning_msg, warning_line in parsed.warnings:
             self.create_warning(
-                f"{name!r}: {_errors}",
+                f"{name!r}: {warning_msg}",
                 MystWarnings.DIRECTIVE_PARSING,
-                line=position,
+                line=warning_line if warning_line is not None else position,
                 append_to=self.current_node,
             )
 

myst_parser/mocking.py

@@ -140,7 +140,7 @@ def parse_directive_block(
         # TODO should argument_str always be ""?
         parsed = parse_directive_text(directive, "", "\n".join(content))
         if parsed.warnings:
-            raise MarkupError(",".join(parsed.warnings))
+            raise MarkupError(",".join(w for w, _ in parsed.warnings))
         return (
             parsed.arguments,
             parsed.options,