Restructure testing and directive block parsing (#59)

In this PR I have:

1. Restructured the test folder and added an `AstRenderer` and lower level tests for the source text to Markdown (mistletoe) AST.
2. Added `docs/develop/test_infrastructure.md` to explain the testing inrastructure.
3. Restructured the directive block parsing, to make it more modular/understandable and improved error checking and reporting.

fixes #54, fixes #55, fixes #51

.pre-commit-config.yaml

@@ -4,7 +4,7 @@
 exclude: >
     (?x)^(
       \.vscode/settings\.json|
-      tests/commonmark/commonmark\.json|
+      tests/test_commonmark/commonmark\.json|
       .*\.xml
     )$
 

.travis.yml

@@ -22,7 +22,7 @@ install:
         pip install coveralls
       fi
 before_script:
-    - (cd tests/commonmark && ./spec.sh)
+    - (cd tests/test_commonmark && ./spec.sh)
 script:
 - |
   if [[ "$TEST_TYPE" == "pytest" ]]; then

docs/develop/index.md

@@ -6,4 +6,5 @@ codebase, and some guidelines for how you can contribute.
 ```{toctree}
 contributing.md
 architecture.md
+test_infrastructure.md
 ```

docs/develop/test_infrastructure.md

@@ -0,0 +1,17 @@
+# Testing Infrastructure
+
+Where possible, additions to the code should be carried out in a
+[test-driven development](https://en.wikipedia.org/wiki/Test-driven_development)
+manner:
+
+> **Write failing tests that the code should pass, then write code to pass the tests**.
+
+The tests are run using [pytest](https://docs.pytest.org)/[TravisCI](https://travis-ci.org) for unit tests, and [sphinx-build](https://www.sphinx-doc.org/en/master/man/sphinx-build.html)/[CircleCI](https://circleci.com) for documentation build tests.
+
+The tests are ordered in a hierarchical fashion:
+
+1. In `tests/test_syntax` are tests that check that the source text is being correctly converted to the Markdown ([mistletoe](https://github.com/miyuchina/mistletoe)) AST.
+2. In `tests/test_commonmark` the [CommonMark](https://github.com/commonmark/CommonMark.git) test set is run; to check that the parser is complying with the CommonMark specification.
+3. In `tests/test_renderers` are tests that check that the Markdown AST is being correctly converted to the docutils/sphinx AST. This includes testing that roles and directives are correctly parsed and run.
+4. In `tests/test_sphinx` are tests that check that minimal sphinx project builds are running correctly, to convert myst markdown files to HTML.
+5. In `.circleci` the package documentation (written in myst format) is built and tested for build errors/warnings.

docs/using/syntax.md

@@ -439,7 +439,7 @@ Is below, but it won't be parsed into the document.
 
 (syntax/targets)=
 
-### Targets
+### Targets and Cross-Referencing
 
 Targets are used to define custom anchors that you can refer to elsewhere in your
 documentation. They generally go before section titles so that you can easily refer
@@ -451,11 +451,32 @@ Target headers are defined with this syntax:
 (header_target)=
 ```
 
-They can then be referred to with the "ref" inline role:
+They can then be referred to with the [ref inline role](https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-ref):
 
 ```
 {ref}`header_target`
 ```
 
-For example, see this ref: {ref}`syntax/targets` and here's a ref back to the top of
-this page {ref}`example_syntax`.
+By default, the reference will use the text of the target (such as the section title), but also you can directly specify the text:
+
+```
+{ref}`my text <header_target>`
+```
+
+For example, see this ref: {ref}`syntax/targets`, and here's a ref back to the top of
+this page: {ref}`my text <example_syntax>`.
+
+Alternatively using the markdown syntax:
+
+```markdown
+[my text](header_target)
+```
+
+is synonymous with using the [any inline role](https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-any):
+
+```
+{any}`my text <header_target>`
+```
+
+Using the same example, see this ref: [](syntax/targets), and here's a ref back to the top of
+this page: [my text](example_syntax).

myst_parser/ast_renderer.py

@@ -0,0 +1,35 @@
+"""Abstract syntax tree renderer for myst."""
+from itertools import chain
+import json
+
+from mistletoe import block_token, span_token
+from mistletoe import ast_renderer
+
+from myst_parser import span_tokens as myst_span_tokens
+from myst_parser import block_tokens as myst_block_tokens
+
+
+class AstRenderer(ast_renderer.ASTRenderer):
+    def __init__(self):
+        """This AST render uses the same block/span tokens as the docutils renderer."""
+
+        _span_tokens = self._tokens_from_module(myst_span_tokens)
+        _block_tokens = self._tokens_from_module(myst_block_tokens)
+
+        super(ast_renderer.ASTRenderer, self).__init__(
+            *chain(_block_tokens, _span_tokens)
+        )
+
+        span_token._token_types.value = _span_tokens
+        block_token._token_types.value = _block_tokens
+
+    def render(self, token, to_json=False):
+        """
+        Returns the string representation of the AST.
+
+        Overrides super().render. Delegates the logic to get_ast.
+        """
+        ast = ast_renderer.get_ast(token)
+        if to_json:
+            return json.dumps(ast, indent=2) + "\n"
+        return ast

myst_parser/block_tokens.py

@@ -1,7 +1,5 @@
 import re
 
-import yaml
-
 from mistletoe import block_token, span_token
 import mistletoe.block_tokenizer as tokenizer
 
@@ -39,6 +37,19 @@
 class FrontMatter(block_token.BlockToken):
     """Front matter YAML block.
 
+    ::
+
+        ---
+        a: b
+        c: d
+        ---
+
+    NOTE: The content of the block should be valid YAML,
+    but its parsing (and hence syntax testing) is deferred to the renderers.
+    This is so that, given 'bad' YAML,
+    the rest of the of document will still be parsed,
+    and then the renderers can apply there own error reporting.
+
     Not included in the parsing process, but called by Document.__init__.
     """
 
@@ -53,8 +64,7 @@ def __init__(self, lines):
         if end_line is None:
             end_line = len(lines)
         self.range = (0, end_line)
-        yaml_block = "\n".join(lines[1 : end_line - 1])
-        self.data = yaml.safe_load(yaml_block) or {}
+        self.content = "\n".join(lines[1 : end_line - 1])
         self.children = []
 
     @classmethod
@@ -319,12 +329,12 @@ class CodeFence(block_token.CodeFence):
         language (str): language of code block (default to empty).
     """
 
-    pattern = re.compile(r"( {0,3})((?:`|~){3,}) *([^`~\s]*) *([^`~]*)")
+    pattern = re.compile(r"^( {0,3})((?:`|~){3,}) *([^`~\s]*) *([^`~]*)$")
 
     def __init__(self, match):
         lines, open_info, self.range = match
         self.language = span_token.EscapeSequence.strip(open_info[2])
-        self.arguments = span_token.EscapeSequence.strip(open_info[3])
+        self.arguments = span_token.EscapeSequence.strip(open_info[3].splitlines()[0])
         self.children = (span_token.RawText("".join(lines)),)
 
     @classmethod