Update Documentation

README.md

@@ -5,12 +5,21 @@
 [![Documentation Status][rtd-badge]][rtd-link]
 [![Code style: black][black-badge]][black-link]
 [![PyPI][pypi-badge]][pypi-link]
-<!-- [![Conda][conda-badge]][conda-link] -->
+[![Conda][conda-badge]][conda-link]
 
 An extended commonmark compliant parser, with bridges to docutils & sphinx.
 
 ## Usage
 
+To install the MyST parser, run the following in a
+[Conda environment](https://docs.conda.io) (recommended):
+
+```bash
+conda install -c conda-forge myst-parser
+```
+
+or
+
 ```bash
 pip install myst-parser[sphinx]
 ```
@@ -26,86 +35,6 @@ pip install -e .[sphinx,code_style,testing,rtd]
 
 To use the MyST parser in Sphinx, simply add: `extensions = ["myst_parser"]` to your `conf.py`.
 
-## Parsed Token Classes
-
-For more information, also see the [CommonMark Spec](https://spec.commonmark.org/0.28/).
-
-### Block Tokens
-
-- **FrontMatter**: A YAML block at the start of the document enclosed by `---`
-- **HTMLBlock**: Any valid HTML (rendered in HTML output only)
-- **LineComment**: `% this is a comment`
-- **BlockCode**: indented text (4 spaces)
-- **Heading**: `# Heading` (levels 1-6)
-- **SetextHeading**: underlined header
-- **Quote**: `> this is a quote`
-- **CodeFence**: enclosed in 3 or more backticks.
-  If it starts with a `{name}`, then treated as directive.
-- **ThematicBreak**: `---`
-- **List**: bullet points or enumerated.
-- **Table**: Standard markdown table styles.
-- **Footnote**: A substitution for an inline link (e.g. `[key][name]`), which can have a reference target (no spaces), and an optional title (in `"`), e.g. `[key]: https://www.google.com "a title"`
-- **Paragraph**: General inline text
-
-### Span (Inline) Tokens
-
-- **Role**: `` `{name}`interpreted text` ``
-- **Math**: `$a=1$` or `$$a=1$$`
-- **HTMLSpan**: any valid HTML (rendered in HTML output only)
-- **EscapeSequence**: `\*`
-- **AutoLink**: `<http://www.google.com>`
-- **Target**: `(target)=` (precedes element to target, e.g. header)
-- **InlineCode**: `` `a=1` ``
-- **LineBreak**: Soft or hard (ends with spaces or `\`)
-- **Image**: `![alt](src "title")`
-- **Link**: `[text](target "title")` or `[text][key]` (key from `Footnote`)
-- **Strong**: `**strong**`
-- **Emphasis**: `*emphasis*`
-- **RawText**
-
-## Contributing
-
-### Code Style
-
-Code style is tested using [flake8](http://flake8.pycqa.org),
-with the configuration set in `.flake8`,
-and code formatted with [black](https://github.com/ambv/black).
-
-Installing with `myst-parser[code_style]` makes the [pre-commit](https://pre-commit.com/)
-package available, which will ensure this style is met before commits are submitted, by reformatting the code
-and testing for lint errors.
-It can be setup by:
-
-```shell
->> cd MyST-Parser
->> pre-commit install
-```
-
-Optionally you can run `black` and `flake8` separately:
-
-```shell
->> black .
->> flake8 .
-```
-
-Editors like VS Code also have automatic code reformat utilities, which can adhere to this standard.
-
-### Pull Requests
-
-To contribute, make Pull Requests to the `master` branch (this is the default branch). A PR can consist of one or multiple commits. Before you open a PR, make sure to clean up your commit history and create the commits that you think best divide up the total work as outlined above (use `git rebase` and `git commit --amend`). Ensure all commit messages clearly summarise the changes in the header and the problem that this commit is solving in the body.
-
-Merging pull requests: There are three ways of 'merging' pull requests on GitHub:
-
-- Squash and merge: take all commits, squash them into a single one and put it on top of the base branch.
-    Choose this for pull requests that address a single issue and are well represented by a single commit.
-    Make sure to clean the commit message (title & body)
-- Rebase and merge: take all commits and 'recreate' them on top of the base branch. All commits will be recreated with new hashes.
-    Choose this for pull requests that require more than a single commit.
-    Examples: PRs that contain multiple commits with individually significant changes; PRs that have commits from different authors (squashing commits would remove attribution)
-- Merge with merge commit: put all commits as they are on the base branch, with a merge commit on top
-    Choose for collaborative PRs with many commits. Here, the merge commit provides actual benefits.
-
-
 [travis-badge]: https://travis-ci.org/ExecutableBookProject/MyST-Parser.svg?branch=master
 [travis-link]: https://travis-ci.org/ExecutableBookProject/MyST-Parser
 [coveralls-badge]: https://coveralls.io/repos/github/ExecutableBookProject/MyST-Parser/badge.svg?branch=master

docs/develop/contributing.md

@@ -1,5 +1,11 @@
 # Contributing
 
+[![CI Status][travis-badge]][travis-link]
+[![Coverage][coveralls-badge]][coveralls-link]
+[![CircleCI][circleci-badge]][circleci-link]
+[![Documentation Status][rtd-badge]][rtd-link]
+[![Code style: black][black-badge]][black-link]
+
 ## Code Style
 
 Code style is tested using [flake8](http://flake8.pycqa.org),
@@ -63,3 +69,14 @@ Merging pull requests: There are three ways of 'merging' pull requests on GitHub
     Examples: PRs that contain multiple commits with individually significant changes; PRs that have commits from different authors (squashing commits would remove attribution)
 - Merge with merge commit: put all commits as they are on the base branch, with a merge commit on top
     Choose for collaborative PRs with many commits. Here, the merge commit provides actual benefits.
+
+[travis-badge]: https://travis-ci.org/ExecutableBookProject/MyST-Parser.svg?branch=master
+[travis-link]: https://travis-ci.org/ExecutableBookProject/MyST-Parser
+[coveralls-badge]: https://coveralls.io/repos/github/ExecutableBookProject/MyST-Parser/badge.svg?branch=master
+[coveralls-link]: https://coveralls.io/github/ExecutableBookProject/MyST-Parser?branch=master
+[circleci-badge]: https://circleci.com/gh/ExecutableBookProject/MyST-Parser.svg?style=shield
+[circleci-link]: https://circleci.com/gh/ExecutableBookProject/MyST-Parser
+[rtd-badge]: https://readthedocs.org/projects/myst-parser/badge/?version=latest
+[rtd-link]: https://myst-parser.readthedocs.io/en/latest/?badge=latest
+[black-badge]: https://img.shields.io/badge/code%20style-black-000000.svg
+[black-link]: https://github.com/ambv/black

docs/using/install.md

@@ -1,11 +1,21 @@
 # Installing the MyST Parser
 
+[![PyPI][pypi-badge]][pypi-link]
+[![Conda][conda-badge]][conda-link]
+
 Installing the MyST parser provides access to two tools:
 
 * A MyST-to-docutils parser and renderer.
 * A Sphinx parser that utilizes the above tool in building your documenation.
 
-To install the MyST parser, run the following:
+To install the MyST parser, run the following in a
+[Conda environment](https://docs.conda.io) (recommended):
+
+```bash
+conda install -c conda-forge myst-parser
+```
+
+or
 
 ```bash
 pip install myst-parser[sphinx]
@@ -19,3 +29,8 @@ cd MyST-Parser
 git checkout master
 pip install -e .[sphinx,code_style,testing,rtd]
 ```
+
+[pypi-badge]: https://img.shields.io/pypi/v/myst-parser.svg
+[pypi-link]: https://pypi.org/project/myst-parser
+[conda-badge]: https://anaconda.org/conda-forge/myst-parser/badges/version.svg
+[conda-link]: https://anaconda.org/conda-forge/myst-parser

docs/using/syntax.md

@@ -229,7 +229,7 @@ For more information, also see the [CommonMark Spec](https://spec.commonmark.org
     `a=1`
     ```
 * - LineBreak
-  - Soft or hard (ends with spaces or \)
+  - Soft or hard (ends with spaces or backslash)
   - ```md
     A hard break\
     ```
@@ -709,7 +709,8 @@ this page: [my text](example_syntax).
 
 ### Footnotes
 
-Footnote labels can be any alpha-numeric string (no spaces), and are case-insensitive.
+Footnote labels **start with `^`** and can then be any alpha-numeric string (no spaces),
+which is case-insensitive.
 The actual label is not displayed in the rendered text; instead they are numbered,
 in the order which they are referenced.
 All footnote definitions are collected, and displayed at the bottom of the page
@@ -719,12 +720,12 @@ Note that un-referenced footnote definitions will not be displayed.
 ```md
 This is a footnote reference.[^myref]
 
-[^myref]: This is the footnote definition.
+[^myref]: This **is** the footnote definition.
 ```
 
 This is a footnote reference.[^myref]
 
-[^myref]: This is the footnote definition.
+[^myref]: This **is** the footnote definition.
 
 ````{important}
 Although footnote references can be used just fine within directives, e.g.[^myref],

docs/using/use_api.md

@@ -5,8 +5,7 @@
 MyST-Parser may be used as an API *via* the `myst_parser` package.
 
 ```{seealso}
-% - {ref}`Programmatic Use of Mistletoe <mistletoe:intro/api_use>`
-- {ref}`Programmatic Use of Mistletoe <mistletoe:intro/usage>`
+- {ref}`Programmatic Use of Mistletoe <mistletoe:intro/api_use>`
 - {ref}`The MyST-Parser API <api/main>`
 ```
 

myst_parser/__init__.py

@@ -1,4 +1,4 @@
-__version__ = "0.5.1"
+__version__ = "0.6.0a2"
 
 
 def text_to_tokens(text: str):

myst_parser/docutils_renderer.py

@@ -20,6 +20,7 @@
 import yaml
 
 from mistletoe import block_tokens, block_tokens_ext, span_tokens, span_tokens_ext
+from mistletoe.base_elements import SourceLines
 from mistletoe.renderers.base import BaseRenderer
 from mistletoe.parse_context import get_parse_context, ParseContext
 
@@ -113,8 +114,10 @@ def add_line_and_source_path(self, node, token):
 
     def nested_render_text(self, text: str, lineno: int):
         """Render unparsed text."""
+        # TODO propogate SourceLines metadata from parent document
+        lines = SourceLines(text, start_line=lineno, standardize_ends=True)
         doc_token = myst_block_tokens.Document.read(
-            text, start_line=lineno, front_matter=True, reset_definitions=False
+            lines, front_matter=True, reset_definitions=False
         )
         # TODO think if this is the best way: here we consume front matter,
         # but then remove it. this is for example if includes have front matter
@@ -184,7 +187,7 @@ def render_front_matter(self, token):
 
         """
         try:
-            data = yaml.safe_load(token.content) or {}
+            data = token.get_data()
         except (yaml.parser.ParserError, yaml.scanner.ScannerError) as error:
             msg_node = self.reporter.error(
                 "Front matter block:\n" + str(error), line=token.position[0]
@@ -911,8 +914,10 @@ def inline_text(self, text: str, lineno: int):
             current_node=paragraph,
             parse_context=get_parse_context(),
         )
+        # TODO propogate SourceLines metadata from parent document
+        lines = SourceLines(text, start_line=self._lineno, standardize_ends=True)
         doc_token = myst_block_tokens.Document.read(
-            text, start_line=self._lineno, front_matter=False, reset_definitions=False
+            lines, front_matter=False, reset_definitions=False
         )
         # we mark the token as nested so that footnotes etc aren't rendered
         doc_token.is_nested = True

myst_parser/sphinx_parser.py

@@ -1,9 +1,13 @@
 from docutils import frontend, nodes
 from sphinx.parsers import Parser
+from sphinx.util import logging
 
+from mistletoe.base_elements import SourceLines
 from myst_parser.docutils_renderer import SphinxRenderer
 from myst_parser.block_tokens import Document
 
+SPHINX_LOGGER = logging.getLogger(__name__)
+
 
 class MystParser(Parser):
     """Docutils parser for CommonMark + Math + Tables + RST Extensions """
@@ -175,6 +179,12 @@ def parse(self, inputstring: str, document: nodes.document):
             pass
         renderer = SphinxRenderer(document=document)
         with renderer:
-            # TODO capture parsing errors and report via docutils/sphinx
-            doc = Document.read(inputstring)
+            # Log to sphinx (e.g. to warn of duplicate link/footnote definitions)
+            renderer.parse_context.logger = SPHINX_LOGGER
+            lines = SourceLines(
+                inputstring,
+                metadata={"docname": self.env.docname},
+                standardize_ends=True,
+            )
+            doc = Document.read(lines)
             renderer.render(doc)

setup.py

@@ -36,7 +36,7 @@
     ],
     keywords="markdown lexer parser development docutils sphinx",
     python_requires=">=3.6",
-    install_requires=["mistletoe-ebp~=0.10.0a2"],
+    install_requires=["mistletoe-ebp~=0.10.0a3"],
     extras_require={
         "sphinx": ["pyyaml", "docutils>=0.15", "sphinx>=2,<3"],
         "code_style": ["flake8<3.8.0,>=3.7.0", "black", "pre-commit==1.17.0"],

tests/test_syntax/test_ast/test_front_matter_basic_strings0_.yml

@@ -6,7 +6,7 @@ front_matter:
 
     '
   position:
-  - 0
+  - 1
   - 3
   type: FrontMatter
 link_definitions: {}