Move to markdown-it-py parser (#123)

This commit implements the move from `mistletoe` to `markdown-it-py`, as the underlying markdown parser. The reason for this is are discussed in executablebooks/meta#44 (comment) and executablebooks/meta#47, and the PR #123 discusses in more details the update.

Additional changes:

- Update `pydata-sphinx-theme` requirement
- Improve testing and move to GitHub Actions CI
- Add tests and fixes for reporter warnings and include directive
- Add documentation of sphinx parser options
- Apply doc fixes suggested by @rossbar in #121
- Add warning for non-consecutive headings

.circleci/config.yml

@@ -13,7 +13,6 @@ jobs:
             - cache-pip
       - run: |
           pip install --user -e .[sphinx,rtd]
-          pip install -r docs/requirements.txt
       - save_cache:
           key: cache-pip
           paths:

.github/ISSUE_TEMPLATE/bug_report.md

@@ -20,8 +20,17 @@ Steps to reproduce the behavior:
 **Expected behavior**
 A clear and concise description of what you expected to happen.
 
-**Screenshots**
-If applicable, add screenshots to help explain your problem.
+If relevant, a minimal example of the input text should be supplied,
+together with a screen-shot of the output Sphinx document and/or command-line output, e.g.
+
+```markdown
+some text...
+```
+
+```console
+$ make html
+ERROR ...
+```
 
 **Environment (please complete the following information):**
  - Python Version [e.g. 3.7.1]

.github/workflows/tests.yml

@@ -0,0 +1,62 @@
+# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
+# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
+
+name: Python package
+
+on: [push, pull_request]
+  # push:
+  #   branches: [ master ]
+  # pull_request:
+  #   branches: [ master ]
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [3.6, 3.7, 3.8]
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v1
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install .[sphinx,code_style,testing]
+    - name: Pre-commit checks
+      run: |
+        pre-commit run --all-files || ( git status --short ; git diff ; exit 1 )
+    # cd tests/test_commonmark && ./spec.sh
+    - name: Test with pytest
+      run: |
+        pip install pytest
+        pytest --cov=myst_parser --cov-report=
+    - name: Upload to coveralls
+      run: |
+        pip install coveralls
+        coveralls
+      env:
+          COVERALLS_REPO_TOKEN: ${{ secrets.COVERALLS_KEY }}
+
+  publish:
+
+    name: Publish to PyPi
+    needs: build
+    if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout source
+        uses: actions/checkout@v2
+      - name: Build package
+        run: |
+          pip install wheel
+          python setup.py sdist bdist_wheel
+      - name: Publish
+        uses: pypa/gh-action-pypi-publish@v1.1.0
+        with:
+          user: __token__
+          password: ${{ secrets.PYPI_KEY }}

.gitignore

@@ -127,3 +127,5 @@ dmypy.json
 
 # Pyre type checker
 .pyre/
+
+_archive/

.pre-commit-config.yaml

@@ -5,7 +5,8 @@ exclude: >
     (?x)^(
       \.vscode/settings\.json|
       tests/test_commonmark/commonmark\.json|
-      .*\.xml
+      .*\.xml|
+      tests/.*/fixtures/.*\.md
     )$
 
 repos:

.readthedocs.yml

@@ -8,4 +8,3 @@ python:
         extra_requirements:
           - sphinx
           - rtd
-      - requirements: docs/requirements.txt

.vscode/settings.json

@@ -16,5 +16,6 @@
     "python.linting.flake8Enabled": true,
     "python.linting.enabled": true,
     "autoDocstring.customTemplatePath": "docstring.fmt.mustache",
-    "python.pythonPath": "/anaconda/envs/ebp/bin/python"
+    "python.pythonPath": "/anaconda/envs/ebp/bin/python",
+    "restructuredtext.confPath": "${workspaceFolder}/docs"
 }

MANIFEST.in

@@ -1 +1,2 @@
 include LICENSE
+include myst_parser/cli/spec.md

README.md

@@ -1,6 +1,6 @@
 # MyST-Parser
 
-[![CI Status][travis-badge]][travis-link]
+[![Github-CI][github-ci]][github-link]
 [![Coverage][coveralls-badge]][coveralls-link]
 [![Documentation Status][rtd-badge]][rtd-link]
 [![Code style: black][black-badge]][black-link]
@@ -35,8 +35,8 @@ pip install -e .[sphinx,code_style,testing,rtd]
 
 To use the MyST parser in Sphinx, simply add: `extensions = ["myst_parser"]` to your `conf.py`.
 
-[travis-badge]: https://travis-ci.org/ExecutableBookProject/MyST-Parser.svg?branch=master
-[travis-link]: https://travis-ci.org/ExecutableBookProject/MyST-Parser
+[github-ci]: https://github.com/ExecutableBookProject/MyST-Parser/workflows/Python%20package/badge.svg?branch=master
+[github-link]: https://github.com/ExecutableBookProject/MyST-Parser
 [coveralls-badge]: https://coveralls.io/repos/github/ExecutableBookProject/MyST-Parser/badge.svg?branch=master
 [coveralls-link]: https://coveralls.io/github/ExecutableBookProject/MyST-Parser?branch=master
 [rtd-badge]: https://readthedocs.org/projects/myst-parser/badge/?version=latest

docs/api/index.md

@@ -5,7 +5,6 @@
 ```{toctree}
 :maxdepth: 2
 
-tokens.rst
 directive.rst
 renderers.rst
 sphinx_parser.rst

docs/api/renderers.rst

@@ -3,71 +3,69 @@
 MyST Renderers
 --------------
 
-MyST-Parser builds on the mistletoe
-:ref:`core renderers <mistletoe:renderers/core>`
-by including the extended tokens, listed in :ref:`api/tokens`,
-and adding bridges to docutils/sphinx:
 
-HTML
-....
+These renderers take the markdown-it parsed token stream and convert it to
+the docutils AST. The sphinx renderer is a subclass of the docutils one,
+with some additional methods only available *via* sphinx
+.e.g. multi-document cross-referencing.
 
-.. autoclass:: myst_parser.html_renderer.HTMLRenderer
-    :special-members: __init__, __enter__, __exit__
-    :members: default_block_tokens, default_span_tokens
-    :undoc-members:
-    :member-order: alphabetical
-    :show-inheritance:
 
+Docutils
+........
 
-JSON
-....
-
-.. autoclass:: myst_parser.json_renderer.JsonRenderer
-    :special-members: __init__, __enter__, __exit__
-    :members: default_block_tokens, default_span_tokens
+.. autoclass:: myst_parser.docutils_renderer.DocutilsRenderer
+    :special-members: __output__, __init__
+    :members: render, nested_render_text, add_line_and_source_path, current_node_context
     :undoc-members:
-    :member-order: alphabetical
+    :member-order: bysource
     :show-inheritance:
 
-Docutils
-........
 
-.. autoclass:: myst_parser.docutils_renderer.DocutilsRenderer
-    :special-members: __init__, __enter__, __exit__
-    :members: default_block_tokens, default_span_tokens, new_document
+Sphinx
+......
+
+.. autoclass:: myst_parser.sphinx_renderer.SphinxRenderer
+    :special-members: __output__
+    :members: handle_cross_reference, render_math_block_eqno
     :undoc-members:
     :member-order: alphabetical
     :show-inheritance:
 
+Mocking
+.......
+
+These classes are parsed to sphinx roles and directives,
+to mimic the original docutls rST specific parser elements,
+but instead run nested parsing with the markdown parser.
 
-.. autoclass:: myst_parser.docutils_renderer.MockInliner
+.. autoclass:: myst_parser.mocking.MockInliner
     :members:
     :undoc-members:
     :show-inheritance:
 
-.. autoclass:: myst_parser.docutils_renderer.MockState
+.. autoclass:: myst_parser.mocking.MockState
     :members:
     :undoc-members:
     :show-inheritance:
 
-.. autoclass:: myst_parser.docutils_renderer.MockStateMachine
+.. autoclass:: myst_parser.mocking.MockStateMachine
     :members:
     :undoc-members:
     :show-inheritance:
 
-.. autoclass:: myst_parser.docutils_renderer.MockIncludeDirective
+.. autoclass:: myst_parser.mocking.MockIncludeDirective
     :members:
     :undoc-members:
     :show-inheritance:
 
-Sphinx
-......
 
-.. autoclass:: myst_parser.docutils_renderer.SphinxRenderer
-    :special-members: __init__, __enter__, __exit__
-    :members: default_block_tokens, default_span_tokens, mock_sphinx_env
-    :undoc-members:
-    :member-order: alphabetical
-    :show-inheritance:
+Additional Methods
+..................
+
+.. autofunction:: myst_parser.docutils_renderer.make_document
 
 .. autofunction:: myst_parser.docutils_renderer.dict_to_docinfo
+
+.. autofunction:: myst_parser.sphinx_renderer.minimal_sphinx_app
+
+.. autofunction:: myst_parser.sphinx_renderer.mock_sphinx_env

docs/api/sphinx_parser.rst

@@ -3,11 +3,12 @@
 Sphinx Parser
 -------------
 
-This class builds on the :py:class:`~myst_parser.docutils_renderer.SphinxRenderer`
+This class builds on the :py:class:`~myst_parser.sphinx_renderer.SphinxRenderer`
 to generate a parser for Sphinx, using the :ref:`Sphinx parser API <sphinx:parser-api>`:
 
 .. autoclass:: myst_parser.sphinx_parser.MystParser
-    :members:
-    :no-undoc-members:
+    :members: default_config, supported, parse
+    :undoc-members:
+    :member-order: bysource
     :show-inheritance:
     :exclude-members: __init__

docs/conf.py

@@ -50,8 +50,11 @@
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = "pandas_sphinx_theme"
+html_theme = "pydata_sphinx_theme"
 html_logo = "_static/logo.png"
+html_theme_options = {
+    "github_url": "https://github.com/ExecutableBookProject/MyST-Parser"
+}
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,