Update from fork

Update mirroring:

Fork:  ExecutableBookProject/mistletoe
Branch: myst
Commit: 4840fcd0aa4fb5976ce470640e444c34fc8b8291

.travis.yml

@@ -2,8 +2,6 @@ language: python
 cache: pip
 matrix:
     include:
-      - python: 3.5
-        env: TEST_TYPE="pytest"
       - python: 3.6
         env: TEST_TYPE="pytest"
       - python: 3.7

.vscode/settings.json

@@ -14,4 +14,5 @@
     "python.linting.flake8Enabled": true,
     "python.linting.enabled": true,
     "python.pythonPath": "/anaconda/envs/ebp/bin/python",
+    "autoDocstring.customTemplatePath": "docstring.fmt.mustache"
 }

README.md

@@ -24,16 +24,16 @@ Unfortunately, mistletoe is not currently being actively maintained
 (as of June 8th 2019), and so this fork has been created to allow for a
 deployed release that can be utilised by EBP. Here is a working list of 'up-streamable' changes that would be desired of mistletoe that this version has begun to implement:
 
-- Move testing from `unittest` to `pytest`: `pytest` is now the *de facto* testing architecture and vastly improves the usability/flexibility of testing.
-- Introduce `pre-commit` code linting and formatting: This standardizes the code style across the package, and ensures that new commits and Pull Requests also conform to it.
-- Introduce `ReadTheDocs` documentation
-- Add a conda-forge distribution of the package
-- Improve the AST API and documentation: I view [panflute](http://scorreia.com/software/panflute/index.html)'s implementation of the pandoc API in python, as the gold standard for how a pythonic AST API should be written and documented. Some tweaks to the current token class objects, and creating auto-generated RTD documentation, could achieve this.
-- Storage of source line/column ranges: LSP and good rendering reporting of warnings/errors, requires access to the source line and column ranges of each parsed token.
-- Asynchronous parsing: LSP requires documents to be parsed asynchronously. Currently, mistletoe contains a number of global state objects, which make parsing inherently not thread-safe. The simple solution to this is to store these items as `threading.local` objects. A related but slightly more complete solution is to introduce the idea of a 'scoped session', similar to that used by sqlalchemy for database access: [Contextual/Thread-local Sessions](https://docs.sqlalchemy.org/en/13/orm/contextual.html#unitofwork-contextual)
-- Improve extensibility of block tokens: A Markdown parser is inherently a Finite State-Machine + Memory (a.k.a Push-down Automata (PDA)), with parsing tokens as states (for a good example of a python state-machine see [pytransitions](https://github.com/pytransitions/transitions)). The problem with extensibility, is that inherently states are interdependent; when introducing a new state/token you must provide logic to all the other tokens, w.r.t to when to transition to this new token. Currently, MyST Parser sub-classes nearly all the Mistletoe block tokens to implement the extensions it requires, but it would be ideal if there was a more systematic approach for this.
-- Improve extensibility for span tokens: Mistletoe does allow for span token extensions to be added, at least in a simple way. However, as with block tokens above, there is often an interconnectivity to them, especially when considering nested span tokens. As of 7cc2c92, MyST-Parser now overrides some of Mistetoe's core logic to achieve correct parsing of Math tokens, but if possible this should be made more general.
-- Improve rendering logic: Currently, there is no concept of recursive walk-throughs or 'visitor' patterns in the Misteltoe `BaseRenderer`, which is a better method for rendering tree like structures (as used by docutils/panflute). Also, the current token instantiating (within context managers) needs improvement (see [miyuchina/mistletoe#56](https://github.com/miyuchina/mistletoe/issues/56)).
+- [x] Move testing from `unittest` to `pytest`: `pytest` is now the *de facto* testing architecture and vastly improves the usability/flexibility of testing.
+- [x] Introduce `pre-commit` code linting and formatting: This standardizes the code style across the package, and ensures that new commits and Pull Requests also conform to it.
+- [x] Introduce `ReadTheDocs` documentation
+- [x] Add a conda-forge distribution of the package
+- [ ] Improve the AST API and documentation: I view [panflute](http://scorreia.com/software/panflute/index.html)'s implementation of the pandoc API in python, as the gold standard for how a pythonic AST API should be written and documented. Some tweaks to the current token class objects, and creating auto-generated RTD documentation, could achieve this.
+- [ ] Storage of source line/column ranges: LSP and good rendering reporting of warnings/errors, requires access to the source line and column ranges of each parsed token.
+- [x] Asynchronous parsing: LSP requires documents to be parsed asynchronously. Currently, mistletoe contains a number of global state objects, which make parsing inherently not thread-safe. The simple solution to this is to store these items as `threading.local` objects. A related but slightly more complete solution is to introduce the idea of a 'scoped session', similar to that used by sqlalchemy for database access: [Contextual/Thread-local Sessions](https://docs.sqlalchemy.org/en/13/orm/contextual.html#unitofwork-contextual)
+- [ ] Improve extensibility of block tokens: A Markdown parser is inherently a Finite State-Machine + Memory (a.k.a Push-down Automata (PDA)), with parsing tokens as states (for a good example of a python state-machine see [pytransitions](https://github.com/pytransitions/transitions)). The problem with extensibility, is that inherently states are interdependent; when introducing a new state/token you must provide logic to all the other tokens, w.r.t to when to transition to this new token. Currently, MyST Parser sub-classes nearly all the Mistletoe block tokens to implement the extensions it requires, but it would be ideal if there was a more systematic approach for this.
+- [ ] Improve extensibility for span tokens: Mistletoe does allow for span token extensions to be added, at least in a simple way. However, as with block tokens above, there is often an interconnectivity to them, especially when considering nested span tokens. As of 7cc2c92, MyST-Parser now overrides some of Mistetoe's core logic to achieve correct parsing of Math tokens, but if possible this should be made more general.
+- [ ] Improve rendering logic: Currently, there is no concept of recursive walk-throughs or 'visitor' patterns in the Misteltoe `BaseRenderer`, which is a better method for rendering tree like structures (as used by docutils/panflute). Also, the current token instantiating (within context managers) needs improvement (see [miyuchina/mistletoe#56](https://github.com/miyuchina/mistletoe/issues/56)).
 
 [mistletoe]: https://github.com/miyuchina/mistletoe
 [ebp-link]: https://github.com/ExecutableBookProject

contrib/github_wiki.py

@@ -3,8 +3,8 @@
 """
 
 import re
-from mistletoe.span_token import SpanToken
-from mistletoe.html_renderer import HTMLRenderer
+from mistletoe.base_elements import SpanToken
+from mistletoe.renderers.html import HTMLRenderer
 
 
 __all__ = ["GithubWiki", "GithubWikiRenderer"]

contrib/jira_renderer.py

@@ -23,8 +23,8 @@
 
 import html
 from itertools import chain
-from mistletoe import block_token, span_token
-from mistletoe.base_renderer import BaseRenderer
+from mistletoe import block_tokens, span_tokens
+from mistletoe.renderers.base import BaseRenderer
 
 
 class JIRARenderer(BaseRenderer):
@@ -40,7 +40,7 @@ def __init__(self, *extras):
             extras (list): allows subclasses to add even more custom tokens.
         """
         self.listTokens = []
-        super().__init__(*chain([block_token.HTMLBlock, span_token.HTMLSpan], extras))
+        super().__init__(*chain([block_tokens.HTMLBlock, span_tokens.HTMLSpan], extras))
 
     def render_strong(self, token):
         template = "*{}*"
@@ -125,15 +125,15 @@ def render_list_item(self, token):
         return result
 
     def render_inner(self, token):
-        if isinstance(token, block_token.List):
-            if token.start:
+        if isinstance(token, block_tokens.List):
+            if token.start_at:
                 self.listTokens.append("#")
             else:
                 self.listTokens.append("*")
 
         rendered = [self.render(child) for child in token.children]
 
-        if isinstance(token, block_token.List):
+        if isinstance(token, block_tokens.List):
             del self.listTokens[-1]
 
         return "".join(rendered)
@@ -192,7 +192,7 @@ def render_html_block(token):
         return token.content
 
     def render_document(self, token):
-        self.footnotes.update(token.footnotes)
+        self.link_definitions.update(token.link_definitions)
         return self.render_inner(token)
 
 

contrib/mathjax.py

@@ -2,8 +2,8 @@
 Provides MathJax support for rendering Markdown with LaTeX to html.
 """
 
-from mistletoe.html_renderer import HTMLRenderer
-from mistletoe.latex_renderer import LaTeXRenderer
+from mistletoe.renderers.html import HTMLRenderer
+from mistletoe.renderers.latex import LaTeXRenderer
 
 
 class MathJaxRenderer(HTMLRenderer, LaTeXRenderer):

contrib/scheme.py

@@ -2,18 +2,18 @@
 from functools import reduce
 import re
 
-from mistletoe import BaseRenderer, span_token, block_token
-from mistletoe.core_tokens import MatchObj
+from mistletoe import BaseRenderer, base_elements
+from span_tokenizer import tokenize_span
+from mistletoe.nested_tokenizer import MatchObj
+from mistletoe.parse_context import get_parse_context
 
 
-class Program(block_token.BlockToken):
+class Program(base_elements.BlockToken):
     def __init__(self, lines):
-        self.children = span_token.tokenize_inner(
-            "".join([line.strip() for line in lines])
-        )
+        self.children = tokenize_span("".join([line.strip() for line in lines]))
 
 
-class Expr(span_token.SpanToken):
+class Expr(base_elements.SpanToken):
     @classmethod
     def find(cls, string):
         matches = []
@@ -32,7 +32,7 @@ def __repr__(self):
         return "<Expr {}>".format(self.children)
 
 
-class Number(span_token.SpanToken):
+class Number(base_elements.SpanToken):
     pattern = re.compile(r"(\d+)")
     parse_inner = False
 
@@ -43,7 +43,7 @@ def __repr__(self):
         return "<Number {}>".format(self.number)
 
 
-class Variable(span_token.SpanToken):
+class Variable(base_elements.SpanToken):
     pattern = re.compile(r"([^\s()]+)")
     parse_inner = False
 
@@ -54,7 +54,7 @@ def __repr__(self):
         return "<Variable {!r}>".format(self.name)
 
 
-class Whitespace(span_token.SpanToken):
+class Whitespace(base_elements.SpanToken):
     parse_inner = False
 
     def __new__(self, _):
@@ -76,8 +76,9 @@ def __init__(self):
             "Number": self.render_number,
             "Variable": self.render_variable,
         }
-        block_token._token_types.value = []
-        span_token._token_types.value = [Expr, Number, Variable, Whitespace]
+        parse_context = get_parse_context()
+        parse_context.block_tokens = []
+        parse_context.span_tokens = [Expr, Number, Variable, Whitespace]
 
         self.env = ChainMap(
             {

contrib/toc_renderer.py

@@ -5,7 +5,7 @@
 """
 
 import re
-from mistletoe.html_renderer import HTMLRenderer
+from mistletoe.renderers.html import HTMLRenderer
 
 
 class TOCRenderer(HTMLRenderer):
@@ -30,9 +30,9 @@ def __init__(self, depth=5, omit_title=True, filter_conds=[], *extras):
     @property
     def toc(self):
         """
-        Returns table of contents as a block_token.List instance.
+        Returns table of contents as a block_tokens.List instance.
         """
-        from mistletoe.block_token import List
+        from mistletoe.block_tokens import List
 
         def get_indent(level):
             if self.omit_title:

docs/using/index.md

@@ -44,7 +44,7 @@ and rendering to HTML. The function also accepts an additional argument
 
 ```python
 import mistletoe
-from mistletoe.latex_renderer import LaTeXRenderer
+from mistletoe.renderers.latex import LaTeXRenderer
 
 with open('foo.md', 'r') as fin:
     rendered = mistletoe.markdown(fin, LaTeXRenderer)
@@ -60,7 +60,7 @@ from mistletoe import Document, HTMLRenderer
 
 with open('foo.md', 'r') as fin:
     with HTMLRenderer() as renderer:
-        rendered = renderer.render(Document(fin))
+        rendered = renderer.render(Document.read(fin))
 ```
 
 ### From the command-line
@@ -125,15 +125,15 @@ mistletoe is the fastest CommonMark compliant implementation in Python.
 Try the benchmarks yourself by running:
 
 ```sh
-$ python3 test/benchmark.py  # all results in seconds
-Test document: test/samples/syntax.md
+$ python3 test/test_samples/benchmark.py  # all results in seconds
+Test document: syntax.md
 Test iterations: 1000
 Running tests with markdown, mistune, commonmark, mistletoe...
 ==============================================================
-markdown: 33.28557115700096
-mistune: 8.533771439999327
-commonmark: 84.54588776299897
-mistletoe: 23.5405140980001
+markdown: 40.270715949
+mistune: 11.054077996000004
+commonmark: 44.426582849
+mistletoe: 34.47910147500001
 ```
 
 We notice that Mistune is the fastest Markdown parser,
@@ -331,7 +331,7 @@ of most of them for you. Simply pass your custom token class to
 `super().__init__()` does the trick:
 
 ```python
-from mistletoe.html_renderer import HTMLRenderer
+from mistletoe.renderers.html import HTMLRenderer
 
 class GithubWikiRenderer(HTMLRenderer):
     def __init__(self):
@@ -350,7 +350,7 @@ def render_github_wiki(self, token):
 Cleaning up, we have our new renderer class:
 
 ```python
-from mistletoe.html_renderer import HTMLRenderer, escape_url
+from mistletoe.renderers.html import HTMLRenderer, escape_url
 
 class GithubWikiRenderer(HTMLRenderer):
     def __init__(self):
@@ -376,7 +376,7 @@ from contrib.github_wiki import GithubWikiRenderer
 
 with open('foo.md', 'r') as fin:
     with GithubWikiRenderer() as renderer:
-        rendered = renderer.render(Document(fin))
+        rendered = renderer.render(Document.read(fin))
 ```
 
 For more info, take a look at the `base_renderer` module in mistletoe.

docstring.fmt.mustache

@@ -0,0 +1,20 @@
+{{! Sphinx Docstring Template }}
+{{summaryPlaceholder}}
+
+{{extendedSummaryPlaceholder}}
+
+{{#args}}
+:param {{var}}: {{descriptionPlaceholder}}
+{{/args}}
+{{#kwargs}}
+:param {{var}}: {{descriptionPlaceholder}}
+{{/kwargs}}
+{{#exceptions}}
+:raises {{type}}: {{descriptionPlaceholder}}
+{{/exceptions}}
+{{#returns}}
+:return: {{descriptionPlaceholder}}
+{{/returns}}
+{{#yields}}
+:yield: {{descriptionPlaceholder}}
+{{/yields}}

mistletoe/__init__.py

@@ -2,19 +2,19 @@
 Make mistletoe easier to import.
 """
 
-__version__ = "0.8.2"
+__version__ = "0.9.0"
 __all__ = [
-    "html_renderer",
-    "ast_renderer",
-    "block_token",
+    "renderers",
+    "base_elements",
+    "block_tokens",
     "block_tokenizer",
-    "span_token",
+    "span_tokens",
     "span_tokenizer",
 ]
 
-from mistletoe.block_token import Document
-from mistletoe.base_renderer import BaseRenderer  # noqa: F401
-from mistletoe.html_renderer import HTMLRenderer
+from mistletoe.block_tokens import Document
+from mistletoe.renderers.base import BaseRenderer  # noqa: F401
+from mistletoe.renderers.html import HTMLRenderer
 
 
 def markdown(iterable, renderer=HTMLRenderer):
@@ -23,4 +23,4 @@ def markdown(iterable, renderer=HTMLRenderer):
     Enables inline and block-level HTML tags.
     """
     with renderer() as renderer:
-        return renderer.render(Document(iterable))
+        return renderer.render(Document.read(iterable))