Merge add footnote tokens #7 from ExecutableBookProject/add-footnote

This PR introduces an initial implementation of footnotes: ``` [^a] [^a]: a footnote ``` **NOTE**: currently this only supports single line footnotes, but it is intended that this will eventually support multi-line: ``` [^a] [^a]: a footnote with a block below, that can contain > quotes and other block/span elements and is terminated on the first unindented line, that occurs after an empty line. ```
executablebooks · Mar 10, 2020 · f879ba5 · f879ba5
2 parents 10bc394 + b7cf7e8
commit f879ba5
Show file tree

Hide file tree

Showing 59 changed files with 1,594 additions and 574 deletions.
diff --git a/contrib/jira_renderer.py b/contrib/jira_renderer.py
@@ -148,7 +148,7 @@ def render_table(self, token):
         # The primary difficulty seems to be passing down alignment options to
         # reach individual cells.
         template = "{inner}\n"
-        if hasattr(token, "header"):
+        if getattr(token, "header", None) is not None:
             head_template = "{inner}"
             header = token.children[0]
             head_inner = self.render_table_row(header, True)

diff --git a/contrib/scheme.py b/contrib/scheme.py
@@ -144,6 +144,9 @@ def render_inner(self, token):
             result = self.render(child)
         return result
 
+    def render_program(self, token):
+        return self.render_inner(token)
+
     def render_expr(self, token):
         proc, *args = token.children
         proc = self.render(proc)

diff --git a/docs/api/base_elements.rst b/docs/api/base_elements.rst
@@ -20,11 +20,18 @@ Block Token
     :show-inheritance:
 
 
+.. autoclass:: mistletoe.base_elements.SourceLines
+    :members:
+    :special-members: __next__
+    :undoc-members:
+    :show-inheritance:
+
 Span Token
 ..........
 
 .. autoclass:: mistletoe.base_elements.SpanToken
     :members:
+    :special-members: __init__
     :undoc-members:
     :show-inheritance:
 

diff --git a/docs/api/core_span_tokens.rst b/docs/api/core_span_tokens.rst
@@ -13,98 +13,75 @@ This is a special token that runs a nested parse of the inline string
 and extracts nested tokens.
 
 .. autoclass:: mistletoe.span_tokens.CoreTokens
-    :members:
-    :no-undoc-members:
     :show-inheritance:
-    :exclude-members: __init__
 
 Strong
 ......
 
 .. autoclass:: mistletoe.span_tokens.Strong
-    :members:
-    :no-undoc-members:
     :show-inheritance:
-    :exclude-members: __init__
 
 Emphasis
 .........
 
 .. autoclass:: mistletoe.span_tokens.Emphasis
-    :members:
-    :no-undoc-members:
     :show-inheritance:
-    :exclude-members: __init__
 
 InlineCode
 ..........
 
 .. autoclass:: mistletoe.span_tokens.InlineCode
-    :members:
-    :no-undoc-members:
+    :members: pattern, parse_inner, parse_group
+    :undoc-members:
     :show-inheritance:
-    :exclude-members: __init__
 
 
 Image
 .....
 
 .. autoclass:: mistletoe.span_tokens.Image
-    :members:
-    :no-undoc-members:
     :show-inheritance:
-    :exclude-members: __init__
 
 Link
 ....
 
 .. autoclass:: mistletoe.span_tokens.Link
-    :members:
-    :no-undoc-members:
     :show-inheritance:
-    :exclude-members: __init__
 
 AutoLink
 .........
 
 .. autoclass:: mistletoe.span_tokens.AutoLink
-    :members:
-    :no-undoc-members:
+    :members: pattern, parse_inner, parse_group
+    :undoc-members:
     :show-inheritance:
-    :exclude-members: __init__
 
 EscapeSequence
 ..............
 
 .. autoclass:: mistletoe.span_tokens.EscapeSequence
-    :members:
-    :no-undoc-members:
+    :members: pattern, parse_inner, parse_group
+    :undoc-members:
     :show-inheritance:
-    :exclude-members: __init__
 
 LineBreak
 .........
 
 .. autoclass:: mistletoe.span_tokens.LineBreak
-    :members:
-    :no-undoc-members:
+    :members: pattern, parse_inner, parse_group
+    :undoc-members:
     :show-inheritance:
-    :exclude-members: __init__
 
 RawText
 .......
 
 .. autoclass:: mistletoe.span_tokens.RawText
-    :members:
-    :no-undoc-members:
     :show-inheritance:
-    :exclude-members: __init__
 
 HTMLSpan
 .........
 
 .. autoclass:: mistletoe.span_tokens.HTMLSpan
-    :members:
-    :no-undoc-members:
+    :members: pattern, parse_inner, parse_group
+    :undoc-members:
     :show-inheritance:
-    :exclude-members: __init__
diff --git a/docs/api/extension_tokens.rst b/docs/api/extension_tokens.rst
@@ -12,21 +12,17 @@ Strikethrough
 .............
 
 .. autoclass:: mistletoe.span_tokens_ext.Strikethrough
-    :members:
-    :no-undoc-members:
+    :members: pattern, parse_inner, parse_group
+    :undoc-members:
     :show-inheritance:
-    :exclude-members: __init__
-
 
 Math
 ....
 
 .. autoclass:: mistletoe.span_tokens_ext.Math
-    :members:
-    :no-undoc-members:
+    :members: pattern, parse_inner, parse_group
+    :undoc-members:
     :show-inheritance:
-    :exclude-members: __init__
-
 
 
 FrontMatter
@@ -66,3 +62,22 @@ TableCell
     :no-undoc-members:
     :show-inheritance:
     :exclude-members: __init__
+
+
+FootReference
+.............
+
+.. autoclass:: mistletoe.span_tokens_ext.FootReference
+    :members: pattern, parse_inner, parse_group
+    :undoc-members:
+    :show-inheritance:
+
+
+Footnote
+.........
+
+.. autoclass:: mistletoe.block_tokens_ext.Footnote
+    :members: label_pattern, start, read
+    :undoc-members:
+    :show-inheritance:
+    :exclude-members: __init__
diff --git a/docs/conf.py b/docs/conf.py
@@ -109,6 +109,7 @@ def run_apidoc(app):
     ("py:class", "ForwardRef"),
     ("py:class", "NoneType"),
     ("py:class", "Any"),
+    ("py:class", "Dict"),
     ("py:class", "json.encoder.JSONEncoder"),
 ]
 

diff --git a/docs/index.md b/docs/index.md
@@ -2,18 +2,9 @@
 
 # mistletoe-ebp
 
-[![CI Status][travis-badge]][travis-link]
-[![Coverage][coveralls-badge]][coveralls-link]
-[![Code style: black][black-badge]][black-link]
-[![PyPI][pypi-badge]][pypi-link]
-[![Conda][conda-badge]][conda-link]
-
 mistletoe is a Markdown parser in pure Python,
 designed to be fast, spec-compliant and fully customizable.
 
-This is a version of [mistletoe] maintained by the [Excutable Book Project (EBP)][ebp-link]. It tracks the `myst` branch of [ExecutableBookProject/mistletoe](https://github.com/ExecutableBookProject/mistletoe)
-which eventually, it is hoped, will be merged into mistletoe itself.
-
 Apart from being the fastest
 CommonMark-compliant Markdown parser implementation in pure Python,
 mistletoe also supports easy definitions of custom tokens.
@@ -23,15 +14,23 @@ without touching any of the core components.
 
 Remember to spell mistletoe in lowercase!
 
+```{note}
+This is a version of [mistletoe] maintained by the [Excutable Book Project (EBP)][ebp-link].
+It tracks the `myst` branch of [ExecutableBookProject/mistletoe](https://github.com/ExecutableBookProject/mistletoe)
+which eventually, it is hoped, will be merged into mistletoe itself.
+
+[ebp-link]: https://github.com/ExecutableBookProject
+[mistletoe]: https://github.com/miyuchina/mistletoe
+```
+
 ## Features
 
 * **Fast**:
   mistletoe is the fastest implementation of CommonMark in Python,
   that is, 2 to 3 times as fast as [Commonmark-py][commonmark-py],
   and still roughly 30% faster than [Python-Markdown][python-markdown].
   Running with PyPy yields comparable performance with [mistune][mistune].
-
-  See the [performance](#performance) section for details.
+  See the {ref}`Performance section<intro/performance>` for details.
 
 * **Spec-compliant**:
   CommonMark is [a useful, high-quality project][oilshell].
@@ -43,9 +42,14 @@ Remember to spell mistletoe in lowercase!
   Strikethrough and tables are supported natively,
   and custom block-level and span-level tokens can easily be added.
   Writing a new renderer for mistletoe is a relatively
-  trivial task.
+  trivial task. See the {ref}`Developer section <develop/intro>` for details.
 
-  You can even write [a Lisp][scheme] in it.
+* **LSP compliant**:
+  `mistletoe-ebp` aims to make it easy to use in implementations of the
+  [Language Server Protocol][lsp], the requirements being that;
+  (a) it is thread-safe for asynchronous parsing, and
+  (b) the line and character ranges of the source text are recorded
+  for each element in the syntax tree.
 
 Some alternative output formats:
 
@@ -79,18 +83,6 @@ using/contributing.md
 api/index.rst
 ```
 
-[ebp-link]: https://github.com/ExecutableBookProject
-[travis-badge]: https://travis-ci.org/ExecutableBookProject/mistletoe-ebp.svg?branch=master
-[travis-link]: https://travis-ci.org/ExecutableBookProject/mistletoe-ebp
-[coveralls-badge]: https://coveralls.io/repos/github/ExecutableBookProject/mistletoe-ebp/badge.svg?branch=master
-[coveralls-link]: https://coveralls.io/github/ExecutableBookProject/mistletoe-ebp?branch=master
-[black-badge]: https://img.shields.io/badge/code%20style-black-000000.svg
-[pypi-badge]: https://img.shields.io/pypi/v/mistletoe-ebp.svg
-[pypi-link]: https://pypi.org/project/mistletoe-ebp
-[conda-badge]: https://anaconda.org/conda-forge/mistletoe-ebp/badges/version.svg
-[conda-link]: https://anaconda.org/conda-forge/mistletoe-ebp
-[black-link]: https://github.com/ambv/black
-[mistletoe]: https://github.com/miyuchina/mistletoe
 [mistune]: https://github.com/lepture/mistune
 [python-markdown]: https://github.com/waylan/Python-Markdown
 [python-markdown2]: https://github.com/trentm/python-markdown2
@@ -102,3 +94,4 @@ api/index.rst
 [example-392]: https://spec.commonmark.org/0.28/#example-392
 [icon]: https://www.freepik.com
 [cc-by]: https://creativecommons.org/licenses/by/3.0/us/
+[lsp]: https://microsoft.github.io/language-server-protocol/
diff --git a/docs/using/contributing.md b/docs/using/contributing.md
@@ -1,3 +1,5 @@
+(contribute)=
+
 # Contributing
 
 ## Code Style

diff --git a/docs/using/develop.md b/docs/using/develop.md
@@ -2,6 +2,12 @@
 
 ## Developer's Guide
 
+[![CI Status][travis-badge]][travis-link]
+[![Coverage][coveralls-badge]][coveralls-link]
+[![Code style: black][black-badge]][black-link]
+[![PyPI][pypi-badge]][pypi-link]
+[![Conda][conda-badge]][conda-link]
+
 To understand the core tokens that mistletoe parses, take a look at:
 
 - {ref}`tokens/base`
@@ -186,3 +192,15 @@ with open('foo.md', 'r') as fin:
     with GithubWikiRenderer() as renderer:
         rendered = renderer.render(Document.read(fin))
 ```
+
+
+[travis-badge]: https://travis-ci.org/ExecutableBookProject/mistletoe-ebp.svg?branch=master
+[travis-link]: https://travis-ci.org/ExecutableBookProject/mistletoe-ebp
+[coveralls-badge]: https://coveralls.io/repos/github/ExecutableBookProject/mistletoe-ebp/badge.svg?branch=master
+[coveralls-link]: https://coveralls.io/github/ExecutableBookProject/mistletoe-ebp?branch=master
+[black-badge]: https://img.shields.io/badge/code%20style-black-000000.svg
+[pypi-badge]: https://img.shields.io/pypi/v/mistletoe-ebp.svg
+[pypi-link]: https://pypi.org/project/mistletoe-ebp
+[conda-badge]: https://anaconda.org/conda-forge/mistletoe-ebp/badges/version.svg
+[conda-link]: https://anaconda.org/conda-forge/mistletoe-ebp
+[black-link]: https://github.com/ambv/black