✨ NEW: Add Docutils MyST config and CLI #426

cpitclaudel · 2021-09-17T03:51:02Z

This is a follow-up to my previous PR (GH-418), part of an effort (GH-420) to make myst_parser integrate better with Docutils.
This exposes (almost) all MyST configuration settings in docutils.conf, making it possible to configure myst_parser for use without Sphinx.

I've added a simple entry point to make it easier to experiment (the second commit on this branch). Here's a quick demo:

# We have access to the standard docutils command line interface:
$ python3 -m myst_parser.docutils_ -h
...
Usage
=====
  docutils_.py [options] [<source> [<destination>]]

Generates (X)HTML documents from standalone MyST sources.  Reads from <source>
(default is stdin) and writes to <destination> (default is stdout).  See
<http://docutils.sf.net/docs/user/config.html> for the full reference.

Options
=======
General Docutils Options
------------------------
...

MyST options
------------
--myst-commonmark-only=MYST_COMMONMARK_ONLY
--myst-enable-extensions=MYST_ENABLE_EXTENSIONS
--myst-dmath-allow-labels=MYST_DMATH_ALLOW_LABELS
--myst-dmath-allow-space=MYST_DMATH_ALLOW_SPACE
--myst-dmath-allow-digits=MYST_DMATH_ALLOW_DIGITS
--myst-dmath-double-inline=MYST_DMATH_DOUBLE_INLINE
--myst-update-mathjax=MYST_UPDATE_MATHJAX
--myst-mathjax-classes=MYST_MATHJAX_CLASSES
--myst-disable-syntax=MYST_DISABLE_SYNTAX
--myst-url-schemes=MYST_URL_SCHEMES
--myst-heading-anchors=MYST_HEADING_ANCHORS
--myst-footnote-transition=MYST_FOOTNOTE_TRANSITION
--myst-sub-delimiters=MYST_SUB_DELIMITERS
--myst-words-per-minute=MYST_WORDS_PER_MINUTE
...

# We can compile markdown documents directly:
$ printf "# Title\n:::{note}\nIt works!\n:::" > test.md
$ python3 -m myst_parser.docutils_ --myst-enable-extensions=colon_fence test.md 
...
<body>
<div class="tex2jax_ignore mathjax_ignore document" id="title">
<h1 class="title">Title</h1>

<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">It works!</p>
</div>
</div>
</body>
</html>

Remaining issues:

Math doesn't work, because of [draft] Remove option mathjax_classes #406. I think it would be reasonable to include the fix from that PR (add a transform that adds mathjax_process where relevant), but I'm happy to consider alternatives.
There's no documentation for CLI options, because the documentation is on the web but not in the code.
I need help with writing tests, as I don't have time to understand how the tests work at the moment :/

This is part of executablebooksGH-420, an effort to make myst_parser integrate better with Docutils. * myst_parser/docutils_.py (Parser): Use new _myst_docutils_setting_tuples function to generate a settings_spec. (Parser.parse): Use new create_myst_config function to reconstruct a MdParserConfig object from a Docutils configuration object.

for more information, see https://pre-commit.ci

chrisjsewell · 2021-09-17T04:06:10Z

cheers, I’ll give a proper look soon

There's no documentation for CLI options, because the documentation is on the web but not in the code.

It would be very possible to self document the MDConfig attr class via metadata. I’ve already done it in other projects: https://github.com/chrisjsewell/ase-notebook/blob/eb12723f0aed4e774484f0e8bce10b3785c0a2e5/ase_notebook/configuration.py#L114

I need help with writing tests, as I don't have time to understand how the tests work at the moment :/

is it ok if I push directly to your branch?

cpitclaudel · 2021-09-17T12:56:25Z

is it ok if I push directly to your branch?

Please do, thanks a lot!

cpitclaudel · 2021-09-17T12:57:23Z

It would be very possible to self document the MDConfig attr class via metadata.

Yep, that would be ideal. Ideally that would also be used to generate the relevant section of the docs, to avoid having two sources of truth. I just didn't find the time to do it ^^

codecov · 2021-12-05T01:21:46Z

Codecov Report

Merging #426 (2782088) into master (86bf84d) will decrease coverage by 0.51%.
The diff coverage is 79.34%.

@@            Coverage Diff             @@
##           master     #426      +/-   ##
==========================================
- Coverage   90.84%   90.32%   -0.52%     
==========================================
  Files          16       16              
  Lines        1912     1985      +73     
==========================================
+ Hits         1737     1793      +56     
- Misses        175      192      +17

Flag	Coverage Δ
pytests	`90.32% <79.34%> (-0.52%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
myst_parser/sphinx_.py	`100.00% <ø> (+100.00%)`	⬆️
myst_parser/docutils_.py	`78.88% <75.00%> (-21.12%)`	⬇️
myst_parser/docutils_renderer.py	`92.83% <100.00%> (+0.03%)`	⬆️
myst_parser/main.py	`88.61% <100.00%> (+0.81%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 86bf84d...2782088. Read the comment docs.

chrisjsewell · 2021-12-05T02:02:08Z

Ok @cpitclaudel, how's this!?

Now you have separate CLI commands for writers: myst-docutils-html myst-docutils-html5 myst-docutils-latex myst-docutils-pseudoxml myst-docutils-xml

and full doc-strings:

$ myst-docutils-xml --help
Usage
=====
  myst-docutils-xml [options] [<source> [<destination>]]

Generates Docutils-native XML from standalone MyST sources. Reads from
<source> (default is stdin) and writes to <destination> (default is stdout).
See <http://docutils.sf.net/docs/user/config.html> for the full reference.

Options
=======
...
MyST options
------------
--myst-commonmark-only=MYST_COMMONMARK_ONLY
                        Use strict CommonMark parser (type: bool, default:
                        False)
--myst-enable-extensions=MYST_ENABLE_EXTENSIONS
                        Enable extensions (type: comma-delimited, default:
                        'dollarmath')
--myst-dmath-allow-labels=MYST_DMATH_ALLOW_LABELS
                        Parse `$$...$$ (label)` (type: bool, default: True)
--myst-dmath-allow-space=MYST_DMATH_ALLOW_SPACE
                        dollarmath: allow initial/final spaces in `$ ... $`
                        (type: bool, default: True)
--myst-dmath-allow-digits=MYST_DMATH_ALLOW_DIGITS
                        dollarmath: allow initial/final digits `1$ ...$2`
                        (type: bool, default: True)
--myst-dmath-double-inline=MYST_DMATH_DOUBLE_INLINE
                        dollarmath: parse inline `$$ ... $$` (type: bool,
                        default: False)
--myst-disable-syntax=MYST_DISABLE_SYNTAX
                        Disable syntax elements (type: comma-delimited,
                        default: '')
--myst-url-schemes=MYST_URL_SCHEMES
                        URL schemes to allow in links (type: comma-delimited,
                        default: 'http,https,mailto,ftp')
--myst-footnote-transition=MYST_FOOTNOTE_TRANSITION
                        Place a transition before any footnotes (type: bool,
                        default: True)
--myst-words-per-minute=MYST_WORDS_PER_MINUTE
                        For reading speed calculations (type: int, default:
                        200)

I removed the addition of the section level mathjax classes, when using docutils (since this is only useful with sphinx.ext.mathjax)

I also removed the heading_anchors option, since heading anchor slugs are currently only handled by the sphinx renderer.
It might be nice to eventually allow this functionality for docutils

still to add some extra testing and documentation...

cpitclaudel · 2021-12-05T04:45:35Z

😍

cpitclaudel · 2021-12-05T04:47:46Z

I forget if the heading_anchors option is related to this recently implemented feature in docutils: https://sourceforge.net/p/docutils/feature-requests/28/

cpitclaudel · 2021-12-05T05:01:24Z

On the topic of script names, since Docutils uses rst2XXX.py, maybe we should have myst2xxx.py for consistency?
Although see https://sourceforge.net/p/docutils/feature-requests/82/:

Docutils 0.17 comes with tools/docutils-cli.py:
New generic command line front end that allows the free selection of
reader, parser, and writer components.

chrisjsewell · 2021-12-05T07:07:53Z

maybe we should have myst2xxx.py for consistency

Yeh, I was hesitant to have this, since it does not make it clear that it will only parse using docutils and not sphinx, e.g. you can't properly parse any files that use sphinx/sphinx-extension roles/directives

chrisjsewell · 2021-12-05T07:14:15Z

Although see https://sourceforge.net/p/docutils/feature-requests/82/

I very much agree with this issue though; it's a shame there is currently no easy "middle ground" between the single document docutils API/CLI, lacking any support for sphinx features, and the sphinx-build CLI, requiring use of the full, complex, project build system

chrisjsewell · 2021-12-05T09:05:56Z

myst_parser/docutils_.py

+        setting = f"myst_{attribute.name}"
+        val = getattr(settings, setting, DOCUTILS_UNSET)
+        with suppress(AttributeError):
+            delattr(settings, setting)


@cpitclaudel why do you need to delete the setting here?

(note here I made the getting/deleting optional, since if you include a MyST file from an RST file, the MyST settings will not have been loaded)

I forget the details :/ I think it was to avoid namespace pollution? I'm guessing I was worried about having the settings both directly in the document under myst_* and in a config object, so I was planning to remove all the myst_* settings from document.settings and instead have a single object document.settings.myst_config.

Of course, this explanation would make more sense if I had then actually stored the config object as document.settings.myst_config, but the code doesn't seem to do that, so either I forgot or my guess above is wrong ^^

I think I'll remove it for now then, as it doesn't appear to break anything

chrisjsewell · 2021-12-05T10:26:34Z

I forget if the heading_anchors option is related to this recently implemented feature in docutils: sourceforge.net/p/docutils/feature-requests/28

If you want to open this as a future improvement issue

chrisjsewell · 2021-12-05T15:09:09Z

Documentation added 😄 https://myst-parser--426.org.readthedocs.build/en/426/docutils.html

chrisjsewell · 2021-12-05T15:13:17Z

Ok I think this is good to go! @cpitclaudel do you want to review your own PR lol, and let me know if you are happy for it to be merged

chrisjsewell · 2021-12-06T15:34:03Z

@cpitclaudel let me know if you happy for this to be merged cheers, eager to get out version 0.16.0 😄 (#426 -> #456 -> #457 -> 🚀)

chrisjsewell · 2021-12-11T02:38:07Z

Ok times up 😛 but I'm sure its all fine

cpitclaudel · 2021-12-11T22:46:27Z

Ok times up but I'm sure its all fine

Yep! Thanks a lot for moving ahead, I'm terrible at email and super busy these days. Very excited to see things moving in this direction 👍

cpitclaudel and others added 3 commits September 16, 2021 23:18

👌 IMPROVE: Add a command-line entry point in _docutils.py

46aa23f

[pre-commit.ci] auto fixes from pre-commit.com hooks

be40bf7

for more information, see https://pre-commit.ci

cpitclaudel mentioned this pull request Sep 17, 2021

Build single pages with docutils (instead of Sphinx) #420

Closed

cpitclaudel mentioned this pull request Sep 23, 2021

Adjust to breaking change in format_href in pybtex mcmtroffaes/pybtex-docutils#16

Merged

This was referenced Oct 16, 2021

Support for ..include to parse Markdown files (README.md in parent dir) readthedocs/recommonmark#191

Closed

Split docutils only functionality into separate package #347

Closed

chrisjsewell mentioned this pull request Nov 20, 2021

to_docutils() fails to produce a valid document for certain constructs #447

Closed

chrisjsewell added 2 commits December 4, 2021 19:09

Merge branch 'master' into docutils_conf

2eae526

Add CLI entry-points and help strings

25c8e7f

chrisjsewell linked an issue Dec 5, 2021 that may be closed by this pull request

Build single pages with docutils (instead of Sphinx) #420

Closed

improve

4900009

Update docutils_renderer.py

fee3f88

chrisjsewell added 3 commits December 5, 2021 09:01

Add tests

015c452

improve

6462de4

update

db79afc

chrisjsewell reviewed Dec 5, 2021

View reviewed changes

chrisjsewell added 3 commits December 5, 2021 10:19

Update test_docutils.py

3e3c2e2

Update test_docutils.py

8ffb92f

Update test_docutils.py

90a4e8f

chrisjsewell added 2 commits December 5, 2021 11:37

Add sphinx-build test for including myst in rst

30ba7d7

Add documentation

34322ce

remove delattr

f08e0e1

chrisjsewell approved these changes Dec 5, 2021

View reviewed changes

chrisjsewell changed the title ~~👌 IMPROVE: Support configuration through docutils.conf~~ ✨ NEW: Add Docutils MyST configuration and CLI Dec 5, 2021

chrisjsewell changed the title ~~✨ NEW: Add Docutils MyST configuration and CLI~~ ✨ NEW: Add Docutils MyST config and CLI Dec 5, 2021

Merge branch 'master' into docutils_conf

2782088

This was referenced Dec 6, 2021

🐛 FIX: Ensure parent files are re-built if include file changes #451

Merged

Convert single md to rst using command #441

Closed

🔧 MAINTAIN: Add publishing job for myst-docutils #456

Merged

chrisjsewell merged commit 560f641 into executablebooks:master Dec 11, 2021

cpitclaudel deleted the docutils_conf branch December 11, 2021 23:51

chrisjsewell mentioned this pull request Dec 17, 2021

to_html() output #307

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ NEW: Add Docutils MyST config and CLI #426

✨ NEW: Add Docutils MyST config and CLI #426

cpitclaudel commented Sep 17, 2021

chrisjsewell commented Sep 17, 2021

cpitclaudel commented Sep 17, 2021

cpitclaudel commented Sep 17, 2021

codecov bot commented Dec 5, 2021 •

edited

Loading

chrisjsewell commented Dec 5, 2021 •

edited

Loading

cpitclaudel commented Dec 5, 2021

cpitclaudel commented Dec 5, 2021

cpitclaudel commented Dec 5, 2021 •

edited

Loading

chrisjsewell commented Dec 5, 2021 •

edited

Loading

chrisjsewell commented Dec 5, 2021

chrisjsewell Dec 5, 2021

chrisjsewell Dec 5, 2021

cpitclaudel Dec 5, 2021

chrisjsewell Dec 5, 2021

chrisjsewell commented Dec 5, 2021

chrisjsewell commented Dec 5, 2021

chrisjsewell commented Dec 5, 2021

chrisjsewell commented Dec 6, 2021

chrisjsewell commented Dec 11, 2021

cpitclaudel commented Dec 11, 2021

✨ NEW: Add Docutils MyST config and CLI #426

✨ NEW: Add Docutils MyST config and CLI #426

Conversation

cpitclaudel commented Sep 17, 2021

chrisjsewell commented Sep 17, 2021

cpitclaudel commented Sep 17, 2021

cpitclaudel commented Sep 17, 2021

codecov bot commented Dec 5, 2021 • edited Loading

Codecov Report

chrisjsewell commented Dec 5, 2021 • edited Loading

cpitclaudel commented Dec 5, 2021

cpitclaudel commented Dec 5, 2021

cpitclaudel commented Dec 5, 2021 • edited Loading

chrisjsewell commented Dec 5, 2021 • edited Loading

chrisjsewell commented Dec 5, 2021

chrisjsewell Dec 5, 2021

Choose a reason for hiding this comment

chrisjsewell Dec 5, 2021

Choose a reason for hiding this comment

cpitclaudel Dec 5, 2021

Choose a reason for hiding this comment

chrisjsewell Dec 5, 2021

Choose a reason for hiding this comment

chrisjsewell commented Dec 5, 2021

chrisjsewell commented Dec 5, 2021

chrisjsewell commented Dec 5, 2021

chrisjsewell commented Dec 6, 2021

chrisjsewell commented Dec 11, 2021

cpitclaudel commented Dec 11, 2021

codecov bot commented Dec 5, 2021 •

edited

Loading

chrisjsewell commented Dec 5, 2021 •

edited

Loading

cpitclaudel commented Dec 5, 2021 •

edited

Loading

chrisjsewell commented Dec 5, 2021 •

edited

Loading