Proposal to sunset recommonmark in favor of MyST-Parser

[ericholscher]

Hey folks, with a sad heart I feel like it's probably time to write this update and make it official. We are planning to officially end support for recommonmark in the near future, and I wanted to give folks a chance to give us feedback on this proposal. We generally recommend everyone to switch to MyST-Parser instead -- read below for more background and a proposed timeline.

Long version

recommonmark is built on top of commonmark.py (originally CommonMark-py by @rolandshoemaker) to fill a gap that we observed in the ecosystem: many people wanted to write their Sphinx documentation using Markdown. We are happy to see that so many projects use it nowadays: there is no better confirmation that our observation was true.

However, the project has been quite inactive for the past two years (with a minor update on December 2020 to fix an annoying bug), and we have not been responsive enough to bug reports and feature requests. Besides, the landscape of Markdown on Sphinx has changed quite dramatically in recent times: out of frustration for the lack of CommonMark-compliant, extensible Markdown parsing libraries in Python, the Executable Books project has created MyST-Parser, a project that goes beyond the original goals of recommonmark, extends Markdown to add useful features present in reStructuredText, and enjoys excellent community support.

For the reasons outlined above, we would like to effectively deprecate recommonmark and encourage users to migrate their documentation to MyST-Parser. We acknowledge that this migration can take some effort, and that users might miss some features currently not available on MyST-Parser like AutoStructify, but we think it is in the best interest for the community to consolidate interest around the most active and promising alternative.

Plans and timeline

We are currently posting this to the recommonmark GitHub repo, and we'll ask for feedback through our various channels.
The goal it to make sure the transition is as seemless as possible,
so the current proposed timeline is:

• April 15, 2021: Post this issue and gather feedback
• April 30, 2021: Officially move forward with deprecation
  • Publish a transition guide with the basic steps required to migrate to myst-parser for users. We hope this should be a small adjustment, but there are some syntax differences that might take a bit of time to resolve.
  • Update the recommonmark repo README with a deprecation notice
  • Update the documentation with a deprecation notice
  • Release a new release to pypi with a deprecation notice and a deprecation notice in the code.
• By May 31, 2021: Work with the community to get documentation & references updated for recommonmark to recommend myst-parser
  • The Read the Docs documentation already did this
  • Sphinx markdown page
  • docutils beta markdown support
  • Other major references in the community that we can find

Changes for users

Of note, the published pypi packages will continue to work. We don't plan to remove them, or remove the documentation for older versions of the packages. This change will only mean that new versions of the package will not be released.

If you are happy with the current implementation, you are welcome to continue using it without support.

Feedback requested

• @lu-zero @pfultz2 @tqchen @sgenoud as the other major contributors
• @choldgraf @chrisjsewell as main developers of MyST-Parser
• @theacodes @ahopkins @LysandreJik @henryiii @Borda @cooperlees as maintainers of some high profile projects leveraging recommonmark
• @grubert as a docutils maintainer.
• @tk0miya as a sphinx maintainer

Thanks again for working with us to advance the state of markdown in Sphinx, and we hope to continue to build support around this idea with MyST.

Update: translation help

@jpmckinney posted a script in the thread to help convert:

In case it's useful to others, I had hacked together this one-time script to convert recommonmark eval_rst blocks: https://gist.github.com/jpmckinney/88c846651ab5411d1ed3a6fe222ce08f/e8f5c6febc147c512990ab3ab8809386a9b7d765

@chrisjsewell also have a repo to help: https://github.com/executablebooks/rst-to-myst

[ahopkins]

Speaking for the Sanic community, this sounds like a reasonable proposal to keep the project moving forward.

A few years ago we tried to move our documentation to Markdown because our contributors complained about RST. Recently, we completed an overhaul of the documentation to separate API docs (RST using autosphinx) from the User docs (markdown using Vuepress on our own hosted site). Therefore, from our perspective, this will not be impactful.

Thank you to the RTD team for the thoughtful approach, and good luck with MyST!

[lu-zero]

The features provided looks fairly nifty, I like the roadmap all in all.

What are the migration problems though?

[chrisjsewell]

Thanks @ericholscher. As the main maintainer of myst-parser, I am happy to help with any quaestions/improvements that would help facilitate the move 😄

Some initials notes:

myst-parser was initially written using recommonmark as a template, so a lot of the "core" translations of Markdown -> docutils/sphinx AST should be very similar.

In terms of RST to MyST conversion, I have been working on https://github.com/executablebooks/rst-to-myst, which is just about ready.

In terms of the AutoStructify, one major difference is the toctree generation (enable_auto_toc_tree, auto_toc_maxdepth, auto_toc_tree_section). Instead of converting

.. toctree::
   :maxdepth: 1

   doc1
   doc2

to

* [Title1](doc1.md)
* [Title2](doc2.md)

one would instead use:

```{toctree}
:maxdepth: 1

doc1
doc2
```

Alternatively, I am also working on https://github.com/executablebooks/sphinx-external-toc, which extracts our code in jupyter-book to define the full sitemap in a _toc.yml, into a fully fledged sphinx extension.

For the other flags:

• enable_math and enable_inline_math are available using the dollarmath extension (see here)
• enable_eval_rst: is available by default
• url_resolver: is available by default (although I should check if there are any differences)
• known_url_schemes: is available as myst_url_schemes (see here)

[tqchen]

glad that there is a path forward for the markdown story in sphinx

[chrisjsewell]

Oh and I know @grubert recently added experimental support Markdown in docutils, using this package, and I missed a response about issues with myst-parser dependencies for this use (e.g. having sphinx as a dependency).
I've opened an issue here about that: executablebooks/MyST-Parser#347

[henryiii]

Half of the projects I've "converted" so far have actually not had any md usage, just included remarkdown for historical reasons. (Partially due to issues including md into rst in the same page, IIRC). The other half were seamless, myst-parser was a drop in replacement. I don't think I have any projects that were fully md, though (might have at most one), so TOC and such are likely not to hit me. Most usage was in including a few pages like readme or changelog.

One of the bigger conversions I'm aware of would likely be LLVM, which uses remarkdown internally for documentation. I've opened a post here. https://llvm.discourse.group/t/switch-recommonmark-to-myst-parser/3259

[pfultz2]

@chrisjsewell Just curious as it wasn't clear in the documentation but does myst-parser resolve links with the :any: role by default(like what recommonmark does)? And is there a way to specify a different role for links(which is currently not possible with recommonmark)? The documentation shows how roles can be used, but doesn't mention if or how they can be used in links.

[jpmckinney]

In addition to eval_rst, MyST parser has a syntax to support Sphinx directives directly: https://myst-parser.readthedocs.io/en/latest/api/directive.html

In case it's useful to others, I had hacked together this one-time script to convert recommonmark eval_rst blocks: https://gist.github.com/jpmckinney/88c846651ab5411d1ed3a6fe222ce08f/e8f5c6febc147c512990ab3ab8809386a9b7d765

I think this might now be covered more robustly by rst-to-myst mentioned above: https://github.com/executablebooks/rst-to-myst#advanced-usage

[chrisjsewell]

Just curious as it wasn't clear in the documentation but does myst-parser resolve links with the :any: role by default(like what recommonmark does)?

MyST resolves Markdown links (e.g. [title](link)) similar to recommonmark; identifying URLS if they have a scheme like http: (and optionally only if it matches one in myst_url_schemes), then other links use a modified version of :any: that also resolves:

• Document references with extensions; [text](./doc.md)
• Document references with anchors; [text](./doc.md#target)
• Nested syntax in alternative text; [**nested bold**](reference)

(See here: https://github.com/executablebooks/MyST-Parser/blob/ba63021c6eaf447c2d77402b0f6733be94d43095/myst_parser/myst_refs.py#L82)

is there a way to specify a different role for links

Not currently. TBH no one had asked for it before, so I hadn't considered it