Python-Markdown 3.0 Road Map

[waylan]

This is the current plan to work towards a 3.0 release. This is a "living document" and will be edited over time. Please leave comments below, but be aware that older comments may not apply to the current state of this document. The Roapmap can now be found here.

1. Create a new Organization on GitHub for "Python-Markdown" and fork waylan/Python-Markdown to Python-Markdown/markdown. All work on 3.0 will happen at this new repository. The 2.x series will remain at waylan/Python-Markdown, which will be in "maintenance mode" and only receive bug fixes (2.6.x) going forward.
2. Each built-in extension will be broken out into a separate package at Python-Markdown/<package name> and, upon release, uploaded to PyPI as a separate package. Users can do pip install extensionname for each extension they want. For that matter, "extra" will simply be a simple wrapper and a list of requirements (the various individual extensions) which pip would install automatically. A few changes to the testing framework will be required to specify different locations of tests for each extension.

This will result in multiple benefits:

• This would eliminate the current movement toward the lengthy string-based-names (markdown.extensions.extra) for the built-in extensions.
• This will make it clear to third party extension authors how they should expect users to install their extensions. No special cases for anyone.
• Some of the existing built-in extensions could be passed on to other people for future maintenance. With a GitHub org, this is easy and the repo doesn't even need to move.
• Some third-party repos could even be hosted within the org. A set of requirements would likely need to be met before inclusion. The list of requirements would need to be created first.
• All extensions can be on their own release schedules and do not need to wait for the next release of Markdown. This should drastically slow down the development of Markdown itself as most feature additions have been for extensions. A much slower moving target will make it easier for extension authors to maintain their extensions against the API.
• This will make it easier for an existing extension to be forked when a user wants slightly different behavior.
• Third party extensions will be able to use the updated testing framework to run tests for their extensions.

3. All of the existing deprecation paths would be fast-tracked. The changes would hit hard with no warnings in 3.0. As no backward comparability considerations would be taken, various little bits and pieces of the API would be cleaned up. For example:

• InlinePatterns => inlineprocessors (which match all other types)
• Some keywords on markdown.Markdown might be moved to extensions??
• Some processor types currently use self.md and others self.markdown to refer to the Markdown instance. This will be normalized to self.md.
• Remove the md_globals keyword from the extension API. Years ago, state and/or settings were stored in module level globals. We no longer use globals (aside from a few things in markdown.utils); everything is now stored on the class instance.
• Update attr_list parser to use shlex lib. Unicode support was added in 2.7.3
• ??

4. The changes to the various blockprocessors described in Support fenced code block within lists and blockquotes #53 will be implemented (the test method of blockproccesors will be dropped). This is probably the largest body of work and must be completed before 3.0-Final is completed.
5. An effort should be made to improve the inline handling code. In addition to the renaming (mentioned in 3 above), a better solution should be explored than the current placeholder madness to allow proper nesting. Most likely, such a solution would also address the issue raised in Render time is superlinear in the size of a paragraph #161. I would love to remove the requirement that all inline regex patterns be wrapped by (.*) both before and after the supplied regex before compiling.
6. Given the current state of Python package management (much better than in years past), the current policy of no third-party dependencies will be re-examined. Of course each case will need to be weighed carefully. Some specific things that come to mind:

• Explore using Setuptools for setup.py.
• Explore using Setuptools' "entrypoints" in place of the existing string-based-names for extensions.
• Explore using a proper HTML parser for handling raw HTML.
• A third party documentation generator will mostly likely be adopted (see Use the mkdocs package to generate documentation #369).
• Explore alternatives to our homegrown OrderedDict (the standard lib one won't work as it only allows adding items to the end).

[mitya57]

Wow, that looks like a very big plan!

What namespace are you going to use for extensions names? If markdown.extensions.foo is too long (as you say), and foo is too generic (and may be already taken)?

[waylan]

@mitya57 that's why I suggested using Setuptools' "entrypoints" (second item in 6). See this for a basic explanation of entrypoints. You can see them being used here.

My thought is that we lose all of the existing support for string-based-names and only support passing in instances of Extension subclasses. Then, if an extension author would like to make his/her extension available via a string, they need to use entrypoints. Essentially, upon installation, the extension would be registered as being available with a given string as its name. Therefore 'foo' would work fine, even if the actual Extension class was foobar.bar.baz.FooExtension. Then the user could pass in 'foo' and get the extension. We completely eliminate Python dot notation which is not usually understood by Python noobs and/or document authors.

[waylan]

Oh and one other thing, by using entrypoints, someone could potentially replace one extension with another. For example, if you have both foo.bar.LousyExtension and bar.baz.GreatExtension, just register the extension you want with the name you want and it doesn't matter that the other extension is still installed on your system. And as the actual namespace is irrelevant, they can be in completely different namespaces which will be completely transparent to the enduser.

That said, I'm thinking that the built-in extensions will probably all be installed at the root namespace and with "mdx_" prefix because that's what most third-party extensions already do. So, without entrypoints you would need to do:

from mdx_extra import ExtraExtension

On the other hand, that does put a lot of additional packages in the root namespace.

[facelessuser]

I think splitting the extensions out is a generally a good idea. Now Python Markdown out of the box will parse Markdown. All the other bells and whistles will be distinguished as literal extras. It will also encourage people to develop there code in separate extensions as the previous extras are no longer shipped with Python Markdown. There would no longer be a gain for anyone trying to push their extension objectives into the previously built-in ones.

Entrypoints is also an interesting approach as well. I remember taking a look at how Pygments included its modules as they use entrypoints as well..

I'll be interested to see how number 4 turns out as I started looking into that, but wasn't too excited when I realized how the current lists handled blocks. That's going to be some work.

[facelessuser]

Thinking about it more, I will say though, that having the default extensions in one place opposed to split into individual repos makes it easier to see how the most commonly used extensions interact with each other. Currently, you know that those are the one's that most people use, and you can then code your plugins to insert around them accordingly and since the order of extension inclusions is extremely important. Assuming the default extensions will be under the PythonMarkdown group, it can probably still be assumed that those will be the most commonly used, but having them broken up will make it a pain to pull all the projects to see the interaction between them. I do understand the benefit of not having to install the one's you don't use though. I'm not really suggesting anything here, just mentioning I guess the one benefit of having the defaults bundled together.

[ryneeverett]

Some third-party repos could even be hosted within the org. A set of requirements would likely need to be met before inclusion. The list of requirements would need to be created first.

Following up on @facelessuser's thoughts, it seems like one obvious requirement will be rigorous compatibility testing with all other extensions.

I am curious where the extension tests will live though. Compatibility tests needs to be independent of the extension repos to avoid not only duplication, but commits to one extension breaking the tests of another extension without even realizing it.

I would suggest keeping a lot of the extension testing infrastructure in the core repo, but to move it into the markdown namespace. Each extension (and also third party extensions) could inherit from markdown.TestExtensionClass. This base class would not only provide the testing infrastructure as it currently does, but could also run the compatibility tests in the core repo alongside the individual extension tests.

[waylan]

@ryneeverett & @facelessuser you make some good points. I had given testing some consideration. We already have tests which simply pass known good plain Markdown through some of the built-in extensions (the ones that add features not change features) -- simply to make sure they don't break anything. I expect that those tests would move with the extension. Also, as we already use tox (which is very good at ensuring all required test dependencies are installed), we can test the more common extensions with the core to ensure that changes to the core don't break the extensions.

The one thing I hadn't considered was the tests regarding the various extensions interacting with each other. Obviously, we could also use tox to install dependencies so that we can run tests for one extension with another extension. However, that means that the extensions need to run tests of their own independent from the core. This will require a little extra work on the testing framework. The benefit is that the testing framework will then also be usable by third party extensions. We'll need to add a way to specify which directory the framework looks for tests in (currently it assumes the directory that the framework itself is in).

Thanks for the feedback. It is much appreciated.

[waylan]

FYI, I've created the fork at https://github.com/Python-Markdown/markdown and have been slowly working on this. Feedback is certainly welcome. I've completed the following tasks so far:

• Remove all deprecated code in core
• Refactor testing framework to be usable by third party extensions
• Load string based extensions via entry points (I really like this)
• Use Setuptools (for all features of the setup script except build_docs)

[waylan]

The Roadmap is now on the wiki and this issue is being closed. All future discussion should happen on the new repo at https://github.com/Python-Markdown/markdown.