Plugin API.

[erichulser]

Hey Tom,

I've worked on my own documentation generator for a while and used that but am in the process of upgrading it, so I wanted to see what was out there besides Sphinx (which is way to bloated). I came across this project and I really like the simplicity of it. One of the requirements I have for documentation is auto-module/class docs as a means to augment the prose style.

I know you've already stated that you have no interest in that, but I was wondering if you'd be interested in allowing me to work with you on this project. I have all the code for generating docs that output a format like the ones used in Qt (http://qt-project.org/doc/qt-4.8/qdialog.html) from the python docs. I would propose simply adding a plugin architecture to mkdocs (similar to django/pyramid) that would allow a developer to run additional addons so as to not pollute the core of the project. Something like:

site_name: my_site
extensions:
       mkdocs_autodoc
pages:
       - ['index.md', 'Introduction']
       - ['user-guide/creating.md', 'User Guide', 'Creating a new Marshmellow project']
       - ['autodoc:my_project', 'API', 'My Project']

Where there would be a render hook that would dynamically generate the content from the my_project package as markdown files that could be then used and referenced from the prose style. I would be happy to work with you on the structure and provide the work I've been doing for the auto-class documentation as a plugin.

Please let me know if this is something that you'd be interested in discussing, otherwise I may end up forking this project and implementing that as an option.

[d0ugal]

So, I'm generally +1 on the 'plugin architecture' idea. It's something I have been pondering a bit.

[erichulser]

Awesome, is that something that you guys have a plan/roadmap for? Do you need help implementing?

[d0ugal]

I'll probably have to deferr to @tomchristie for this.

However, it is something that I do think it a good idea and I would like to see. To make MkDocs work for more people and in more situations we should allow them to be able to extend it. This does however go against a few of the goals:

• We want to create standalone installers for different platforms (like Windows) so users don't need to know they are using Python necessarily. Making a plugin architecture work in this situation would be tricky.
• So far we have tried to avoid exposing any form of Python API. The closest we have is markdown_extensions in the config.

I'm sure I had a third in mind but I can't think of it now.

[tomchristie]

Certainly be interesting to see someone work on this, but I'd be wary of pulling anything in when there's still a bunch of work ahead in order to get to a solid 1.0 release.

What would be most likely to convince me on something like this would be documentation pull requests that demonstrate the API design work, and show exactly how these plugins would work from a user perspective.

Personally I think we're probably still too early in the day to start thinking about maintaince-increasing features like a plugin architecture, but that's not to say that I wouldn't be interested in seeing how it might be tackled.

[d0ugal]

I think we can work towards it by defining the build pipeline better and exposing API's to use internally (at first). Internal only use means we don't need to worry even slightly about supporting external plugins until it settles.

I have been working on support for local searching which requires a few things: The search index needs to be created during the page builds, I need to figure out if the theme supports it (by looking for the js lib, I think, not sure yet) and probably some other stuff.

To make this work I need to hook into the process in a number of places, doing this manually is a bit nasty and each section is slowly getting more complicated.

tl;dr - improving the abractions and build processes internally would make it much easier to support a plugin api and add other features we do want to support in core.

[tomchristie]

All sounds reasonable.

[SvenDowideit]

I would use a plugin API that allows us to add pre and post process steps, for brute force changes (though given the experience we have with TWiki/Foswiki, I would also suggest you don't do this.

mmmm, actually, the safest thing you could add, would be a mechanism that allows the registration of a tag ie some begin and end marker, and a class which will be given only the pre-parsed sub-section of markup that applies to it.

contextless markup processing causes much less in the way of lock in - and if you use a pre-built parse tree (ish), you're not totally tied to the original formatting (and can validate too)

but it does depend on the cleverness of the parser you use - regexp based ones are generally less robust.

[SvenDowideit]

moby/moby#8976 (comment) is one example, where we really would prefer to have one section of information (what kernel versions and features are needed) be transcluded into many documents. Ideally, I'd use a foswiki style %INCLUDE{"topicname" section="paragraph identifier"}% - so the original maintained text is inline in an existing document.

[erichulser]

I'm not sure if you want me to make a pull request for this, I took a first pass stab at the API architecture + some extensions (autodocumentation + tree heirarchy)

https://github.com/ProjexSoftware/mkdocs

The new *.md files are the result of the mkdocs.contrib.autodoc extension:

mkdocs autodoc-generate <module> ..

The mkdocs.yml updated to include the extensions:

mkdocs_extensions:
    - mkdocs.contrib.autodoc
    - mkdocs.contrib.tree
pages:
    ...
    ['tree:api/mkdocs', 'API', 'mkdocs']

The documentation for the usage of the extensions is within the extensions themselves.

The API is just an event system (defined in mkdocs.events) -- you can define any number of types of events and have extensions listening for them. I wanted to be as minimally invasive to the base code as possible, I don't care if the code changes for the API, but the overall functionality as it served the purpose I was looking for which is automatic code documentation.

[d0ugal]

Woah, that's a huge code dump. It looks interesting, but I have a few questions/concerns.

• I'm not sure we want to include any builtin extensions. You have added two to contrib - autodoc and tree. Initially at least I'd like to see just a plugin system that allows these to live externally.
• Is the autodoc code based on another library or is this something you have just written for MkDocs? Thinking of licensing stuff here.
• We absolutely don't want to include auto-generated docs for MkDocs. Doing so would be a promise to support a Python API.
• Obviously we still need tests.

Personally I'd be much more interested in autodoc functionality that enabled me to write prose. I'm not sure how this would work, but I'd expect markers or something in the Markdown that then pull in the relevant functions. So, rather that dumplicating the information in my docs, i write a paragraph and add that will then find and include the method dignature for me. I'm still thinking this idea through tho'

[d0ugal]

For this to potentially work, we need to split up the dicsussion. A pull request for the extension API addition and then a pull request each for the potentially included extensions.

[tomchristie]

I'm not sure we want to include any builtin extensions.

I'm sure. We don't. :)

The only thing pulling third party extensions into the core project does is mean that it's suddenly our job to maintain them, instead of the author's. That would suck.

A pull request for the extension API addition and then a pull request each for the potentially included extensions.

Just the first of those two. Of course we do need to also see something built against that API in order to validate it, but not as a pull request into this project.

If MkDocs becomes well adopted, we need to keep the project absolutely as minimal as possible. Focusing on a core, well documented, minimal product, has to be the overriding design choice with anything like this.

[erichulser]

Sure, understood.

I cleaned it all up so now only the changes to the core exist in that project and I created a pull request for it.

The actual implementation for the extensions I moved to my own repo as public projects:

https://github.com/erichulser/mkdocs_autodoc
https://github.com/erichulser/mkdocs_tree

pip install mkdocs_autodoc
pip install mkdocs_tree

Works the same way as before, just now they are their own repos.

I wrote up a little explanation of the code changes in the pull request that made these possible, and can easily modify as needed once you decide upon a good/acceptable API structure.

Is the autodoc code based on another library or is this something you have just written for MkDocs? Thinking of licensing stuff here.

This is all my code, but I would give it as a mkdocs extension vs. its own project (as it is now). I'm hoping to use this as a replacement of my own documentation system. I had the opposite need, so I'd written this API documentation generation and am now starting to need more prose based support. I found your project through readthedocs and it seemed like we could each benefit from the other, so I'm trying to deprecate my system in favor of adding the auto-documentation as a plugin.

We absolutely don't want to include auto-generated docs for MkDocs. Doing so would be a promise to support a Python API.

Yea, no worries -- that was really more just an example of the output.

Personally I'd be much more interested in autodoc functionality that enabled me to write prose. I'm > not sure how this would work, but I'd expect markers or something in the Markdown that then pull > in the relevant functions.

I can work on things like that -- I've had the opposite need as I'm working on documenting API calls where I don't necessarily expect or want people to have to jump into the code vs. something like this which is more functional usage. The idea being I start with prose, but when I say a "XYZ" class is returned, they can click the link and see what properties are available once they have that object, similar to how Qt's documentation is setup.

[waylan]

@d0ugal stated above:

Personally I'd be much more interested in autodoc functionality that enabled me to write prose. I'm not sure how this would work, but I'd expect markers or something in the Markdown that then pull in the relevant functions. So, rather that dumplicating the information in my docs, i write a paragraph and add that will then find and include the method dignature for me. I'm still thinking this idea through tho'

Just an FYI, I created a proof-of-concept Markdown Extension that does just that: mddocs. No mkdocs Plugin API required. Just add mddocs to the list of markdown_extensions. It is currently experimental and I wouldn't use it for anything serious until is stabilizes -- if it ever does (I'm not sure it is even a good idea yet).

I mention it here because I think most, if not all, of the proposed reasons for including a Plugin API to mkdocs can be handled by building specific Markdown Extensions. For example, Pelican (a more complex sorta competitor to mkdocs) offers a plugin API and many of their plugins are re-implementing things that Markdown's Extension API already makes possible. The one exception is perhaps related to #144. There would need to be a way to pass some context variables into the Markdown Extension to make them available within the Markdown parser to swap out a placeholder of some kind. It would be easily doable with a Markdown Extension that shipped with mkdocs (perhaps defined in the same namespace as the context), but near impossible with a third party extension without a Plugin API. I didn't look too closely, but does the proposed Plugin API even allow for this use case?

[waylan]

@petri perhaps. However, we already fully support Python-Markdown's extensions which would be used to alter the Markdown parser's behavior. No need for the Plugin API to do that. However, if this requires the existing Markdown parser to be replaced with a different parser, then no, this is not a planned feature of the API (see this comment).

[honzajavorek]

I'd like to make some programmatic changes (like auto-interlinking with GitHub issues, auto-generating docs for CLI options, etc.) to my docs before they get published. I can write a script to make these changes, but then mkdocs serve will show me different docs then what will get published. Also, I need to get it published to ReatTheDocs - I'm not the one to build the production docs, they do it in their build process. So I probably either need to extend their build process somehow (how? they don't mention this would be possible with mkdocs) or I need to ensure my script gets picked up by standard mkdocs build, which they probably use.

Since the plugin API doesn't seem to be ready yet, I wanted to ask whether there's any kind of workaround. I'm pretty fluent in Python, but I didn't find any Python API I could hook anything into.

[waylan]

@honzajavorek there are two ways others have accomplished that. Either run scripts before or after mkdocs build. Scripts run before add additional Markdown documents to the docs_dir and scripts run after add HTML files to the site_dir. Which works better for you depends on your specific needs.

That said, you're not likely to get anything to work with the ReadTheDocs service. As I understand it, they are stuck on MkDocs version 0.14 and their build runs on their servers. In other words, you can't modify their build process or build locally before pushing up to their servers.

[honzajavorek]

@waylan Thanks for the super-fast and super-useful reply! In that case, I'm probably left either with moving away from RTD (not an easy option at this moment) or switching to Sphinx 😢 (since as far as I understand it, Sphinx should be extensible and RTD should respect that).

[waylan]

FYI, I have rebased (a copy of) the plugin branch onto the 1.0.dev branch. You can see it here: 1.0.dev...waylan:plugin1.0

[waylan]

FYI, I have created a PR for the Plugin API. You can review it in #1223. Any and all feedback welcome.

[kylef]

@petri

@waylan, api blueprint has its own markdown-based syntax. so maybe @oamado meant to ask whether the coming plugin api would make it possible to create a plugin to render api blueprint format using mkdocs...?

I've made a Pygments Lexer for API Blueprint (https://github.com/kylef/pygments-apiblueprint) which can be used with mkdocs (see example: apiaryio/dredd#779).