sphinx-build should auto-generate autodoc/summary rst files that can be customized using a Python callback

[ben-spiller]

Is your feature request related to a problem? Please describe.
I want to automatically and recursively generate documentation for my Python modules, using Python code in conf.py to customize the generated rst files for each module.

Currently there are two different (kindof competing/overlapping) mechanisms for auto-generating rsts, and both use jinja templates as the only way to customize the output. After a lot of time experimenting I found neither of them quite did the job:

• "apidoc" tool can recursively generate rst files for modules under a specific directory... but

  • it's painful having to invoke it as a separate tool (sphinx-build should do everything), and
  • the jinja templating doesn't allow any significant customization (e.g. I wanted a separate heading for each class in the module), and no way to extend using custom code in conf.py if you want to

• "autosummary" tool can generate rst files for modules (containing autodoc and/or autosummary directives), can be run during the main sphinx-build invocation using the generate option, and has more powerful templating for the contents of each module than autodoc... but:

  • it doesn't process packages recursively (so I had to write some custom code in my conf.py to create the initial .rst files),
  • doesn't follow autodoc's logic for member skipping (autosummary and autodoc-skip-member don't work well together #6798), and
  • templates don't have access to the actual object so can't customize template based on isinstance() or on directives found in the docstrings
  • still doesn't allow any Python-based customization of the rst generation process meaning there's no way to add custom functionality (e.g. or solving requirements like Autosummary extension has many deficits for proper using #4058) or work around bugs/missing features e.g. autosummary generator and templates should support module data (ie fields/constants) #6794.
  • I'd also argue that since the "autosummary" directive is about generating a table summarizing and linking to the autodoc for a set of items, architecturally it's a really weird place for the more general capability of auto-generating autodoc rsts for each module (certainly as a new sphinx user it confused me for a while that I had to use autosummary to generate autodoc!).

In the end I had to write my own custom Python code to walk the modules and generate rst in the format I wanted (piggybacking on autodoc's filter_members() and ModuleAnalyzer code and autosummary's FakeDirective mechanism for accessing autodoc Documenter classes... all pretty complex!), which isn't ideal and probably will break in future sphinx releases.

Fundamentally, Sphinx already contains some powerful logic for iterating over a hierarchy of modules/classes and filtering out private/undocumented/user-defined members, so I'm proposing we expose that excellent functionality to conf.py authors for the cases not supported by the templating mechaism.

Describe the solution you'd like
I propose we add a new extension dedicated to the task of automatically generating rts files for modules. This would be similar to autosummary_generate but:

• with a clearer name (given it's for generating autodoc and/or autosummaries, not only summaries) e.g. "autodocgen", "automodulegen"
• with automatic recursively walking of (one or more) modules
• with a callback so you can customize the generated .rst content using Python code, in case the textual templates aren't sufficient.

The callback for generating .rst content would need a way to access Sphinx's functionality for enumerating, categorizing, filtering and extracting the docstring for modules, classes, etc (basically a cleaner version of what autosummary already does).

Could look like this:

def generate_module_rst(app, moduledoc: Documentable):
   # default implementation would be to delegate to a template (using the fields from moduledoc) similar to autosummary, but users could write custom Python via a config callable if they want to
  # user can return None if they wish to skip rst documentation for this module - that provides an easy way to implement arbitrary skipping rules

class Documentable:
   """Holds information about a documentable item such as a module, class, function, data. This is essentially a thin public API wrapper around the autodoc Documenter classes that exposes everything from Sphinx that someone might possibly want to use when generating rst content."""
   kind # function, class, module, data, etc
   docstring # because getting this for data/attribute is very hard and Sphinx can already do it
   qualname # fully qualified name
   obj # instance of object to be documented
   app # so we can get configuration info
  __documenter # private field holding the non-public autodoc XXXDocumenter class, which is needed for getting the members

   def get_members(self, filter=True): # get the members of this item e.g. for a module the (sub)modules/classes/data etc it contains
    # filter can be used to toggle autodoc's filtering of private/undoc/skip-member items

All the hard bits are already implemented in Sphinx... but hard to get at. The proposal is really just about some fairly modest wrappers to expose all that goodness to the person writing the conf.py file. I didn't find any other proposal that solves it in such a neat way, and gives full control to the person writing the conf.py file (without making them also reimplement things Sphinbx can already do).

This capability would provide a superset of what apidoc/autosummary_generate can do, and those could be easily implemented in terms of the more powerful primitives suggested here. By having some common code whether you want to generate autoXXX directives or autosummary directives, we avoid the need to keep dealing with bugs and feature requests from the diverging autodoc+autosummary implementations.

Describe alternatives you've considered
I considered sketching an implementation out with a PR but didn't have time (at least right now), and thought it'd be best to float the general idea with you guys before investing any time.

I see lots of issues, stackoverflow posts and third-party sphinx extensions around this topic so think it could really help the sphinx community, and maybe provide a neater and more potent architectural solution to various other PRs and bugs underway e.g.

• if people really thought that autosummary is the best place for general-purpose module-level rst generation capability, we could add a Python callback there with some refactoring
• Sphinx apidoc extension #4101 proposes doing apidoc generation from sphinx-build; but I reckon the autosummary implementation code is a better thing to base this on than apidoc as it's more powerful
• The https://autoapi.readthedocs.io/ extension does the automatic part well, but forces you to edit your files to specify all/api directives everywhere which isn't always practical for an existing project, and no way to exclude modules, and still no Python-based customization of rst output
• Waiting for review: Add get_members function to apidoc templates (better_apidoc backport) #6768 suggests adding a get_members() function I think woudl be addressed by the proposal above, but on its own would be a fair bit less powerful overall - still leaves you stuck if the the sandboxed world of jinja isn't powerful enough for what you want to do
• Autosummary extension has many deficits for proper using #4058 proposed adding lots and lots of new jinja options... again the above proposal gives someone with that requirement the power to implement it themselves, without the need for sphinx itself to implement and test so much extra functionality
• I wondered about making a custom extension for this... which could be done but then lots of newbies might not find it, so better in sphinx itself... also since this is all about making use of internal/non-public functionality in autodoc it'd be less likely to break in future sphinx versions if it were itself part of sphinx

[tk0miya]

Thank you for proposal. I agree python-related feature of Sphinx is very complicated and confusable. And I still don't understand perfectly how they works. In addition, unfortunately, many extensions depends on current implementation of these modules. So we need to keep interfaces of them to keep compatibility. As a result, IMO, we need to scrap auto* modules and build new one like sphinx.ext.autodoc2.

[goerz]

I would think that #6768 should pretty much do whatever you need to do (my needs are very similar to yours). Of course, you're right that ultimately, nothing is more flexible than complete custom python code, but the (Python-implemented) get_members function in combination with all of Jinja's templating features is extremely flexible already. It will definitely allow you to categorize members. Also, get_members is able to process lists that you define at a module level. The __all__ list is standard, but personally I tend to also have a __private__ list, and you can define your own lists for categorization beyond what get_members can auto-detect.

[goerz]

There is also sphinxcontrib-apidoc which wraps apidoc from conf.py. Once #6768 is merged, I'll create a PR for that project to add explicit support for the templating flags.