Autodoc should output generated rst

[amueller]

When using autodoc or autosummary, the generated rst is never saved as far as I can tell.
So if you go to a page that was generated with autosummary, like this:
http://scikit-learn.org/dev/modules/generated/sklearn.cluster.DBSCAN.html#sklearn.cluster.DBSCAN

and you click "show this page's source" you'll just get the autosummary template:
http://scikit-learn.org/dev/_sources/modules/generated/sklearn.cluster.DBSCAN.txt

That's not very helpful for debugging. I think storing the generated RST would be more helpful, possibly optionally?

[tk0miya]

+1
But I'm not good at autodoc and autosummary. So PR are always welcome.

FYI: You can use -vvv option of sphinx-build to debug.

[0x2b3bfa0]

As per this Stack Overflow answer, it would be nearly impossible to write these files with a custom builder and writer given the nature of document trees:

There is none, and it is hard, if not impossible to implement one beyond trivial ReST markup.

ReST roles and directives may execute arbitrary code at parse time. Particularly a role or directive can create and insert nodes of arbitrary types with arbitrary content into the document tree. Thus there is no direct mapping between a document tree and ReST source code and consequently it is impossible to obtain the original ReST source code – or at least something that comes close – for an arbitrary document tree.

A real world example are various directives from Sphinx, e.g. py:class. These directives insert pending_xref nodes into the document tree which are resolved into real cross-references at a later point of time. However, there is no single directive that corresponds to a pending_xref node thus there is no ReST source that directly corresponds to a document tree containing such nodes.

One can implement a writer for simple standard ReST markup, i.e. headlines, paragraphs, emphasis, and the like. I do not know of any implementation but that is trivial to do on yourself. That may or may not be enough for your purposes.

If your documents contain roles, directives or substitutions complete ReST-to-ReST transformations are impossible. However, you can sort-of cheat by unregistering all roles and directives first, and then registering a function that catches all roles and directives and preserves them literally in the document tree. Based on such a tree, you can restore the source (or at least get close to this). Substitutions however are lost, because these are applied at an early stage of parsing and do not appear in the resulting tree.

Nevertheless, in this particular case, it would be possible to hook this line and save the generated reStructuredText as a file before it gets parsed:

sphinx/sphinx/ext/autodoc/directive.py

Line 156 in d635d94

logger.debug('[autodoc] output:\n%s', '\n'.join(params.result))

Users could be able to provide an output path from conf.py by using the autodoc_default_options dictionary:

autodoc_default_options = {
    'output_path': './something'
}

Then, we could retreve the provided output path and the file contents from directive.py through the configuration attributes:

output = self.config.autodoc_default_options.get('output_path')
contents = '\n'.join(params.result)

The missing piece, would be finding appropriate names for each output file.

[0x2b3bfa0]

Another less hacky alternative could be separating autodoc from the build process or, at least, offering a command line option with that functionality.