set up Sphinx autodoc to put our docstrings in our reference docs.

[gilch]

There is considerable duplication of documentation between the reStructuredText docs and Hy's internal docstrings. I propose we move as much as reasonably possible into the docstrings themselves, to make the references more accessible at the repl, and for language tools like emacs. @jkitchin has started this effort with #1040.

I think Sphinx can fetch docstrings programmatically from Python objects rather than parsing them from text. This means that it could theoretically work for Hy docstrings as well, though I've never seen this done before.

Our documentation effort is lagging behind Hy's actual capabilities. Part of this is due to the extra effort required to document things in two places, and then keep them in sync afterwards.

The new setup will require considerable reorganization of the documentation. For each function (and potentially class, module etc.) we'll have to take the better of the two or combine them somehow.

Our testing is similarly lagging behind. While Hy's doctests shouldn't completely replace unit tests (the priority must be understandable examples, rather than completeness), it is better than the nothing we sometimes get. More importantly, making our reference examples into doctests in our docstrings will ensure that they actually work as intended. Currently, many of our example docs are broken. They should all be automatically testable, even those from the tutorial.

Doctests show a Python repl session and don't currently work with a Hy repl, but we need our Hy examples to be in Hy, not Python. See #1019.

Hy lacks Python's (raw) triple quotes. Hy's strings can certainly represent any possible text given enough backslashes, but this will make it difficult to write reStructuredText (which also needs backslash escapes) and Hy doctests (which will certainly use quoted strings themselves) in our docstrings. I've proposed adding heredocs as string literals to the reader (#1042) to resolve this.

[jkitchin]

I am not sure you can get the autodoc feature of Sphinx to work with hy the way it does with Python. There is a lot of redirection between hy functions/macros and where the doc strings are. For example, here https://github.com/jkitchin/hyve/blob/master/hylp.hy#L103 I expanded some code you posted earlier to get the Python object for a given hy symbol, so that I can use the python tools for accessing docstrings, source files and linenumbers. I am pretty sure Sphinx would see things from the Python side in using the autodoc features.

On the other hand it is not that difficult to write code that would generate html docs from these docstrings. pydoc already has a lot of this framework. So there probably could be some code in the Sphinx docs that generates documentation for the hy functions and macros.

The benefit of this approach would be getting things the way you want them, e.g. sorted, or separated by categories, etc... The downside is more than one line of .. autodoc: ...

[gilch]

I'd have to re-read the Sphinx documentation. We might have to experiment to find the best approach.

I expanded some code you posted earlier

I used the or/false short-circuiting trick to avoid evaluation. You don't need it in your macro (you evaluate sym anyway for do-import), so It could be simplified:

(defmacro get-python-object [sym]
  "Get the Python object for the symbol SYM.
  SYM is a function or module.
  If SYM has a . in it, import the base module, unless the dot is the first character. We can't get these yet."
  `(do
     (do-import ~sym)
     (try
       (->> ~sym
            (.get hy.compiler._compile_table '~sym)
            (.get (get hy.macros._hy_macros nil) '~sym)
            (.get (get hy.macros._hy_macros "__main__") '~sym)
            (.get (get hy.macros._hy_macros "__console__") '~sym)
            (.get (vars hy.core.shadow) '~sym)
            (.get (vars hy.core.language) '~sym))
       (except [e NameError] None)))

[gilch]

I tried out autodoc a bit. It can get us surprisingly far, but it chokes on function names that are not valid Python identifiers. Sphinx is supposed to be pretty extensible though. We might be able to implement our own extension for Hy files. This would also be useful tooling for third-party Hy users to have.

[allison-casey]

My Hy domain sphinx extension has autodoc working for almost everything we'd need for hy's core library and I've started the overhaul that would close this issue. It's going to be a massive pull request though since it'll touch the docstrings of almost every hy builtin, contrib, and extras modules. Do y'all have any guidance for how you'd want to approach the pull request? it's just a tad bigger than any pull request I've done for hy before lol. Also any requests for things y'all want for the hy docs while I'm at it? So far I have a Cheatsheet.rst with categories and links to each builtin/contrib method and a unified API.rst with autodoc'd methods for hy.core.language, macros, bootstrap, and shadow. Lastly, I'm using sphinx.ext.napoleon to make the docstrings easier to write and follow a consistent format. Are we okay with this?

Here's what the docstring for the when macro look like now.

(defmacro with [args &rest body]
  "Wrap execution of `body` within a context manager given as bracket `args`.

  ``with`` is used to wrap the execution of a block within a context manager. The
  context manager can then set up the local system and tear it down in a controlled
  manner. The archetypical example of using ``with`` is when processing files.
  ``with`` can bind context to an argument or ignore it completely, as shown below:

  Examples:
    >>> (with [arg (expr)] block)
    >>> (with [(expr)] block)
    >>> (with [arg (expr) (expr)] block)

    The following example will open the ``NEWS`` file and print its content to the
    screen. The file is automatically closed after it has been processed.

    >>> (with [f (open \"NEWS\")] (print (.read f)))

    ``with`` returns the value of its last form, unless it suppresses an exception
    (because the context manager's ``__exit__`` method returned true), in which
    case it returns ``None``. So, the previous example could also be written

    >>> (print (with [f (open \"NEWS\")] (.read f)))

    Shorthand for nested with* loops:

    >>> (with [x foo y bar] baz)
    (with* [x foo]
      (with* [y bar]
        baz)).
  "
  (_with 'with* args body))

[Kodiologist]

Do y'all have any guidance for how you'd want to approach the pull request

Just make sure the massive copy-paste from the docs files into docstrings are in a separate commit from the more interesting code or documentation changes. My philosophy is that changes that create a lot diff lines, but don't involve any real programming that a reader would want to review, should be separated, by commits, from the real programming.

I haven't used the napoleon syntax before, but I think it's fine so long as it supports all the reST constructs that we need.

[allison-casey]

Just make sure the massive copy-paste from the docs files into docstrings are in a separate commit from the more interesting code or documentation changes. My philosophy is that changes that create a lot diff lines, but don't involve any real programming that a reader would want to review, should be separated, by commits, from the real programming.

A lot of the heavy lifting is done by hydomain so I don't think there should be many code changes 🤞 I'll make sure any are in their own commits though

I haven't used the napoleon syntax before, but I think it's fine so long as it supports all the reST constructs that we need.

I've used Napoleon (Google style docstrings) for pretty much every python project I work on and have found it much easier to navigate than reST style. You can still use all the regular reST constructs if you want, it just makes certain things easier to write and read. example