wip: MkDocs

[pawamoy]

It's a draft, but you can already review if you'd like 🙂
To preview the docs locally:

cd qtile
git checkout mkdocs
python -m venv .venv
. .venv/bin/activate
python -m pip install mkdocs black griffe-inherited-docstrings
python -m pip install $(mkdocs get-deps)
PYTHONPATH=. python -m mkdocs serve

If you want to speed up the reloading, you can disable mkdocstrings temporarily (which takes up most of the build time) with this:

MKDOCSTRINGS=false PYTHONPATH=. python -m mkdocs serve

Preview online, on my fork's pages: https://pawamoy.github.io/qtile/

---

1. The first step I'm taking is converting from reStructuredText to Markdown, both in the docs and in the sources (docstrings). The docs layout stays unchanged, although the URLs are now "directory" URLs (not ending with index.html or .html). We can keep old URLs if needed, or setup redirections to the new ones (later in the process).
2. The second step will be polishing the API reference rendering. That will probably involve fixing or improving docstrings so that they follow one particular style (Google, Numpydoc, or crazy one: PEP 727). I personally prefer and recommand the Google-style, since it's the only one markup-agnositc (and therefore naturally plays well with Markdown docstrings). Current docstrings seem to use the Numpydoc-style, though I didn't review the whole code base. Polishing the API will also possibly involve a few additions to the source code, such as declaration of __all__ variables.
3. The third step will be polishing the docs layout, if that's something we want. Since the theme is different (Material for MkDocs instead of ReadTheDocs), the navigation renders differently (pages on the left, headings on the right), and the current layout might not be super adapted to this new way of rendering navigation. We can also consider keeping the ReadTheDocs theme, which is a built-in theme of MkDocs, though way less supported by mkdocstrings (in short: Material pretty API docs, RTD not pretty). Material for MkDocs also has many useful features. I invite you to check their docs: https://squidfunk.github.io/mkdocs-material/.

If you serve the docs locally, or navigate the online docs, you'll notice some rendering issues and broken links: that's normal, I'll fix everything progressively. Once we're out of draft though, please mention these issues in reviews. I'll only move out of draft once I'm confident the docs are "correct" (no broken links, no rendering issues, no missing contents).

---

Tasks

• Setup MkDocs
• Convert reStructuredText documents to Markdown documents
• Convert reStructuredText + Sphinx-style/Numpydoc-style docstrings to Markdown + Google-style docstrings
• Translate custom Sphinx directives to Markdown equivalent (markdown-exec/pymdownx blocks)
• Rework docs layout (navigation)
• Setup redirections from old pages to new pages
• Update docs-related contributing documentation
• Find kind reviewers willing to review this humongous PR
• Finish cleaning up (remove any Sphinx-related content left)

[elParaguayo]

How do I go about viewing these docs?

[pawamoy]

@elParaguayo thanks for asking! I updated the PR description 🙂

[ramnes]

Looks really nice, thank you for tackling this!

I think the layout page would heavily benefit from having the layouts listed on the left. And it currently shows layout methods that aren't really useful to end users while ignoring other methods that are really important. For example, see Bsp and its example, and then the methods listed:

Just curious, what's your plan for the directives such as qtile_graph and qtile_commands?

Note to myself: drop this devilish text format used in the changelog and move to Markdown as well, that will be much easier to read in the docs

[elParaguayo]

If mkdocs has an equivalent for custom directives, I can look at migrating those two.

[pawamoy]

I think the layout page would heavily benefit from having the layouts listed on the left.

Completely agree. I like the navigation in the current docs: you see all available layouts or widgets in the left sidebar, without noise like methods, etc.

And it currently shows layout methods that aren't really useful to end users while ignoring other methods that are really important.

This draft basically renders all documented attributes and methods (inherited as well) of these layout classes, but it's possible to choose exactly which one to render. However, I'm surprised by "ignoring other methods": if they do not show up, it means that they are not documented (they don't have docstrings). Maybe they exist on the base layout class, and are overridden in the child classes? It's possible that Sphinx picks up the parent docstring (that's what inspect.getdoc does, it looks up in the MRO). We have an extension for this: griffe-inherited-docstrings. It's available to sponsors only, but should be soon available to everyone, once the $500/month funding goal is reached (I use a sponsorware strategy called Insiders for my open source projects).

Just curious, what's your plan for the directives such as qtile_graph and qtile_commands?
If mkdocs has an equivalent for custom directives, I can look at migrating those two.

We have multiple options.

• If we really want to go the "directive equivalent" route, we could use PyMDown Blocks. Blocks are a way to create custom "directives". They all share the same syntax, and can be nested. They would look like this:

  /// qtile_graph | core
  ///
  
  /// qtile_commands | libqtile.backend.x11.core
      object-node: core
      no-title: true
  ///

• Or we can use Markdown Exec (+ PyMDown Snippets). We expose qtile_graph and qtile_command functions in the Python environment that builds docs, and write such code blocks:

  ```python exec="1"
  from somewhere import qtile_graph
  qtile_graph("core")
  ```
  
  ```python exec="1"
  from somewhere import qtile_graph
  qtile_command("libqtile.backend.x11.core", object_node="core", title=False)
  ```

  If exposing them (from somewhere) is too annoying, it's also possible to inject the functions into a code block using snippets, and reuse the same execution session in subsequent code blocks 🤯:

  ```python exec="1" session="docs"
  --8<-- "scripts/qtile_graph.py"
  --8<-- "scripts/qtile_command.py"
  ```
  
  ```python exec="1" session="docs"
  qtile_graph("core")
  ```
  
  ```python exec="1" session="docs"
  qtile_command("libqtile.backend.x11.core", object_node="core", title=False)
  ```

(--8<-- looks like a line with scissors)

[elParaguayo]

I think Markdown Exec could work here. Jina templates should still work too (with some tweaks).

I'll clone your branch and have a play. Can you add any further changes you make as new commits so the root commit here is unchanged.

[pawamoy]

Just a note that I didn't forget about this, I'm just very busy these days 😄
I'll try to at least resolve conflicts next week 🙂

[pawamoy]

Note to myself. Once the resulting docs look good and are complete/correct (compared to the current ones), rebase commits to ease code review:

1. Rename pages (rst -> md)
2. Setup MkDocs
3. Convert pages from rst to md
4. Convert docstrings from rst to md
5. Clean up Sphinx (files and contents)

[pawamoy]

I have reworked my commits to make them easier to review. The 3rd and 4th are the hardest one (lots of changes, converting RST to MD).

Please let me know if you're OK with switching to Google-style docstrings, or if I should stick to Numpydoc-style 🙂

[ramnes]

I don't think we're following Numpydoc, just using rST in the docstrings as we can. 😅

Anything would work, pick what you prefer.

[pawamoy]

Here are the rules I followed to write/update docstrings:

• One-line summary.
• Optional body separated from summary by one blank line.
• Sentences start with a capital letter and end with punctuation (usually a dot). That includes parameter descriptions.
• Parameters are documented in the __init__ method docstring, not in the class docstring.
• Don't repeat parameter types in docstrings. Prefer adding them to the signature instead, using type annotations.
• No dynamic docstrings (using formatting features such as %(), {}, concatenation with +, assignments to __doc__, or f-strings). The exception is docstrings of dynamically constructed objects: these can set their docstrings by assigning to __doc__, and the docstring can be as dynamic as we want (we will have to rely on dynamic analysis anyway to retrieve them).
• Docstrings are written in Markdown, use appropriate markup.
• For admonitions, we use markdown-callouts.

There are other rules I did not check or enforce here, because tools like Ruff can do that automatically in the time it takes me to blink:

• No line jump at the start of a docstring.
• No blank line at the end of a docstring.
• Docstrings are tripled-quoted using double-quotes.

And yet again other rules I did not enforce because it's a matter of preference and is not super important:

• Line jump (or not) at the start of a parameter description.

Further changes can be made in separate PRs.

Since we're using mkdocstrings here, which uses Griffe to parse docstrings, refer to Griffe's docs for a description of the Google-style: https://mkdocstrings.github.io/griffe/docstrings/#google-style.

[pawamoy]

OK I actually ran Ruff with the following settings:

[lint]
# Enable all `pydocstyle` rules, limiting to those that adhere to the
# Google convention via `convention = "google"`, below.
select = ["D"]

# On top of the Google convention, disable `D417`, which requires
# documentation for every function parameter.
ignore = ["D203", "D213", "D407", "D408", "D409", "D410", "D413", "D417"]

[lint.pydocstyle]
convention = "google"

ruff check --fix-only libqtile --unsafe-fixes

It fixed all the missing dots at the end of summaries as well as blank lines at the start/end of docstrings.

[pawamoy]

Now that docstrings are all formatted using the Google-style, the docstring parser from Griffe is able to tell us when some parameters name are wrong or missing type info:

WARNING -  griffe: libqtile/layout/slice.py:150: No type or annotation for parameter 'windows'
WARNING -  griffe: libqtile/layout/slice.py:151: No type or annotation for parameter 'mapping'
WARNING -  griffe: libqtile/layout/tree.py:673: No type or annotation for parameter 'sorter'
WARNING -  griffe: libqtile/layout/tree.py:676: No type or annotation for parameter 'create_sections'
WARNING -  griffe: libqtile/widget/maildir.py:75: No type or annotation for returned value 1
WARNING -  griffe: libqtile/widget/mpd2widget.py:86: No type or annotation for parameter 'status_format'
WARNING -  griffe: libqtile/widget/mpd2widget.py:86: Parameter 'status_format' does not appear in the function signature

[pawamoy]

@elParaguayo I managed to render the subscribe hooks 🙂

[pawamoy]

I have converted the changelog to Markdown, see the result here: https://pawamoy.github.io/qtile/manual/changelog/.

[pawamoy]

@ramnes I have reworked the navigation for layouts so that they are all displayed on the left, each in a separate page, see https://pawamoy.github.io/qtile/manual/commands/api/layouts/. I had to manually create the pages and list them in the navigation in mkdocs.yml to retain the contents in the "Layouts" page itself (which is a section). We can generate all these sub-pages automatically (with some scripts and config, see the gen-files and literate-nav plugins in mkdocs.yml), but it won't allow us to have contents in the "Layouts" section itself, unless we automate the whole "Commands API" section (which seems possible too). See oprypin/mkdocs-gen-files#31.

I'll rework the Widgets section the same way, because it is currently a single enormous page that has trouble loading even with decent bandwidth / CPU.