wip use quartodoc to build website

[eitsupi]

No description provided.

[max-sixty]

Cool!

Is there any way of seeing how it looks?

Do we necessarily lose the notebook?

[eitsupi]

Thanks for looking at this.
This is still in the early stages of touching up quartodoc and is far from complete.

One problem is that it causes conflicts with pydantic (machow/quartodoc#209) included in the semantic_release.
(I have no loyalty to semantic_release as you do, so I could update the CI of this repository to remove semantic_release separately from this PR, but I didn't feel like tackling such a large task yesterday)

Is there any way of seeing how it looks?

Finally I will include a devcontainer.json with this so it can be rendered in Codespaces.
It can also be run locally by attaching the rendered website as an artifact using GitHub Actions.

Do we necessarily lose the notebook?

I only removed it temporarily because it was annoying that Quarto recognized ipynb and tried to render it, and we can revert back to it later.

However, for ease of management in Git, it might be better to add an equivalent qmd file here and download the ipynb generated from the qmd from the website (looks possible but I have never used this feature).
https://quarto.org/docs/output-formats/html-multi-format.html#overview

[machow]

Ah, I'm a giant prql fan! Should be able to fix the pydantic thing by the end of the week.

RE having to remove the notebook, you should be able to customize the render option in your _quarto.yml to only render .qmd files, etc:

project:
  type: website
  render:
    - "*.qmd"

https://quarto.org/docs/websites/index.html#render-targets

I've cloned / built this PR, happy to help with any issues you hit building the docs!

[eitsupi]

@machow Thanks for looking at this!

My other problem is that quartodoc doesn't seem to work if I move the _quarto.yml to the docs directory like in other repositories.

I saw this:

$ quartodoc build --config docs/_quarto.yml
Traceback (most recent call last):
  File "/home/vscode/.local/lib/python3.11/site-packages/griffe/loader.py", line 114, in load_module
    module_name, package = self.finder.find_spec(module, try_relative_path=try_relative_path)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/griffe/finder.py", line 130, in find_spec
    return module_name, self.find_package(top_module_name)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/griffe/finder.py", line 169, in find_package
    raise ModuleNotFoundError(module_name)
ModuleNotFoundError: pyprql

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vscode/.local/bin/quartodoc", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/__main__.py", line 209, in build
    doc_build()
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/autosummary.py", line 519, in build
    blueprint = blueprint(self.layout, dynamic=self.dynamic, parser=self.parser)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/builder/blueprint.py", line 431, in blueprint
    return trans.visit(el)
           ^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 489, in __call__
    return self._f(self._instance, *args, **kw_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/builder/blueprint.py", line 185, in visit
    return super().visit(el)
           ^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 489, in __call__
    return self._f(self._instance, *args, **kw_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/builder/utils.py", line 45, in visit
    result = self.enter(el)
             ^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 489, in __call__
    return self._f(self._instance, *args, **kw_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/builder/blueprint.py", line 222, in enter
    return super().enter(el)
           ^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 489, in __call__
    return self._f(self._instance, *args, **kw_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/builder/utils.py", line 67, in enter
    result = self.visit(value)
             ^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 489, in __call__
    return self._f(self._instance, *args, **kw_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/builder/blueprint.py", line 185, in visit
    return super().visit(el)
           ^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 489, in __call__
    return self._f(self._instance, *args, **kw_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/builder/utils.py", line 45, in visit
    result = self.enter(el)
             ^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 489, in __call__
    return self._f(self._instance, *args, **kw_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/builder/utils.py", line 86, in enter
    result = self.visit(child)
             ^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 489, in __call__
    return self._f(self._instance, *args, **kw_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/builder/blueprint.py", line 185, in visit
    return super().visit(el)
           ^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 489, in __call__
    return self._f(self._instance, *args, **kw_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/builder/utils.py", line 45, in visit
    result = self.enter(el)
             ^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 489, in __call__
    return self._f(self._instance, *args, **kw_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/builder/utils.py", line 67, in enter
    result = self.visit(value)
             ^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 489, in __call__
    return self._f(self._instance, *args, **kw_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/builder/blueprint.py", line 185, in visit
    return super().visit(el)
           ^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 489, in __call__
    return self._f(self._instance, *args, **kw_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/builder/utils.py", line 45, in visit
    result = self.enter(el)
             ^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 489, in __call__
    return self._f(self._instance, *args, **kw_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/builder/utils.py", line 86, in enter
    result = self.visit(child)
             ^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 489, in __call__
    return self._f(self._instance, *args, **kw_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/builder/blueprint.py", line 185, in visit
    return super().visit(el)
           ^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 489, in __call__
    return self._f(self._instance, *args, **kw_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/builder/utils.py", line 45, in visit
    result = self.enter(el)
             ^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 489, in __call__
    return self._f(self._instance, *args, **kw_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/builder/blueprint.py", line 271, in enter
    obj = self.get_object_fixed(path, dynamic=dynamic)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/builder/blueprint.py", line 149, in get_object_fixed
    return self.get_object(path, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/quartodoc/autosummary.py", line 138, in get_object
    loader.load_module(module)
  File "/home/vscode/.local/lib/python3.11/site-packages/griffe/loader.py", line 120, in load_module
    top_module = self._inspect_module(module)  # type: ignore[arg-type]
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/griffe/loader.py", line 485, in _inspect_module
    module = inspect(
             ^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/griffe/agents/inspector.py", line 97, in inspect
    ).get_module(import_paths)
      ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/griffe/agents/inspector.py", line 176, in get_module
    value = dynamic_import(import_path, import_paths)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/griffe/importer.py", line 61, in dynamic_import
    module = import_module(module_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'pyprql'

Any thoughts?

[machow]

Does it work if you install pyprql? (e.g. with pip install -e . from the root)

I think quartodoc has to set the current working directory to where the _quarto.yml file lives, so if pyprql isn't installed, I wonder if it was being picked up before due to python seeing the pyprql folder in the working directory, but isn't when the working directory is docs.

(If this behavior seems surprising, it seems fair to add the directory quartodoc was called from to sys.path before switching working directories; or maybe not switch directories, but add the folder where _quarto.yml lives to sys.path).

[eitsupi]

Thank you for your prompt reply. understood.
I looked at ibis's justfile, etc., but it didn't install the package right before generating the document, so I didn't realize that it was necessary...
https://github.com/ibis-project/ibis/blob/6f77df99ec62501a522825dea05996b66cb97466/justfile#L139-L158

However, it seems common to want to generate documentation from the latest source code and to place _quarto.yml in the docs directory, so adding a feature that eliminates the need for installation may be a good idea.

[eitsupi]

(Maybe this is a tricky issue, as it has something to do with Python's behavior of not distinguishing between directory names and packages? Also, I don't know if other documentation generators for Python assume prior package installation, so my impressions may be off base.)

[machow]

There are some other advantages to installing--like being able to dynamically load docstrings in quartodoc--but it seems like switching the working directory is too spooky. I opened an issue to use sys.path, so that moving things to a subdirectory shouldn't produce these kinds of errors..!