doc_builder: Search for pyproject.toml as requirements file

[EwoutH]

This commit enables that beside from pip_requirements.txt and requirements.txt, the doc builder also searches for a pyproject.toml as requirements file.

pyproject.toml files are a modern implementation for storing project metadata, including dependencies, as defined in PEP 621. With this new search behaviour that includes pyproject.toml files, project following PEP 621 can enable Read the Docs and write a .readthedocs.yaml file without explicitly having to specify the requirements file, just like when using a requirements.txt file.

It's probably best for some documentation and tests to also be added for requirements defined in pyproject.toml files, please let me know where and how to add them! :)

[humitos]

Hi @EwoutH! Thanks for you contribution.

I just wanted to mention that we are moving away of the "auto-discovering of these files" feature. We strongly recommend people to use the config file (https://docs.readthedocs.io/en/stable/config-file/v2.html) and define how they want to install their dependencies.

In the case of a project using pyproject.toml, they would use something like:

version: 2 

python:
  install:
    - method: pip
    - path: .

Let me know if that clarifies our situation and if you have any doubt.

[EwoutH]

Thanks for the reply! Actually we're using a .readthedocs.yaml file with exactly that configuration. We also have a pyproject.toml file in the root. Unfortunately, that still resulted in an error:

ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'

Looking at other recent builds, it looks like the error is introduced recently though and only on a PR (which didn't change anything to the Read the Docs or pyproject.toml config.

Could it be a one-time error?

[humitos]

@EwoutH the error here is that Read the Docs is not finding your .readthedocs.yaml for some reason. Note that it's using Python 3.7 instead of the one declared by your config file. Maybe it's not on that PR, or similar?

[EwoutH]

A right, it was merged into a maintenance branch, that's quite logical.

If this functionality is phased out, than this PR can be closed.

[EwoutH]

Hi @humitos! We might have a related problem with our Read the Docs configuration. We notices search wasn't working, likely due to incompatible versions of "sphinx", "sphinx-rtd-theme", and "readthedocs-sphinx-ext" being installed. See #7858.

We specify our dependencies in an optional "docs" dependency in a pyproject.toml file.

[project.optional-dependencies]
docs = ["sphinx", "sphinx-rtd-theme", "readthedocs-sphinx-ext", "nbsphinx", "myst", "pyscaffold", "myst-parser"]

Then, our .readthedocs.yaml includes this:

python:
  install:
    - method: pip
      path: .
      extra_requirements:
        - docs

With that, our Read the Docs build log first install in a step sphinx<2, sphinx-rtd-theme<0.5 and readthedocs-sphinx-ext<2.3, and only after that in a separate step our actual .[docs] optional dependencies, which include sphinx-6.1.3 and sphinx-rtd-theme-1.2.0. For some reason readthedocs-sphinx-ext doesn't get updated, and stays on 2.2.0 (from <2.3).

So it seems defining docs dependencies in a pyproject.toml still doesn't properly work with Read the Docs. All details of this case are in this PR: quaquel/EMAworkbench#242

Do you have an suggestion on how to proceed from here?

[stsewd]

For some reason readthedocs-sphinx-ext doesn't get updated, and stays on 2.2.0 (from <2.3).

2.2.0 is the latest version of that extension, but you shouldn't install it by yourself, that dependency is managed by RTD for internal usage during the build process.

Please open a new issue with what exactly isn't working, but I suspect this may be related to readthedocs/sphinx_rtd_theme#1452.

[EwoutH]

I really don't understand how requirements work with ReadTheDocs anymore. I specifically added a docs/requirements.txt file specially for Readthedocs, with sphinx>=6, sphinx_rtd_theme>=1.2 and readthedocs-sphinx-search>=0.3, and even then it still installs sphinx<2 and sphinx-rtd-theme<0.5. Also for some reason it installs setuptools<58.3.0, which is really outdated.

By now I'm just following the docs to the letter, and search still doesn't work.

How do I need to specify these requirements that it installs them correctly?

[EwoutH]

I created a separate issue: #10263