3.3.0: build_sphinx fails

[kloczek]

+ /usr/bin/python3 setup.py build_sphinx -b man --build-dir build/sphinx
running build_sphinx
Running Sphinx v4.1.1

Configuration error:
There is a programmable error in your configuration file:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/sphinx/config.py", line 327, in eval_config_file
    exec(code, namespace)
  File "/home/tkloczko/rpmbuild/BUILD/anyio-3.3.0/docs/conf.py", line 17, in <module>
    v = pkg_resources.get_distribution('anyio').parsed_version
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 482, in get_distribution
    dist = get_provider(dist)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 358, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 787, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'anyio' distribution was not found and is required by the application

I'm not 100% but looks like it is because source code of the module is in src/

[agronholm]

tox -e docs works fine, as do the RTD builds. What you're missing is a virtualenv where you have the doc extra installed.

[kloczek]

It would be need to use tox if source code would be not under src/.
KISS principle ..
Nevertheless hopefully cases like that will be solved in sphinx code.

[agronholm]

The RTD builds are not using tox, and they still work fine. You need to have the project installed (preferably in editable mode) to make the docs build correctly.

[kloczek]

This issue is not abut that buil_sphins is not able to find anyio module but pkg_resources.DistributionNotFound. so it is not related to location of the module source code under src/
I have no idea what kind of error it is. Dou know maybe what exactly it is?

Using tox for me is not viable solution because it has hardcoded generate html and I need man pages roff format.

[agronholm]

That error is caused by the pkg_resources trying to find the installed AnyIO distribution to get its version number.

Try this (in a fresh virtualenv):

pip install -e .[doc]
python setup.py build_sphinx

[kloczek]

That error is caused by the pkg_resources trying to find the installed AnyIO distribution to get its version number.

Try this (in a fresh virtualenv):

pip install -e .[doc]
python setup.py build_sphinx

OK I think that this discussion is over.
What was missing was module metadata which cen be generated using pip install -e . or. (simpler) pythson setup.py egg_info.
pythson setup.py <command> can handle multiple commands so I've choose to add egg_info on ececuting build.
I've opened pypa/setuptools#2761 to call egg_info as same as it is executed during install and test becasue anyio is not onlymodule which have issue the same issue.

Nevertheless thank you for your time :)

[kloczek]

I'm using exewctly the same methoidology to build docuymentation in more than 200 other modules and only anyio is kind of exception.
I want only to say that it should be working without virtual env like in all other modules.

[agronholm]

Why? All my projects use the same template and nobody has complained before.
The reason it's using pkg_resources to get the version is so that the version number would not have to be duplicated in docs/conf.py.

[smurfix]

@kloczek Well, those other packages probably get their version from some hardcoded __version__.py file (or similar), for various sub-optimal reasons. IMHO that's annoying, there should be exactly one source for versioning, which is the SCM tag because that's what you can introspect without inspecting actual content.

Duplicating that information just means that you forget to update one of the two, more often than not. I lost count on how many times I did that.

[kloczek]

A simple fix for this would be to make the docs tox environment pass along any command line arguments. That way you could do something like tox -e docs -- -b epub to get whatever format you want. Would that work for you?

Other modules are obtaining version from for example its own __init__.py or main module .py file.
AFAIK $PYTHONPATH or other env variable cannot point to use other base directory than standard python sitearh/sitelib. From that point of view you are forcing to install module from root or to use tox/venv.
I think that your approach may be breaking the case when someone has installed some additional modules in home directory because despite that actual code of the modules would be taken from local resources pip and even anyio module code will be still using system metadata (I'm not 100% sure of that but is is quite possible that it will be natural consequence of using your approach.
Additionally it definitely causes some difficulties on packaging anyio from non-root account.

As I'm working on packages I have in my devel system installed Everything™️ so one more or one less package installed in devel env is not a problem. However my prod packages build procedure is using lxc zone spawned just to build only one package where on top very small system image is installed all packages listed in rpm spec file BuildRrequires.

PS. Thanks for that example how to pass arguments to command executed in tox. I didn't know that it is possible to do that that way :)

[kloczek]

As for setup.py build_sphinx, how would you build the documentation for projects that use poetry or some other build system other than setuptools?

poetry and few other are completely different type of beast and I still did not completed my python-poetry.spec file :(
To be honest I've started working on that only yesterday and for now I have only:

%prep
%autosetup -p1 -n poetry-%{version}

%build
%__python3 -Bm build --no-isolation -s

#py3_build

%install
%py3_install

%check
%pytest

With poetry and few other issue is that project.toml and setyp.py seems completely do not care about standardising documentation and test suite :( (not only build but install as well)

I've already described that new landscape in my other comment sphinx-doc/sphinx#9517 (comment)
You can take out of that picture tox case as you've already provided some better solution than mine patch for anyio case but other parts of that longer comment seems are still kind of very big obstacle on the way of abandoning setuptools.
Especially IMO complete lack of taking care of building documentation and testing by PEP517 really scares me .. :(

IMO more and more convinced that was designed without typical rpm or Solaris packaging procedures in mind. IMO PEP517 if it will be not redesigned it will be really big pain in the a*s of people like me :/

Going back to part of my python-poetry.spec and new approach with project.toml/setyp.cfg ..
Generally to build documentation you must have all fils of the module deployed somewhere.
Special subset of that new approiach is some example module which installs not only .py files but DSOs.
Looks like .whl file and build module completely do not handle that case :(
sphinx-build and pytest cannot use sdist or .whl as input so currently it is necessary to install .whl using installer or pip and than it is possible to use sphinx-build.

At the moment I have only three modules which are using PEP517

[tkloczko@barrel SPECS]$ grep no-isolation *
python-flit.spec:%__python3 -Bm build --no-isolation
python-flit.spec:%__python3 -Bm build --no-isolation --skip-dependency-check
python-rich.spec:%__python3 -m build --no-isolation --wheel
python-sphinx-inline-tabs.spec:%__python3 -m build --no-isolation

Here is example from my python-flit.spec:

%build
cd flit_core
FLIT_NO_NETWORK=1 \
%__python3 -Bm build --no-isolation
cd -

PYTHONPATH=$PWD/flit_core \
%__python3 -Bm build --no-isolation --skip-dependency-check

sphinx-build -b man -d %{name} doc .

%install
%py3_install_wheel *.whl
cd flit_core
%py3_install_wheel *.whl
cd -

%__install *.3 -Dt %{buildroot}%{_mandir}/man3

%check
%pytest

As you may see I'm using raw sphinx-build command. I'm not happy about that but for now that is all what I can do :(
It is total mess :(

As I have only three examples of using new approach I using that without rpm macros.
I've been looking on fedora approach as well but IMO that approach is broken by design so for now I'm only collection some new cases to be able standardise that in the future to the same level as have now when module build framework still has setuptools support.

And of course PEP517 and setuptools do not provide standardisation of the documentation installation so this is why everywhere where I'm building modules man page I'm using in %install line like above with %__install *.3 -Dt %{buildroot}%{_mandir}/man3. To be able use exactly the same line in all moduless spec files I'm passing to build_sphinx command params --build-dir build/sphinx (I found that many modules have in setup.cfg or setup.py changes default build-dir so this is why I was forced use some exact base path).
In other words that example from my python-flit.spec would be not working for modules producing DSOs :(
Using exactly the same install command would allow me in the future hide documentation install behind new miacro like %py3_install_sphinx or so and introduce it in all specs using single sed command.

As I wrote that part is messy but when setuptools build_sphinx will be fixed I'll try probably standardise some approach to sphinx generated documentation. Without that trying to propose some new/standard approach IMO is a bit pointless.

If you have any though or comments about what you see above please write it those comments here. I'm still
learning about python packaging tooling so still it is quite possible that I've misunderstood some ideas of the new approach.

[agronholm]

With poetry and few other issue is that project.toml and setyp.py seems completely do not care about standardising documentation and test suite :( (not only build but install as well)

What do you mean? pip install . will install any standards compliant project (pyproject.toml) or legacy setuptools/distutils type project (setup.py).

As far as documentation goes, there is no standard for generating it.

[kloczek]

With poetry and few other issue is that project.toml and setyp.py seems completely do not care about standardising documentation and test suite :( (not only build but install as well)

What do you mean? pip install . will install any standards compliant project (pyproject.toml) or legacy setuptools/distutils type project (setup.py).

As far as documentation goes, there is no standard for generating it.

I'm not sure what exactly pip install . does :/
You mean that it will install all files taken from generated dist/.whl files into /build/lib with writing setup.py file in current directory?
If that what exactly is done by that commands this would allow continue use setuptools procedures :)
Probably it will allow hide all PEP517 steps into rpm macros like %py3_build by simple

• check if project.toml and setup.cfg are present
  • if yes: perform %__python3 -Bm build --no-isolation; pip install .
• continue with setuptools

[agronholm]

As far as I understand, pip install . creates a wheel using the backend configured in pyproject.tomland then installs it.

Note that setup.cfg is a setuptools specific file and is not used by other PEP517 backends. Going forward you cannot rely on any setuptools specifics like setup.py. Until standards for actions other than building and installing the project emerge, you need to have a project specific procedure in place for things like generating documentation.

[kloczek]

And yet another short comment ..

Using projest.toml/setup.cfs would be not so painful if builder would be packaging documentation and test units.
It generates setup.cfs/setup.py file so in that case would be possible to use pypi source dost tag balls but that actually will be orthogonal to have kind of guarantee that on top of sdist tar ball will be possible quickly add some patches obtained from SCM.
In other words as you see PEP517 was not designed in mind with real packaging scenarios or it was only designed for scenario when only pip is used as package manager :(
It could work if all modules maintains will be making new release right after sporting some issue.
We are living in world in which we are dealing only with not-well-tested software so time is critical and packaging methodologies must take care of some quick JFDI™️ fixes.
This is why rpm and other package management software provides possibility to make package release without changing package version. IMO python tooling should at least not make additional obstacles when it is necessary to make such JFDI package release.
Sad but for now that is not part of whole PEP build tooling design because that new approach is pip centric :(

Using pip in system images duplicates for example rpm database. For example I would love to see some new version of the pip which would abandon use .egg-info/.dist-info files and as backend will provide socket to plug obtaining metadata directly from packages management software (like rpm or Solaris IPS or other PM call software databases).
This may need some adaptation on both sides (python and PM software) but IMO sooner or later this would be only solution because current duplication of the same data is more and more problematic.

Similar situation with python vs. PM tooling is with perl, Ocaml, rust, texlive and few other when all that software needs to cooperate with rpm/IPS or generally package management software.
Even pkgtool metadata IMO is possible to integrate around typical PM database.

[agronholm]

Using projest.toml/setup.cfs would be not so painful if builder would be packaging documentation and test units.

WTF is "builder"?

have kind of guarantee that on top of sdist tar ball will be possible quickly add some patches obtained from SCM.

Is there something preventing you from downloading commit specific tarballs from, say, Github?

In other words as you see PEP517 was not designed in mind with real packaging scenarios or it was only designed for scenario when only pip is used as package manager :(

And you don't think installing packages from PyPI counts as a "real packaging scenario"?

This is why rpm and other package management software provides possibility to make package release without changing package version. IMO python tooling should at least not make additional obstacles when it is necessary to make such JFDI package release.

JDFI = Just F****** Do It?

I'm not sure I'm following, but do you mean replacing releases with the exact same file names? The PyPI at least does not allow this, so what is the intent here?

Sad but for now that is not part of whole PEP build tooling design because that new approach is pip centric :(

No, it's not. It's standards centric. Pip just happens to be the only complete and mature implementation of said standards.

IMO python tooling should at least not make additional obstacles when it is necessary to make such JFDI package release.

What obstacles are you talking about?

For example I would love to see some new version of the pip which would abandon use .egg-info/.dist-info files and as backend will provide socket to plug obtaining metadata directly from packages management software (like rpm or Solaris IPS or other PM call software databases).

The system pip can be used to install packages to the user's home directory, or virtualenvs. A few years back, pip would, by default, install packages in the same directory where system package managers installed them, often rendering the operating system partially unusable as a result. Thus, having pip manipulate the system package database is a horror we would like to avoid, not encourage.

[kloczek]

Using projest.toml/setup.cfs would be not so painful if builder would be packaging documentation and test units.

WTF is "builder"?

Gosh .. correction 'build` https://github.com/pypa/build/
Sorry.

have kind of guarantee that on top of sdist tar ball will be possible quickly add some patches obtained from SCM.

Is there something preventing you from downloading commit specific tarballs from, say, Github?

As sdist does not provide exactly the same set of files there is no 100% guarantee that it will be possible to apply without adaptation of the patch downloaded from exact commit hash.
Only this.

In other words as you see PEP517 was not designed in mind with real packaging scenarios or it was only designed for scenario when only pip is used as package manager :(

And you don't think installing packages from PyPI counts as a "real packaging scenario"?

No because 'pip` is only for python resources.

[tkloczko@barrel SPECS]$ rpm -qa | wc -l; rpm -qa python-\*| wc -l
6346
543

From that point of view it would be possible to say that pip is trying to solve exactly the same set of problems as set consisting from rpm, rpmbuild, rpmverify, rpmspec and rpmsign. rpm evolved to current form +25 years (since it was first time implemented in perl only).

This is why rpm and other package management software provides possibility to make package release without changing package version. IMO python tooling should at least not make additional obstacles when it is necessary to make such JFDI package release.

JDFI = Just F****** Do It?

Just F*k Do It :P
There is some mutation of the JFDI which is JFDJN (`Just F*k Do It NOW)
Very old Unix admins acronym which is possible to find in any urban dictionary.

I'm not sure I'm following, but do you mean replacing releases with the exact same file names? The PyPI at least does not allow this, so what is the intent here?

Lets say it is CVE issue in exact module it some other critical issue and you want to apply fix with fuill track of that operation. rpm allows do that with jut release new package with exactly the same package version but bumped release. The same is with Solaris IPS which additionally allows query IPS database is my system image exact CVE prone or not (rpm still is not able to learn about that to provide such metadata in package metadata description).
So far I don't see that any python packaging tooling is able to handle such scenario with possibility to roll back whole change (with restoring configuration files changes as well)

Sad but for now that is not part of whole PEP build tooling design because that new approach is pip centric :(

No, it's not. It's standards centric. Pip just happens to be the only complete and mature implementation of said standards.

Reading PEP517 I had other impression but you may be right that it is only my impression.

IMO python tooling should at least not make additional obstacles when it is necessary to make such JFDI package release.

What obstacles are you talking about?

Try to think .about scenario: exact module maintainer just committed to source tree git repo critical fix or on some customer branch has been committed custom fix.
What pip offers to apply that fix quickly and keep track of that fix in packages database?
I don't see anything ..
IPS offers something which is called variants which allows you to install exact variant of exact package.
Lets say you have something like mathplot which has DSOs and you want to have installed that module but optimised for your microarchitecture. IPS handles that kind of scenarios if in IPS repository are offered lets say different binaries optimised for different versions of x86 microarchitectures (like AMD and Intel high-end CPUs).

For example I would love to see some new version of the pip which would abandon use .egg-info/.dist-info files and as backend will provide socket to plug obtaining metadata directly from packages management software (like rpm or Solaris IPS or other PM call software databases).

The system pip can be used to install packages to the user's home directory, or virtualenvs. A few years back, pip would, by default, install packages in the same directory where system package managers installed them, often rendering the operating system partially unusable as a result. Thus, having pip manipulate the system package database is a horror we would like to avoid, not encourage.

OK so that could be partially dropped from MyWishList™️.

Nevertheless I want only to say that many problems with which python packaging tooling is dealing are already solved in case of currently used systm packages management software.
I'm not expecting from that point of view that python tooling would take care of dependencies resolution or other operation which are not performed by for example GNU autotools, meson, cmake/qmake and few other.

Issue is that currently setuptools and new approach are not taking care holistically of the build and install standardisation.setuptool build should be like make all in case make. The same with install. setuptools install doesn't know that it should install for example documentation installation as well.
Why frameworks like unitest, pytest, tox and maaany more are not obligatory plugged under setuptool test?
At the moment python packaging it is set of isolated islands which forces on packaging new module spent time to recognise how exact module needs to be build, installed and tested.
When I've first time heard about PEP517 I've been thinking that all those islands will be somehow sawed/entangled under new plugeable hood. What I see currently is only about replace some operations performed GoodEnough™️ by setuptools without taking care holistically all stages which currently is possible to take care now using that framework. Using setuptools is possible to generate .whl or sdist file.

Don't get me wrong .. I'm not trying to complain. I'm looking on that new approach as IMO partially derailed ..
I would be forced to digest that new approach somehow but without any new added value to the whole packaging methodologies which I've learn in last +25y.