3.5.1: cannot add module metadata in project tree

[kloczek]

I'm trying ot add .{dist|egg}.info metadata in structlog module which is using as build backend flit and looks like it is not possible to do that

+ cd structlog-21.5.0
+ /usr/bin/python3 -sBm build -w
* Creating venv isolated environment...
* Installing packages in isolated environment... (flit_core >=3.4,<4)
* Getting dependencies for wheel...
* Building wheel...
Successfully built structlog-21.5.0-py3-none-any.whl
+ pip install -e .
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/tkloczko/rpmbuild/BUILD/structlog-21.5.0
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: structlog
  Building editable for structlog (pyproject.toml) ... done
  Created wheel for structlog: filename=structlog-21.5.0-py3-none-any.whl size=8107 sha256=e193ec3c74a678106d51a2b593a7d5037c7fa94cc2fdeae0fbb66b7eb0081bc1
  Stored in directory: /tmp/pip-ephem-wheel-cache-68zzmr4f/wheels/35/89/3a/cd47c49b9d6c16185c5a0f83a41dd101771681f0d2bcadf8f3
Successfully built structlog
Installing collected packages: structlog
  Attempting uninstall: structlog
    Found existing installation: structlog 21.5.0
    Uninstalling structlog-21.5.0:
      Successfully uninstalled structlog-21.5.0
Successfully installed structlog-21.5.0

Loks like pip is not able to take [project] section from pyproject.toml.
How can I do that? flit command seems does not offer anything like generate .{dist|egg}.info directory :/

[takluyver]

If you're writing a PEP 517 frontend, you can use the prepare_metadata_for_build_wheel hook to create a .dist-info folder. Normally this would be done in some temporary build folder, but the frontend specifies where it should go, so it can ask for it in the project directory if it wants.

Otherwise, the .dist-info folder is generated in the wheel when you create that, and it also exists in site-packages when you install a distribution. There's normally no reason to have it in the project source tree, and I consider it a misfeature that setuptools tends to create extra files there. If you want it for some reason, you can extract it from a wheel or copy it back after installing.

.egg-info is a setuptools specific thing, not part of any standard. Flit will only ever work with the standardised .dist-info folders.

[kloczek]

There's normally no reason to have it in the project source tree, and I consider it a misfeature that setuptools tends to create extra files there. If you want it for some reason, you can extract it from a wheel or copy it back after installing.

+55% of my rpm packages with python modules renders documentation over sphinx. Amongst those packages ~30% needs access to the package metadata to injet things like version, athor or other to rendered documentation. Usually it is not a big problem because whem setuptools is used as pep517 backend metadata directory is created by default. That means that performed as next step documentation rendering can be done without any issues.

So what exactly needs to be added in project.toml tto have created metadata directory during the build when flit is used?

[takluyver]

Can you point me to an example of a package built with Flit where the Sphinx docs rely on a .dist-info folder? I've written and contributed to Sphinx docs for many different projects and I don't recall ever coming across Sphinx using a .dist-info folder. It also seems like that wouldn't work with Flit for anyone, so I can't see why project maintainers would set it up that way (or keep it that way if they switched to Flit).

[pradyunsg]

@kloczek Please read https://pip.pypa.io/en/latest/reference/build-system/ to understand how setup.py based projects vs pyproject.toml based projects are different. That documentation also goes into how the process for generating metadata for those two styles of build systems is different. You can use the https://github.com/pypa/pep517/ project to write a frontend that can generate metadata as you need.

[kloczek]

I've already pointewd on such example .. structlog module.
I've already mentioned about reason to use modume metadata during documentation rendering which is for example injecting modume version or orgr metadata entries to the rendered documentation.
As thopse metadata are or may chage between releases it make sense to not hardcode those details into documentation source.

So .. does it mean that when flit is used as pep515 backend there is no currently any way to generate metadata diorectory?

[pradyunsg]

So .. does it mean that when flit is used as pep515 backend there is no currently any way to generate metadata diorectory?

There is, and @takluyver has mentioned it already:

If you're writing a PEP 517 frontend, you can use the prepare_metadata_for_build_wheel hook to create a .dist-info folder. Normally this would be done in some temporary build folder, but the frontend specifies where it should go, so it can ask for it in the project directory if it wants.

[kloczek]

@kloczek Please read https://pip.pypa.io/en/latest/reference/build-system/ to understand how setup.py based projects

We are not taking about setuptools but about flit.
Reading that documentation will not help me on altering build ptrocedure to have metadata directory generatyed before rendering documentation .. please stick to the flit subject.

[pradyunsg]

It will.

[pradyunsg]

See also https://www.python.org/dev/peps/pep-0517/#prepare-metadata-for-build-wheel, because I'm sure you will ask a bunch of follow up questions that are already answered in the PEP. :)

[pradyunsg]

@kloczek Your comment seems to be poorly formatted. I request you to read https://help.github.com/articles/getting-started-with-writing-and-formatting-on-github/ and improve the formatting of your comment by editing it in place, using the pencil icon in the corner.

[kloczek]

So moment .. does it mean that flit code needs to be prepareded/fixed using that hook to be able to generate module detadata?

[pradyunsg]

Yes? You need to run the PEP 517 hook, as explained already.

I'm not going to be answering any more direct questions that are clearly answered in the documentation I've already linked you to. Please do the reading work necessary to understand how to achieve what you're trying to do.

If something is unclear, feel free to ask more questions.

[takluyver]

I've already pointed on such example .. structlog module.

You did, my apologies. It appears that structlog is set up that it expects you to install the Python package first, then build the docs. This is what both the Readthedocs config and the docs task in tox.ini will do. If possible, I'd recommend that you do that too.

I tried, out of interest, copying the .dist-info folder back to the project directory and then uninstalling structlog. It doesn't work - the docs build still fails with importlib.metadata.PackageNotFoundError, because it's not looking for the metadata in that location (I tried both with make and the sphinx-build command directly). You could probably patch conf.py to make that work, but that seems like a workaround for a workaround.

@pradyunsg I think there was a sensible question to answer here. It's just come as an X Y problem, where X is 'make Sphinx builds using importlib.metadata work' and Y is 'create a .dist-info folder in the project tree'.

[kloczek]

OK so it woud be really good to implemt calling that hook.
I'v ealready migrated and tested only 1/8 of all my rpm packages with python modules to pep517 based build procedure.

[tkloczko@ss-desktop SPECS]$ grep 'python3dist(flit)' *
python-cssselect2.spec:BuildRequires:   python3dist(flit)
python-installer.spec:BuildRequires:    python3dist(flit)
python-jeepney.spec:BuildRequires:      python3dist(flit)
python-sphinxcontrib-github-alt.spec:BuildRequires:     python3dist(flit)
python-structlog.spec:BuildRequires:    python3dist(flit)
python-threadpoolctl.spec:BuildRequires:        python3dist(flit)
python-tomli.spec:BuildRequires:        python3dist(flit)
python-tomli-w.spec:BuildRequires:      python3dist(flit)
python-typing-extensions.spec:BuildRequires:    python3dist(flit)
[tkloczko@ss-desktop SPECS]$ grep 'python3dist(flit)' * -l| xargs grep %py3_build_sphinx_man
python-cssselect2.spec:%py3_build_sphinx_man
python-jeepney.spec:%py3_build_sphinx_man
python-structlog.spec:%py3_build_sphinx_man

Looking on above list it will be more than one such case.

[kloczek]

It appears that structlog is set up that it expects you to install the Python package first, then build the docs

This is not what precisely is expected. In such cases it is expected access to the module metadata .. only.
That it is only coincidence that after full module instalation metadata are available.
As setuptools by default during build generates metadata directoey in cureret directory I think that flit could follow that behaviour. This will solve typical build, install and testing procedure needs typiucally used on packagin (rpm or Solaris IPS or other when such build is not performed from root).

[kloczek]

BTW flit and sphinx.
I'm using so far setuptools<>sphinx integration to rented man pagase https://www.sphinx-doc.org/en/master/usage/advanced/setuptools.html.
Of course as flit is used it is not possible use straight that way so I;m forced to use dummy setup.py to be able use that procedure.

[tkloczko@ss-desktop SPECS]$ grep 'python3dist(flit)' * -l| xargs grep echo
python-cssselect2.spec:echo 'from setuptools import setup; setup()' > setup.py
python-jeepney.spec:echo 'from setuptools import setup; setup()' > setup.py
python-structlog.spec:echo 'from setuptools import setup; setup()' > setup.py

[tkloczko@ss-desktop SPECS]$ rpm -E %py3_build_sphinx_man
\
        PBR_VERSION=%{version} \
        SETUPTOOLS_SCM_PRETEND_VERSION=%{version} \
        /usr/bin/python3 setup.py build_sphinx -b man --build-dir build/sphinx

It would be nice if flit will have its own flit<>sphinx integration (as I wrote more than half my modules provides possibility to render dosumentation in necessay format)

[kloczek]

And .. probably the best would be if pep517 would provide hook to generate documentation :)

[takluyver]

That it is only coincidence that after full module instalation metadata are available.

No, it's not a coincidence. That is the metadata that accompanies an installed module. It's described in a specification entitled Recording installed projects. The coincidence is that setuptools also tends to dump metadata in the project tree when you do certain operations.

As setuptools by default during build generates metadata directoey in cureret directory I think that flit could follow that behaviour.

Writing the .dist-info in the project tree doesn't directly solve the case you tried, and I consider it a setuptools misfeature - I don't want extra files showing up in my project directory.

I understand that you've developed a whole load of tricks around setuptools, and I'm sorry that we're breaking them. But we've spent a load of time and effort getting away from the idea that everything in packaging must either use setuptools or pretend to be setuptools. We'll help you adapt to that, but we're not interested in arguments that 'you should do this because setuptools does'.

It would be nice if flit will have its own flit<>sphinx integration
And .. probanly the best would be if pep517 would provide hook to generate documentation

We've been over this topic at some length in other issues. Both Flit and PEP 517 are about packaging - that is, how we get from a source tree to an installable artefact and to an installed Python package (importable libraries, runnable scripts, etc.). From the perspective of people working on Python packaging, building documentation is an entirely separate concern, so we're not about to mix it into packaging tools or standards.

At a practical level, it's also not clear to me what 'flit<>sphinx integration' would do. It would be neat in theory if there was one accepted way to build docs regardless of packaging or documentation system, but given that you're already using the Sphinx specific setup.py build_sphinx, it seems much simpler to use sphinx-build directly and not run it through a packaging tool.

Fedora has already developed tools to put Python packages into rpms using PEP 517 interfaces. Maybe you could reuse some of their work? Two Fedora developers gave a talk about it at PackagingCon, which you can see here: https://www.youtube.com/watch?v=pJlqB-PA96E

[kloczek]

Writing the .dist-info in the project tree doesn't directly solve the case you tried, and I consider it a setuptools misfeature - I don't want extra files showing up in my project directory.

So what is the proper/right solution in such cases when after build other steps may need access to module metadata before actual install?
Just in case .. those other steps may have nothing to do with setuptools per se.

At a practical level, it's also not clear to me what 'flit<>sphinx integration' would do

Using setuptools<>sphinx integration on packaging is VERY useful because it doesn't matter where source of the documentation and conf.py files are I can render documentation using exactly the same command (setup.cfg contains documentation root relative directory)

[tkloczko@ss-desktop SPECS]$ grep %py3_build_sphinx_man python-* | wc -l; ls -1 python-*|wc -l
378
738

Fedora has already developed tools to put Python packages into rpms using PEP 517 interfaces

If may I point that Fedora do not care about generate python modules documentation and/or generate that documentation in unified form.
Look .. Fedora has now ~2.7k modules packages and only handful have included documentation (in lass cases than in my 738 packages with python modules).

[tkloczko@ss-desktop SPECS.fedora]$ grep '%package.* doc' python-*spec| wc -l
252

BTW pep517 and Fedora. Still %75 of all Fedora modules are not using pep517

[tkloczko@ss-desktop SPECS.fedora]$ grep %py3_build python-* | wc -l; ls -1 python-*|wc -l
2061
2716

(In that case I would ignore fact that most Fedora packagers do not understand that if someone needs to install package X without documentation that documentation do not need to be packages in separated package because on install time it is possible to decide to install that package with or without documentation by use or not --excludedoc rpm option).

In other words .. Fedora it is not the best example to show something in context of package python modules documentation :(

[pradyunsg]

So what is the proper/right solution in such cases when after build other steps may need access to module metadata before actual install?

There isn't. You'll need to install the package in the environment where you're trying to generate the documentation. Here's an example of how that could work, as a sequence of shell commands:

pip install .  # replace this with whatever you want to do to install structlog
sphinx-build -b html docs/ docs/_build

[kloczek]

Just finished watching that video.
I did not learn anything new from that presentation.
In that presentation there is no even single sentence about python modules documentation standardisation (which I did in my own rpm macros).
They mention for example about testing packages with python modules however they are not using that widely in all possible cases.

[tkloczko@ss-desktop SPECS.fedora]$ grep %check python-*spec -l| wc -l; ls -1 python-*spec |wc -l
1918
2716

[tkloczko@ss-desktop SPECS]$ grep %check python-*spec -l| wc -l; ls -1 python-*spec |wc -l
738
738

I'm strictly using provided test suites so if you will looking on my profile activity recently majority of opened by me tickets is about test suites issues.

[takluyver]

So what is the proper/right solution in such cases when after build other steps may need access to module metadata before actual install?

You may choose any of these options:

• Install the package somewhere (even if temporarily), read its metadata from there
• Build a wheel, extract metadata from that
• Write some Python code that calls the PEP 517 prepare_metadata_for_build_wheel hook to create just .dist-info wherever you need it.

Using setuptools<>sphinx integration on packaging is VERY useful because it doesn't matter where source of the documentation and conf.py files are I can render documentation

I understand that that's useful. But it's nothing to do with packaging, and insisting that it is is not getting you anywhere. It's not going to be a feature in Flit. You will get further by accepting that and thinking about what to do instead than

If you want to make a little helper tool that looks for a relative path to docs in a specific key of pyproject.toml and passes that to sphinx-build, feel free. Even better, the same thing will work whether the package uses flit, setuptools, poetry, or anything else out there.

In practice, most packages I've seen call the folder either doc or docs. So if I were you, I'd just write something that tries each of those in turn for building docs, and then deal with the remaining cases manually.

Still %75 of all Fedora modules are not using pep517

That doesn't surprise me: most Python packages still use setuptools, so there's no need to change how they're packaged. It doesn't subtract from the fact that they have the mechanisms to deal with PEP 517 packages, and they appear to be using them for many packages without major problems.

I'm going to close this now, because I don't think it's going anywhere. You seem very determined to tell us exactly how we should approach packaging, and you don't seem to pay any attention when we disagree. We went through this at length on issue #429 as well. Engaging in an endless argument is a waste of everyone's time.

[kloczek]

pip install . # replace this with whatever you want to do to install structlog

Please look one more time on top of this ticket in which I've already wrote that this does not work (probably because pip do not understand flit metadata description provided in project.toml).
IIRC I've already mentioned in one of the flit tickets that flit is not sharing that section with what setuptools takes from that files.
Result is that pip is not generating module metadata directory.

[kloczek]

I'm going to close this now, because I don't think it's going anywhere. You seem very determined to tell us exactly how we should approach packaging

100% wrong impression .. please.
I'm only describing typical problems of the python modules packaging using general packaging software like rpm or Solaris IPS.
I did not told even single word about do anything.
I've suggested some new features (basing on substantial number of packages cases) however all that was only suggestion which should be verified/discussed before make the decision about which side of all processes should alter own procedures.

IMO fact that when flit is used as backend of the pep517 is not generating module metadata is kind of bug.
Is that flit or pep517 issue?
Looking on the fact it is substantial number of cases where access to module metadata just right after build I think that maybe it would be good to move that operation from al backends to pep517? .. because those metadata should be always in the same format (however I'm not 100% sure and I have no even idea how difficult would be such modification).

Nevertheless seems like delegating generate metadata to the backend modules seems it may be pep517 design issue (correct me if I'm wring) because all backend modules would be sharing here 100% of the code.

[kloczek]

We went through this at length on issue #429 as well. Engaging in an endless argument is a waste of everyone's time.

That ticket case was about proposition of handling documentation build by flit.
This ticket is only about modules metadata which may be need not only during documentation rendering procedures.
Both tickets are sharing some bits but they are generally about two different steps.

[pradyunsg]

IMO fact that when flit is used as backend of the pep517 is not generating module metadata is kind of bug.

It is generating that metadata. Do a pip install . in a Python package (use flit / setuptools or whatever valid build-backend you like), look inside site-packages, notice that there's going to be a corresponding [whatever].dist-info folder.

If you're looking somewhere else, that's the wrong place to look. Beyond that, I suggest you read PEP 517 and the links that have been shared with you earlier in this thread by the folks involved in designing the mechanisms that give you what you want (instead of dismissing the information being shared with you, to help you understand things). 🤷🏽

[kloczek]

It is generating that metadata. Do a pip install . in a Python package

Please try to do that with structlog module (https://github.com/hynek/structlog/)
Again I can only repeat what is on the top of this ticket.

[tkloczko@ss-desktop structlog-21.5.0]$ pip install .
Defaulting to user installation because normal site-packages is not writeable
Processing /home/tkloczko/rpmbuild/BUILD/structlog-21.5.0
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: structlog
  Building wheel for structlog (pyproject.toml) ... done
  Created wheel for structlog: filename=structlog-21.5.0-py3-none-any.whl size=53394 sha256=25248d87e3459092b5c7f59f3e97b743cdc8a5593212af9f6f821bc5a3280d4c
  Stored in directory: /home/tkloczko/.cache/pip/wheels/35/89/3a/cd47c49b9d6c16185c5a0f83a41dd101771681f0d2bcadf8f3
Successfully built structlog
Installing collected packages: structlog
  Attempting uninstall: structlog
    Found existing installation: structlog 21.5.0
    Uninstalling structlog-21.5.0:
      Successfully uninstalled structlog-21.5.0
Successfully installed structlog-21.5.0
[tkloczko@ss-desktop structlog-21.5.0]$ find . -name \*-info
[tkloczko@ss-desktop structlog-21.5.0]$

[kloczek]

Without structlog metadata I cannot make sphinx documentation

running build_sphinx
Running Sphinx v4.3.2

Configuration error:
There is a programmable error in your configuration file:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/sphinx/config.py", line 329, in eval_config_file
    exec(code, namespace)
  File "/home/tkloczko/rpmbuild/BUILD/structlog-21.5.0/docs/conf.py", line 46, in <module>
    release = metadata.version("structlog")
  File "/usr/lib64/python3.8/importlib/metadata.py", line 530, in version
    return distribution(distribution_name).version
  File "/usr/lib64/python3.8/importlib/metadata.py", line 503, in distribution
    return Distribution.from_name(distribution_name)
  File "/usr/lib64/python3.8/importlib/metadata.py", line 177, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: structlog

[takluyver]

@pradyunsg said "look inside site-packages", which your find command does not do. As he added, "If you're looking somewhere else, that's the wrong place to look."

We've both tried to explain that the generated metadata goes in the installation directory, not the project source tree, and it's found in site-packages when building the docs. I did try this locally with structlog: I installed from a git checkout, saw the .dist-info directory in site-packages, and was successfully able to build the docs.

[kloczek]

We've both tried to explain that the generated metadata goes in the installation directory, not the project source tree, and it's found in site-packages when building the docs. I did try this locally with structlog: I installed from a git checkout, saw the .dist-info directory in site-packages, and was successfully able to build the docs.

And I've been trying to explain that metadata *directory but not that directory inside .whl archive produced as part of the build process should be added. That metatadata inside of the .whl are useless for sphinx.
Modules which are using setuptools as pep517 backend produces .egg-info directory (this will soon change to .dist-info by default).
Looks like poetry has the same issue like flit but that is another story.

Again: about 60% of my all rpm spec files used to package python modules contains sphinx documentation. About 30% of all those modules to build documentation using sphinx needs module metadata to inject things like author, version or other bits to render documentation in exact output format.
So far flit is used by only handful modules so it is not big deal .. yet.

Really .. please to understand whole workflow of the build process during packaging into for example rpm or Solaris IPS package with rendering documentation as well.

[takluyver]

It is produced in the site-packages directory when you install, not just in the wheel. Building the documentation works with it there. Install the Python package first, then build the docs - or come up with a workaround like installing to a temporary directory and setting PYTHONPATH.

We're going round and round in circles here - I'm not telling you anything new, and you're not telling me anything new. You certainly haven't convinced me that anything needs to change in Flit. I'm sorry, but I'm going to lock this issue now - the endless argument is a waste of both of our time, and a source of frustration. When I get time for open source work, I want to do something more productive than this.