replace setup.py with pyproject.toml ?

[petersilva]

in many environments, I read that setup.py is deprecated, and it looks like we should migrated to pyproject.toml. I'm reading in here:

• https://tttapa.github.io/py-build-cmake/FAQ.html#:~:text=The%20setup.py%20script%20and,declarative%20configuration%20files%20like%20setup.

it quotes the python setup guide, which says: "The setup.py file should be used only when custom scripting during the build is necessary. " ...

the current setup.py also copies other files existing in the directory as fields for the metadata, which again is something pyproject.toml doesn't look like it can do. I mean I would expect any/every project to require the version to be set. and the version will be different every time, so ... the .ini format of .toml has no was to run code in it, so you cannot interrogate the file system to determine the version number... so I guess that counts as "custom scripting" ?
I can't see any package requiring manually setting the version every release...

anyways the pip tools threaten that setup.py support may go away in the future... and that one "should" use pyproject.toml, but since it's completely static, we need to create a script to write the pyproject.toml.

I think I am missing something.

[petersilva]

@MagikEh maybe take a look, I vaguly recall you looking at this stuff in the past.

[petersilva]

another weird question: why do both requirements.txt and pyproject.toml exist? are they the same? do we need to create both?

[petersilva]

so confused... https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/ latest and greatest advice is to create setup.py, but when you use the tools, there are deprecation notices...

[petersilva]

this is about not letting our build/distribution tools be obsolete in the eyes of the python packaging community.
Unfortunately, I don't know what is the current recommendation.

[yt87]

There is a relevant discussion on discuss: https://discuss.python.org/t/user-experience-with-porting-off-setup-py/37502/128
My code uses numpy and scipy, so I try to follow those two projects. That is, I switched to meson-python. Here is one of my pyproject.toml:

# SPDX-FileCopyrightText: 2023 George Trojan
# SPDX-License-Identifier: LGPL-3.0-or-later
#
[build-system]
requires = ["meson-python >=0.13"]
build-backend = "mesonpy"

[project]
name = "getnam"
authors = [{name = "George Trojan", email = "george.trojan@gmail.com"}]
maintainers = [{name = "George Trojan", email = "george.trojan@gmail.com"}]
readme = "README.rst"
license = {file = "LICENCE"}
classifiers = [
    "License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)",
    "Programming Language :: Python",
    "Programming Language :: Python :: 3.11",
    "Topic :: Scientific/Engineering :: Atmospheric Science",
    "Topic :: Software Development :: Libraries :: Python Modules",
]
dynamic = ["version"]
description = "A script to retrieve 45km NAM model from NCEP servers"
requires-python = ">=3.11"
# dependencies handled by conda
#dependencies = [
#    "arhlib",
#    "rich-click",
#]

[project.optional-dependencies]
dev = [
    "mypy",
    "black",
    "ruff",
]

[project.scripts]
getnam = "getnam:cli"

[tool.black]
line-length = 96
target-version = ["py311"]

[tool.mypy]
files = "getnam.py"

[tool.ruff]
fix = true
select = ["B", "E", "EM", "F", "I", "NPY", "PLE", "PTH", "RET", "SIM", "W"]
ignore = []
unfixable = ["B"]
line-length = 96
target-version = "py311"

[tool.ruff.per-file-ignores]
"__init__.py" = ["E402"]

By the way, I think that sr3 would benefit from rich-click

[petersilva]

That reference was really interesting and helpful. We don't have any C in the python implementation. The C stuff is... um just straight C (not python at all no python packaging.) The meson-python seems to be mostly for people with compiled language modules in there.

The article seems to recommend hatchling for pure python projects... I tried the auto-migration... hmm... it changed my minimum python version from 3.6 to 3.7... I don't know if I can do that, I have to go back and look at what old OS's have. It also dropped all the dependency information... filled in many fields with strings that say "temporary" ... meaning, I think it wants me to replace them... sigh... this is a little project on it's own... stuck with it... I think I have something that is kind of ok:

there is an issue740_hatchling branch with a pyproject.toml. It seems to build the package OK, but it put in a lot of stuff I don't understand yet about running tests...
anyways looking at the branch you can see the .toml file.

https://github.com/MetPX/sarracenia/blob/issue740_hatchling/pyproject.toml

I think I caught everything that the setup.py had... so for coverage I guess it is ok.
Still need to confirm:

• can I use hatch on supercomputer/ppp redhat environments?
• Update Developer guide to document the difference process:
  * python3 -m build -> hatch build
  * twine upload -> hatch publish
• confirm it works with the pynsist (windows binary generator.)
• is it ok to bump the minimum version to 3.7? why did it do that?

irony:

• https://www.pyopensci.org/python-package-guide/package-structure-code/python-package-build-tools.html
• that links says about 40% of package use setuptools, and that is the most common one.

Hatch (hatchling? I can't tell the difference betwen the two.) is not popular, but it seems to work ok.

So... this is probably one option, once we work through the caveats above.

[petersilva]

• documentation works as-is... python -m build reads the .toml and uses hatch... so no documentation change.
• added the missing authors, and windows generation patch.
• snag: I don't understand how to produce an RPM package, needed for building for the supercomputer to be placed in system places.
• How to move from: "python3 setup.py bdist_rpm" to something pyproject.toml based?
  • Ideally, we don't build a .spec file, but all the fedora docs seem to start from a spec.
  • perhaps: https://opensource.com/article/22/6/package-python-module-rpm

[petersilva]

sigh... fedora-python/pyp2rpm#297

[yt87]

Two points:

1. For pure C/C++/Fortran and other meson is a replacement for autotools (and hand-crafted make), a competitor to cmake. meson-python was designed to build Python extensions, although it can be used for a single-file script.
2. I just successfully build sarracenia on linux (manjaro), from source within conda environment (python 3.12). The file sarracenia_env.yml mentioned in the documentation is missing, and the link is broken. I used requirements.txt which omits amqp.

[petersilva]

for 1... it's multiple entry points, not a single one, and there are libraries we expect users to use (so pipx is out.) so... it has a few more parts than a single script, but pure python.

for 2... Thanks, I changed the conda instructions to match your experience, and refer to requirements.txt

On omitting amqp, that's not a mistake. Sarracenia now supports AMQP and MQTT, so amqp is not a hard requirement, but an (admittedly super common) choice. You can use amqp or mqtt or both and mix and match.

On the other hand, there is a handy:

sr3 features

command you can use to inventory which features are present, and install the missing ones you need. Outstanding task is to add discussion of the features command to the install tutorials.

further reading #741

aside:

Packaging Requirements

I guess I need to state what we are asking of the packaging method. There a kind of four target environments for installation:

• debian/ubuntu os native packages. (main environnment for stable operational usage.)
• redhat native packages (for supercomputer)
• windows auto-installer (for very vanilla/canned usage.)
• native python packaging (pypi, pip, setuptools, hatch, etc... for people comfortable with python ecosystems.

native python packaging works great pretty much everywhere, and for python developers that's fine. That covers Conda, and venv, and other python envs pretty well. The main problem with python environments is security auditing and patching.

I'm not aware of any way to update for security issues in a python environment, you can blindly update everything, but there is no concept of patching to the latest stable version, or just patching the minimum to get security updates. You also need to audit every python venv, and conda, etc... individually as they have separate copies of dependencies, and so patching becomes a whack-a-mole excercise.

I prefer to use a single python environment per system, and leverage operating system packages whenever possible. Operating system vendors are issuing dozens to hundreds of security advisories per month, and it is infeasible to to look at each one and decide the ones to apply, and even if you do, you end up with a combinatoric problem if different versions of dependencies being tested with eachother. People doing due diligence (as required in most corporate production environments) are going to be applying security patches at least monthly. So folks should be going over every python venv they have, and updating everything in it, every month. I've never heard of anyone that does that...

A lot of users, however, don't want a python environment, they want a package they can use, like anything else installed on the system. OS packages are better for them. On debian/ubuntu, that is completely integrated and seamless, it's wonderful and easy. The debian packages are auto built for supported OS and architecture on
launchpad.net. if there is a vulnerability in libc, or some other package that sarracenia depends on, the os level apt update/apt upgrade will take care of it. The same mechanism covers all the python dependencies.

For redhat, it's the total opposite, there is no way I can see to build OS-native python packages there. It is always going to involve manual tweaking, duplicative config file building, and no or very bad dependency management. It's like they want the OS to be one thing, and applications to use python environments, and not the OS provided python.

For windows, using pynsist for now, and it breaks regularly... every release is a new puzzle... which dependency is missing now? but it leverages the normal python packaging, so not much of an exception, just the available dependencies seem to change in real-time.

[petersilva]

So I just had to build RPM packages for the supercomputer, and that refreshed my memory:

• we strip out the dependencies for the packages we cannot obtain in RPM format when building the rpm: xattr, watchdog, paramiko, magic.
• the rpm is built with python3 setup.py bdist_rpm. I don't know of any equivalent or analogous recipe with a more modern build method.

So that's the hangup. RPM building. the pyproject.toml file, using hatchling works fine, but I'm missing the ability to build RPMS.

[yt87]

You could ask at https://discuss.python.org. I have seen Fedora developers answering questions there.

[petersilva]

thanks, I looked at what's there, there is a 4 year old thread that ends with no answers, I posted there to wake it up.
https://discuss.python.org/t/tool-to-build-a-rpm-package-backed-by-pep-517/4020

[yt87]

So far, nobody responded. But it seems Fedora has macros for use with any build system. I have no idea whether those would be useful to you: https://koji.fedoraproject.org/koji/buildinfo?buildID=2297977.

[petersilva]

OK, the rpm spec file works fine, and how it works is documented in the Developers guide... on fedora 39. works great.
However, the main point of building RPMS for me is to install on the supercomputer, which is on redhat 8, python3.6 ... None of the recommendations used to build the .spec file work, and hatchling won't install. so... can't use any of it.

[petersilva]

ubuntu 18.04 similarly has python 3.6... so until that is retired (currently the most common deployment case.) We can't move forward on this.

[petersilva]

spawned #853