Reshape our packaging system

[skoudoro]

Our current system start to be quite old. We might encounter multiple problems in the close future.

Furthermore:

• distutils is deprecated (PEP-632)
• it is not recommended anymore to call directly setup.py. See this article
• setup.cfg will be 'soon' deprecated in favor of pyproject.toml (see the whole setuptools thread Discussion: support for pyproject.toml configuration pypa/setuptools#1688 (comment))

the build tools environment currently undergoes both a rapid development and unification. Unification comes mainly from PEP-517, PEP-518, PEP-621 and PEP-631.

I suppose we should start using pyproject.toml and review/clean our current system.

check also pep660 for the editable install

some reading:

https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html
https://snarky.ca/what-the-heck-is-pyproject-toml/
https://bernat.tech/posts/growing-pain/

[skoudoro]

Also, we need to update our current versioning system. I am hesitating between setuptools_scm and versioneer.

Any opinion?

[arokem]

I've used setuptools_scm in a couple of projects, and I like it. It's fairly straightforward to set up and has the advantage that we wouldn't have to vendor the entire thing into our codebase. But I think that versioneer is possibly a bit more flexible.

[skoudoro]

Thank you @arokem for the feedback! I will look at setuptools_scm.

Also, I confirm that our CI's are failing when we build the package using python setup.py install (it use distutils / homemade method) but CI's works well when we build the package using pip install .

That's why we have so many failing CI's.

[matthew-brett]

I've used Versioneer a lot - and I wouldn't recommend it. It starts to cause fiddly problems when the tag comes from Git information directly. I'm pretty sure Ralf Gommers would agree - he often complains about it, in Scipy.

For the build system - Scipy is in the process of switching to Meson - it looks nice, and a good way of making really flexible build rules - but it's still not smooth for wheel creation etc.

[skoudoro]

Thank you for the feedback @matthew-brett!

I did not know about meson. I will start to study it, it looks like an interesting option.

For people looking at this issue, there is an interesting thread about meson in Scipy: scipy/scipy#13615

[CAM-Gerlach]

FYI, there's also @henryiii 's CMake-based @scikit-build, which fills a similar niche in the scientific Python community with various upsides and downsides relative to Meson for this.

[eli-schwartz]

scikit-build essentially wraps cmake inside setuptools, IIRC it is a long-term goal to fully untether it from setuptools, but currently the listed discussion points won't actually be solved by scikit-build -- you'll still have setup.py/setup.cfg, rely on distutils, etc.

[henryiii]

Best link to read about current plans are at https://iscinumpy.dev/post/scikit-build-proposal/. ATM, I think it's quite usable as long as you aren't doing anything too complicated in setup.py (which doesn't normally happen with scikit-build, logic is in CMake), but you still (as mentioned above) have the downsides for now. If you do, I'd like to know about it so I have more test cases for the rewrite if it's funded. :) We are able to keep it working and tested, supporting various platforms, etc. You can at least use pypa/build and PEP 517/518 requirements with it today, just like setuptools, avoiding raw setup.py <command> calls.

I think end-of-the-year or so, there will (hopefully) be really good options via scikit-build and mesonpy.

[casperdcl]

running into an issue trying to pip install dipy (AMYPAD/AmyPET#33). I presume dipy is missing a pyproject.toml with at least:

[build-system]
requires = ["setuptools>=42", "packaging", "numpy"]
# optional, as it's the default:
# build-backend = “setuptools.build_meta”

[henryiii]

Neither packaging nor numpy have a build system. I expect you want setuptools and build-backend = “setuptools.build_meta” too. Scikit-build-core and meson-python are both good choices now as well.

[CAM-Gerlach]

Additionally, you'll want to heed the guidance in the build-time dependency section of the NumPy docs and specify the minimum version of NumPy you support at runtime to build against, or just use the oldest-supported-numpy metapackage.