MAINT: use PEP440 vs. distutils #15256

tylerjereddy · 2021-12-20T21:52:05Z

Fixes #15254 (eventually)

version checking in a PEP440-compliant manner is
more robust than using distutils for this purpose
replace accordingly, mostly because the latest setuptools
has finally deprecated i.e., LooseVersion
I think the debate about whether we should use an external
package like packaging for this can be deferred since we
already vendor scipy/_lib/_pep440.py which says at the top:

The LooseVersion and StrictVersion classes that distutils provides don't
work; they don't recognize anything like alpha/beta/rc/dev versions.

that's a bit harsh for some of our purposes, but the CI is failing
with LooseVersion now
may need to be combined with i.e., MAINT: Remove distutils usage in runtests.py to fix deprecation warnings #15255 from Smit as well -- cleaning out the distutils stuff touches several places of course

* version checking in a PEP440-compliant manner is more robust than using `distutils` for this purpose * replace accordingly, mostly because the latest `setuptools` has finally deprecated i.e., `LooseVersion` * I think the debate about whether we should use an external package like `packaging` for this can be deferred since we already vendor `scipy/_lib/_pep440.py` which says at the top: ``` The LooseVersion and StrictVersion classes that distutils provides don't work; they don't recognize anything like alpha/beta/rc/dev versions. ``` * that's a bit harsh for some of our purposes, but the CI is failing with `LooseVersion` now

tylerjereddy · 2021-12-20T21:53:51Z

setup.py

@@ -234,7 +234,8 @@ def get_build_ext_override():
            from pythran.dist import PythranBuildExt
            BaseBuildExt = PythranBuildExt[npy_build_ext]
            # Version check - a too old Pythran will give problems
-            if LooseVersion(pythran.__version__) < LooseVersion('0.9.12'):
+            importlib.import_module('scipy/_lib/_pep440')


this is gruesome, and I'm not sure we are "allowed" to do this, but we cannot import from within scipy directly from within setup.py as far as I can tell (it dumps a bunch of chicken-and-egg type init errors)

There's a linefrom scipy._build_utils.system_info import get_info, NotFoundError inside configuration(), so it's not completely forbidden. But yeah, if it doesn't work then this workaround is fine.

Double checked, this should be fine.

I get build problems using the latest main (as of now):

Traceback (most recent call last): File "/home/nicpa/codes/scipy/setup.py", line 530, in <module> setup_package() File "/home/nicpa/codes/scipy/setup.py", line 514, in setup_package generate_cython() File "/home/nicpa/codes/scipy/setup.py", line 255, in generate_cython if _pep440.parse(pip.__version__) < _pep440.Version('18.0.0'): NameError: name '_pep440' is not defined

this seems related to this PR?

It seems this line is missing in line 255 as well...

I can make a PR if you want?

I wish it were that easy. It will still copy to a tmpdir (slow rebuilds), and hide the build log. I believe the closest equivalent to python setup.py develop is pip install -v --no-build-isolation --no-use-pep517 -e .. Which is a bit sad:(

Hmm... I see... Yeah, this is not really the pip way...

I'm still pondering on how we get an optimal CLI for the Meson build, because pip is unlikely to be the right answer.

And I have been following this closely, I am very much interested in seeing what Meson can do for Python builds. But I kind-of feel that this requires a complete restructure of the build infrastructure of Python if pip can't be patched. And then we go again with build-systems.

Perhaps the infrastructure should be in meson extensions instead?

Just some random thoughts.

Perhaps the infrastructure should be in meson extensions instead?

Yes, probably. I've also looked at Nox, which does a lot of things right but it seems to have the annoying habit of coupling everything it does with virtualenvs, and it's not clear on how to get the good parts of Nox without that.

I don't know nox, but just by having a look at its documentation page, it doesn't seem like it will fix anything?

To me it is important to be able to build against already installed packages. And many of these venv stuff needs customization flags for running custom builds... Hmm....

What I like from a tool like Nox (in principle) is that it'd give building blocks for things that we now do completely by hand in runtests.py and dev.py. We shouldn't need ~600 LoC to implement our own developer CLI.

To me it is important to be able to build against already installed packages.

Yes, this should be the default for any sane interface. You set up a build env once (which can be expensive), and then you build/test/benchmark over and over.

Yes! :)

There's a reason there are so many alternatives, a stable all-across solution is not easy :-D

tylerjereddy · 2021-12-20T21:54:56Z

I may also have misunderstood when to use _pep440.parse vs. _pep440.Version -- I think I was just copying from the few examples already in the code base..

tylerjereddy · 2021-12-20T23:16:20Z

some Windows pythran problems in CI as well here

Smit-create · 2021-12-21T06:56:29Z

Linux tests failures have a workaround and are fixed in gh-15255

rgommers

LGTM, in it goes. Thanks @tylerjereddy

think the debate about whether we should use an external package like packaging for this can be deferred since we already vendor scipy/_lib/_pep440.py which says at the top:

There's no debate I think, we'll always vendor simple utilities like this:)

rgommers · 2021-12-22T20:26:39Z

Added as backport candidate.

tylerjereddy added the maintenance Items related to regular maintenance tasks label Dec 20, 2021

tylerjereddy requested review from larsoner, person142 and rgommers as code owners December 20, 2021 21:52

tylerjereddy commented Dec 20, 2021

View reviewed changes

rgommers approved these changes Dec 22, 2021

View reviewed changes

rgommers merged commit 31c8ba4 into scipy:master Dec 22, 2021

rgommers added the backport-candidate This fix should be ported by a maintainer to previous SciPy versions. label Dec 22, 2021

rgommers added this to the 1.8.0 milestone Dec 22, 2021

rgommers modified the milestones: 1.8.0, 1.9.0 Dec 22, 2021

tylerjereddy deleted the treddy_issue_15254 branch December 22, 2021 21:51

tylerjereddy modified the milestones: 1.9.0, 1.8.0 Jan 16, 2022

tylerjereddy removed the backport-candidate This fix should be ported by a maintainer to previous SciPy versions. label Jan 17, 2022

saudet mentioned this pull request Feb 4, 2022

CI: some Azure cleanups and reactivating Pythran on Windows #15281

Closed

3 tasks

gatagat mentioned this pull request Feb 13, 2022

Remove deprecated LooseVersion from distutils gatagat/lap#36

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MAINT: use PEP440 vs. distutils #15256

MAINT: use PEP440 vs. distutils #15256

tylerjereddy commented Dec 20, 2021

tylerjereddy Dec 20, 2021

rgommers Dec 20, 2021

rgommers Dec 22, 2021

zerothi Jan 12, 2022

zerothi Jan 12, 2022

zerothi Jan 13, 2022

rgommers Jan 13, 2022

zerothi Jan 13, 2022

rgommers Jan 13, 2022

zerothi Jan 13, 2022

tylerjereddy commented Dec 20, 2021

tylerjereddy commented Dec 20, 2021

Smit-create commented Dec 21, 2021

rgommers left a comment

rgommers commented Dec 22, 2021

MAINT: use PEP440 vs. distutils #15256

MAINT: use PEP440 vs. distutils #15256

Conversation

tylerjereddy commented Dec 20, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tylerjereddy commented Dec 20, 2021

tylerjereddy commented Dec 20, 2021

Smit-create commented Dec 21, 2021

rgommers left a comment

Choose a reason for hiding this comment

rgommers commented Dec 22, 2021