Replaced all uses of distutils with setuptools #428

agronholm · 2021-12-24T20:06:03Z

This gets rid of any (direct) distutils dependencies by relying on setuptools instead. It does mean we depend on setuptools as an install dependency but I don't see that as a problem.

Distutils has been deprecated and will be removed in Python 3.12.

The sysconfig.get_platform() and distutils.util.get_platform() functions return "linux-x64_64", not "linux_x64_64" which the previous code and tests assumed.

codecov · 2021-12-24T20:07:03Z

Codecov Report

Merging #428 (d859302) into main (afc0d5e) will increase coverage by 0.54%.
The diff coverage is 82.92%.

@@            Coverage Diff             @@
##             main     #428      +/-   ##
==========================================
+ Coverage   67.03%   67.57%   +0.54%     
==========================================
  Files          11       12       +1     
  Lines         904      916      +12     
==========================================
+ Hits          606      619      +13     
+ Misses        298      297       -1

Impacted Files	Coverage Δ
src/wheel/cli/__init__.py	`23.52% <ø> (-3.26%)`	⬇️
src/wheel/cli/convert.py	`38.70% <66.66%> (-0.40%)`	⬇️
src/wheel/bdist_wheel.py	`49.63% <69.23%> (ø)`
src/wheel/_setuptools_logging.py	`91.66% <91.66%> (ø)`
src/wheel/util.py	`100.00% <100.00%> (ø)`
src/wheel/wheelfile.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update afc0d5e...d859302. Read the comment docs.

agronholm · 2021-12-24T20:17:20Z

I missed one distutils import in wheelfile.py. Fixing.

Setuptools will now always be present so this is pointless. The pkg_resources module was never imported here anyway.

jaraco · 2021-12-26T18:42:10Z

src/wheel/util.py

@@ -30,3 +31,14 @@ def as_bytes(s):
        return s.encode("utf-8")
    else:
        return s
+
+
+def log(msg, *, error=False):


Rather than implementing a custom logger, I'd prefer to re-use Python's logging module. Does that not work in this scenario because the logger is not initialized? :/

Indeed. Even setuptools still uses the distutils logger internally.

I think this is a defect that needs to be addressed in Setuptools. It should supply a log object similar to what distutils does (but probably using Python's logging module).

Ok, but until that's in place, do you see any problem with using this approach now? It should be functionally equivalent to the distutils logging code (just with a newer syntax and leveraging print().

In pypa/setuptools#2973, I've captured the need for a replacement. I realize that Setuptools won't be able to supply a replacement for distutils in such a way that it's available on older Setuptools releases, so wheel will probably not be able to rely on it for a while. Therefore, I'd recommend that wheel provide an interface-compatible fallback, and perhaps just a copy of the implementation that makes it into Setuptools. I may be able to get that implementation out today.

Great, looking forward to it.

In pypa/setuptools#2974, I believe I have a technique that will configure the Python logging mechanism to log to stderr/stdout as distutils.log currently does.

Then, in https://github.com/pypa/wheel/tree/python-logging, I illustrate how this approach could be employed in wheel to log through that mechanism. I could have even restored the logger name and % substitutions, making the diff from main very small, but opted to leave those changes.

The problem with that approach is that for older Setuptools, nothing will have configured the logger, so the default configuration of the logger will apply. In https://github.com/pypa/wheel/tree/python-logging-compat, I apply another commit that copies the code that configures the logger and run that. That approach will give a compatible effect, but still won't honor any calls to distutils.log.set_threshold on older Setuptools. That seems like an acceptable tradeoff (and one this PR has already made).

Please have a look at the Setuptools PR and then consider pulling one or both of the aforementioned commits for this PR.

Thanks! I'll look at those branches later.

…ug/warning levels to logged messages. Fixed missing f-string for 'wheelfile_path'.

henryiii · 2021-12-30T15:43:37Z

Not for right now, but would using pypa/flit instead of setuptools be better? I know a few other tools are planning on switching (packaging IIRC?), and that would get rid of the chicken/egg problem and allow a pyproject.toml file again. flit_core is entirely self contained, I believe. Just a thought.

agronholm · 2021-12-30T16:16:56Z

Can we rely on ubiquitous PEP 517 support everywhere? That would be a requirement for using flit, yes? Anybody whose systems run python setup.py install or similar would have a bad time. Maybe it's best to ditch setup.py altogether then?

henryiii · 2021-12-30T17:08:13Z

Packages will need to soon. It would be great if installer was ready - maybe that's the last piece needed? Systems are beginning to get serious about ditching setup.py commands - possibly spurred on by https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html, and possibly by the inclusion of Flit into the PyPA.

henryiii · 2021-12-30T18:41:04Z

Here's the discussion I was remembering: https://discuss.python.org/t/debundling-the-next-pip-release-will-require-handling-pyproject-toml-based-build-backends/12329

packaging and build likely will make the switch at some point in 2022. If you are in the PyPA discord, we just had a bit of discussion in off-topic.

for more information, see https://pre-commit.ci

henryiii · 2021-12-30T20:03:19Z

Adding future annotations works in the wild, it seems. :)

agronholm · 2021-12-30T20:08:01Z

If you are in the PyPA discord, we just had a bit of discussion in off-topic.

I didn't know there is a PyPA discord server. Where was that announced?

henryiii · 2021-12-30T20:11:09Z

I think there's a discuss.python.org about it? There's a badge in the readme here: https://github.com/pypa/setuptools. The PyPA projects are encouraged to add badges and request channels. There's a "nice" URL for it, but I don't remember what it is.

henryiii · 2021-12-30T20:47:09Z

Here it is: https://discuss.python.org/t/theres-a-pypa-discord-now

agronholm · 2021-12-30T22:37:25Z

@jaraco I think this is ready for merging, yes?

jaraco · 2021-12-30T22:45:35Z

I say go for it!

agronholm · 2021-12-30T22:49:07Z

Thanks everyone!

agronholm · 2022-01-02T13:36:35Z

Not for right now, but would using pypa/flit instead of setuptools be better? I know a few other tools are planning on switching (packaging IIRC?), and that would get rid of the chicken/egg problem and allow a pyproject.toml file again. flit_core is entirely self contained, I believe. Just a thought.

My plan is to have every packaging back-end depend on wheel after the public API is released. That would mean that the back-end wheel itself uses would have to vendor wheel. That's a bummer, but I can't immediately think of a way around that.

henryiii · 2022-01-02T14:02:08Z

flit-core already venders dependencies (only packaging, IIRC)

pfmoore · 2022-01-02T14:16:31Z

An alternative would be to include a custom in-tree backend. It only has to implement build_sdist and build_wheel, and you could hard-code the files to include and use the wheel API to create the wheel. (That's actually an interesting project - I might look at writing a "self-hosting" backend like this myself...)

henryiii · 2022-01-02T14:29:06Z

This is how flit-core itself is built (using itself): https://github.com/pypa/flit/blob/main/flit_core/pyproject.toml

pfmoore · 2022-01-02T15:06:46Z

Oh cool! I hadn't realised flit had done that (yet). I knew they had been doing some work on bootstrapping, but not that they'd separated the build backend out totally.

I guess this means that flit-core won't ever use the API from wheel to build wheels, but that seems totally fine to me, for something that is deliberately intended as a "lowest possible level" backend. After all, the whole point in having the standard is that we don't tie people into having to use a single implementation.

agronholm · 2022-01-02T15:08:06Z

I guess this means that flit-core won't ever use the API from wheel to build wheels

If this is true, then it means I could restore pyproject.yaml for the upcoming release.

pfmoore · 2022-01-02T16:29:16Z

If you switch to flit for the build backend, then yes, I guess it does.

agronholm · 2022-01-02T18:12:07Z

I have a PR for that now: #436. Reviews are welcome.

mayeut · 2022-01-03T11:57:20Z

@gaborbernat, this might break pypa/virtualenv in some corner cases in the future:

Using --no-setuptools without --no-wheel
Using a version of setuptools not compatible with wheel (the specifier here is quite old already so, as long as it is the case, shouldn't be much of an issue but if it changes, that might be another story, e.g. a patch version of wheel requires a version of setuptools that's too recent to be considered by periodic updates => virtual environment is broken, wheel is not usable)

IMHO those are really corner cases but I thought I'd give you a head's up.

gaborbernat · 2022-01-03T12:22:19Z

Thanks for the update, if this goes ahead we should disallow installing wheel without setuptools in virtualenv. A perhaps in that sense at that point in time would be welcome, but indeed is an edge case.

agronholm · 2022-01-03T12:32:36Z

Thanks for the update, if this goes ahead we should disallow installing wheel without setuptools in virtualenv

This adds an install dependency of setuptools to wheel, so do you need to do anything?

gaborbernat · 2022-01-03T12:45:08Z

Of course, virtualenv provisions seed packages that until today have no dependencies. If you're adding dependencies now virtualenv needs to make sure they are satisfied post provision. This adds a whole new complexity to the project. We've had this discussion when you added packaging dependency to the project a while back.

agronholm added 2 commits December 24, 2021 21:56

Replaced distutils with setuptools

55dfd58

Distutils has been deprecated and will be removed in Python 3.12.

Fixed detection of 32-bit Python on 64-bit OS

7d898b7

The sysconfig.get_platform() and distutils.util.get_platform() functions return "linux-x64_64", not "linux_x64_64" which the previous code and tests assumed.

agronholm requested a review from jaraco December 24, 2021 20:06

agronholm added 2 commits December 24, 2021 22:20

Fixed one more distutils import

ea3a53a

Got rid of require_pkgresources()

83eff2c

Setuptools will now always be present so this is pointless. The pkg_resources module was never imported here anyway.

jaraco reviewed Dec 26, 2021

View reviewed changes

Merge branch 'main' into remove-distutils

13cc63b

jaraco mentioned this pull request Dec 26, 2021

Replacement needed for distutils.log pypa/setuptools#2973

Closed

jaraco and others added 5 commits December 26, 2021 16:27

Replace bespoke logger with Python logging logger, restoring info/deb…

2540fef

…ug/warning levels to logged messages. Fixed missing f-string for 'wheelfile_path'.

Ensure logging is configured even on older Setuptools versions.

1632618

Wrapped long lines

1d4ec33

Reformatted code with black

355be15

Merge branch 'main' into remove-distutils

d55ec4d

agronholm and others added 2 commits December 30, 2021 22:02

Merge branch 'main' into remove-distutils

a594385

[pre-commit.ci] auto fixes from pre-commit.com hooks

3aeefcb

for more information, see https://pre-commit.ci

agronholm added 2 commits December 31, 2021 00:30

Merge branch 'main' into remove-distutils

5be6e9f

Fixed a typo

d859302

agronholm merged commit 31d8972 into main Dec 30, 2021

agronholm deleted the remove-distutils branch December 30, 2021 22:48

natsukium mentioned this pull request May 2, 2024

Replace deprecated distutils with sysconfig htseq/htseq#84

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replaced all uses of distutils with setuptools #428

Replaced all uses of distutils with setuptools #428

agronholm commented Dec 24, 2021

codecov bot commented Dec 24, 2021 •

edited

agronholm commented Dec 24, 2021

jaraco Dec 26, 2021

agronholm Dec 26, 2021

jaraco Dec 26, 2021

agronholm Dec 26, 2021

jaraco Dec 26, 2021

agronholm Dec 26, 2021

jaraco Dec 26, 2021

agronholm Dec 26, 2021

henryiii commented Dec 30, 2021

agronholm commented Dec 30, 2021

henryiii commented Dec 30, 2021

henryiii commented Dec 30, 2021

henryiii commented Dec 30, 2021

agronholm commented Dec 30, 2021

henryiii commented Dec 30, 2021

henryiii commented Dec 30, 2021

agronholm commented Dec 30, 2021

jaraco commented Dec 30, 2021

agronholm commented Dec 30, 2021

agronholm commented Jan 2, 2022

henryiii commented Jan 2, 2022

pfmoore commented Jan 2, 2022

henryiii commented Jan 2, 2022

pfmoore commented Jan 2, 2022

agronholm commented Jan 2, 2022

pfmoore commented Jan 2, 2022

agronholm commented Jan 2, 2022

mayeut commented Jan 3, 2022

gaborbernat commented Jan 3, 2022

agronholm commented Jan 3, 2022

gaborbernat commented Jan 3, 2022

Replaced all uses of distutils with setuptools #428

Replaced all uses of distutils with setuptools #428

Conversation

agronholm commented Dec 24, 2021

codecov bot commented Dec 24, 2021 • edited

Codecov Report

agronholm commented Dec 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

henryiii commented Dec 30, 2021

agronholm commented Dec 30, 2021

henryiii commented Dec 30, 2021

henryiii commented Dec 30, 2021

henryiii commented Dec 30, 2021

agronholm commented Dec 30, 2021

henryiii commented Dec 30, 2021

henryiii commented Dec 30, 2021

agronholm commented Dec 30, 2021

jaraco commented Dec 30, 2021

agronholm commented Dec 30, 2021

agronholm commented Jan 2, 2022

henryiii commented Jan 2, 2022

pfmoore commented Jan 2, 2022

henryiii commented Jan 2, 2022

pfmoore commented Jan 2, 2022

agronholm commented Jan 2, 2022

pfmoore commented Jan 2, 2022

agronholm commented Jan 2, 2022

mayeut commented Jan 3, 2022

gaborbernat commented Jan 3, 2022

agronholm commented Jan 3, 2022

gaborbernat commented Jan 3, 2022

codecov bot commented Dec 24, 2021 •

edited