Support PEP 600 tags #293

mattip · 2020-04-08T13:28:45Z

xref gh-280. This adds a whole bunch of PEP 600 tags (my python reports `os.confstr('CS_GNU_LIBC_VERSION') of 2.27): EDIT: after reversing the order in f432bcd

>>> from packaging import tags
>>> print ('\n'.join([str(t) for t in tags.sys_tags()]))
cp37-cp37m-manylinux_2_27_x86_64
cp37-cp37m-manylinux_2_26_x86_64
cp37-cp37m-manylinux_2_25_x86_64
cp37-cp37m-manylinux_2_24_x86_64
cp37-cp37m-manylinux_2_23_x86_64
cp37-cp37m-manylinux_2_22_x86_64
cp37-cp37m-manylinux_2_21_x86_64
cp37-cp37m-manylinux_2_20_x86_64
cp37-cp37m-manylinux_2_19_x86_64
cp37-cp37m-manylinux_2_18_x86_64
cp37-cp37m-manylinux_2_17_x86_64
cp37-cp37m-manylinux2014_x86_64
cp37-cp37m-manylinux_2_16_x86_64
cp37-cp37m-manylinux_2_15_x86_64
cp37-cp37m-manylinux_2_14_x86_64
cp37-cp37m-manylinux_2_13_x86_64
cp37-cp37m-manylinux_2_12_x86_64
cp37-cp37m-manylinux2010_x86_64
cp37-cp37m-manylinux_2_11_x86_64
cp37-cp37m-manylinux_2_10_x86_64
cp37-cp37m-manylinux_2_9_x86_64
cp37-cp37m-manylinux_2_8_x86_64
cp37-cp37m-manylinux_2_7_x86_64
cp37-cp37m-manylinux_2_6_x86_64
cp37-cp37m-manylinux_2_5_x86_64
cp37-cp37m-manylinux1_x86_64
cp37-cp37m-linux_x86_64
cp37-abi3-manylinux_2_27_x86_64
...

mattip · 2020-04-08T14:14:49Z

It seems I need to mock some failures to get coverage to 100%

brettcannon · 2020-04-08T18:28:41Z

Thanks for taking a stab at this! I haven't looked at the code, but a tweak we might want to make to the output is to emit the manylinux1, manylinux2010, and manylinux2014 tags after the matching perennial manylinux tag. We also want to emit the tags in priority order, so newer glibc versions should come first.

mattip · 2020-04-09T17:34:41Z

Ahh, so the correct order, assuming precedence for the perennial tags, would be this?

cp37-cp37m-manylinux_2_27_x86_64
cp37-cp37m-manylinux_2_26_x86_64
cp37-cp37m-manylinux_2_25_x86_64
cp37-cp37m-manylinux_2_24_x86_64
cp37-cp37m-manylinux_2_23_x86_64
cp37-cp37m-manylinux_2_22_x86_64
cp37-cp37m-manylinux_2_21_x86_64
cp37-cp37m-manylinux_2_20_x86_64
cp37-cp37m-manylinux_2_19_x86_64
cp37-cp37m-manylinux_2_18_x86_64
cp37-cp37m-manylinux_2_17_x86_64
cp37-cp37m-manylinux2014_x86_64
cp37-cp37m-manylinux_2_16_x86_64
cp37-cp37m-manylinux_2_15_x86_64
cp37-cp37m-manylinux_2_14_x86_64
cp37-cp37m-manylinux_2_13_x86_64
cp37-cp37m-manylinux_2_12_x86_64
cp37-cp37m-manylinux2010_x86_64
cp37-cp37m-manylinux_2_11_x86_64
cp37-cp37m-manylinux_2_10_x86_64
cp37-cp37m-manylinux_2_9_x86_64
cp37-cp37m-manylinux_2_8_x86_64
cp37-cp37m-manylinux_2_7_x86_64
cp37-cp37m-manylinux_2_6_x86_64
cp37-cp37m-manylinux_2_5_x86_64
cp37-cp37m-manylinux1_x86_64
cp37-cp37m-linux_x86_64

brettcannon · 2020-04-09T18:09:25Z

@mattip yep, exactly! Basically the way to think about the order is if I got every wheel file name for a specific version, I should be able to iterate through what sys_tags() returns and as soon as I have a tag match with a file I stop looking because I found the best possible match for my environment.

This is also why everything in packaging.tags is a generator: so you can short-circuit the search and potentially avoid costly environment checks.

mattip · 2020-04-10T07:55:58Z

I reversed the tags and added tests for lines that weren't in the coverage report. ~~However coverage is now saying the critical part of the code that I added is not covered. How can that be when the tests fail if I remove it?~~ Edit: fixed this in c7d1851.

mattip · 2020-04-16T09:05:17Z

Any thoughts? CI is passing, even though travis thinks it is still pending.

brettcannon · 2020-04-16T22:06:46Z

Sorry, I've personally been swamped lately on top of vacation, so I just haven't had time to review the PR yet.

brettcannon

Just some tweaks to generalize this to > glibc 2.

packaging/tags.py

brettcannon · 2020-04-17T22:01:36Z

packaging/tags.py

+                    plat, (2, glibc_minor)
+                )
+                if manylinux_compat:
+                    yield linux.replace("linux", plat)


Instead of doing the replacing, maybe the map should just hard-code all possible supported architectures since there aren't that many.

It would only save one instance of replace, the PEP 600 tags do not use the map.

mattip · 2020-04-20T06:35:32Z

Tests are passing after fixes as per the review. The code should support newer major versions of glibc, but there still is a logic failure. We assume sys_tags can emit the entire set of valid tags as listed in the comments above. However, if the platform has moved to glibc3.2, what is the highest Y for manylinux_2_Y? We have no way of knowing what the "last glibc2.Y" is. I could just "assume" an arbitrary large number (99?) and a future PR could adapt that to the situation when it arises?

brettcannon · 2020-04-21T21:09:16Z

Huh, good point about the cap on glibc 2's minor number if glibc 3 ever becomes a thing. Maybe we can't generalize this for glibc 3 without hard-coding that? I guess the best fall-back is to go with your suggestion, Matti, and just assume 99 as an upper-bound and if the day comes that glibc 3 exists we can hard-code it going forward. That way we somewhat future-proof for people who don't upgrade while not making it ridiculous for us forever.

mattip · 2020-04-28T11:47:48Z

Added a _LAST_GLIBC_MINOR default dictionary that defaults to 50, and generalized the looping to handle maximum versions of each glibc major version (including using (2, 5) or (2,17) as the last one, depending on architecture). Probably overkill, but should not cause failures. Of course this assumes glibc will make good on their backward-compatibility promise.

packaging/tags.py

tests/test_tags.py

packaging/tags.py

mattip · 2020-04-28T13:10:10Z

Requested changes made, there is one test failure with no log.

pradyunsg · 2020-04-28T15:39:48Z

Triggered a re-run for all jobs.

pradyunsg · 2020-04-28T15:46:04Z

Triggered a "bigger" re-run. :)

mattip · 2020-04-28T16:06:40Z

Trying again

mattip

The refactor to allow calling a _manylinux.manylinux_compatible function required a bit of refactoring. So basically we are back to a full review, sorry.

packaging/tags.py

tests/test_tags.py

mattip · 2020-05-03T23:05:21Z

All but one macOS job passes

mattip · 2020-05-15T16:06:16Z

Any thoughts?

brettcannon · 2020-05-15T19:10:06Z

Any thoughts?

That I wish I had less deadlines at work? 😉

This is at the top of my review queue, but this month is not good for me for open source time.

brettcannon

Very small change to make to use monkeypatch in a test and then a clarifying question on a comment. Otherwise LGTM!

packaging/tags.py

brettcannon · 2020-05-22T20:09:13Z

tests/test_tags.py

+    def test_module_declaration(self, manylinux_module, attribute, glibc, tf):
+        manylinux = "manylinux{}_compatible".format(attribute)
+        setattr(manylinux_module, manylinux, tf)
+        res = tags._is_manylinux_compatible(manylinux, "x86_64", glibc)
+        assert tf is res
+        delattr(manylinux_module, manylinux)


It's easiest to use pytest's monkeypatch for temporarily setting an attribute:

Suggested change

def test_module_declaration(self, manylinux_module, attribute, glibc, tf):

manylinux = "manylinux{}_compatible".format(attribute)

setattr(manylinux_module, manylinux, tf)

res = tags._is_manylinux_compatible(manylinux, "x86_64", glibc)

assert tf is res

delattr(manylinux_module, manylinux)

def test_module_declaration(self, monkeypatch, manylinux_module, attribute, glibc, tf):

manylinux = "manylinux{}_compatible".format(attribute)

monkeypatch.setattr(manylinux_module, "manylinux", tf)

res = tags._is_manylinux_compatible(manylinux, "x86_64", glibc)

assert tf is res

I tried that. monkypatch cannot use setattr on an attribute that was not already present, I get

> monkeypatch.setattr(manylinux_module, manylinux, tf) E AttributeError: <module '_manylinux'> has no attribute \ 'manylinux2014_compatible'

Try using raising=False.

tests/test_tags.py

mattip · 2020-05-23T18:54:19Z

Sorry, I lost the thread of comments that this question relates to:

Do you have something specific you want to change that you haven't and so the comment is on its previous state? Or is there something you want to do that you haven't yet?

brettcannon · 2020-05-23T19:52:32Z

tests/test_tags.py

+    def test_module_declaration(self, manylinux_module, attribute, glibc, tf):
+        manylinux = "manylinux{}_compatible".format(attribute)
+        setattr(manylinux_module, manylinux, tf)
+        res = tags._is_manylinux_compatible(manylinux, "x86_64", glibc)
+        assert tf is res
+        delattr(manylinux_module, manylinux)


Try using raising=False.

brettcannon · 2020-05-23T19:54:45Z

@mattip

Sorry, I lost the thread of comments that this question relates to:
Do you have something specific you want to change that you haven't and so the comment is on its previous state? Or is there something you want to do that you haven't yet?

It's in response to a comment on test_tags.py, line 587 (if you look under the "Files Changed" tab you can find it again):

This is a weird test. Previously it depended on the monkeypatch on line 42 to set _have_compatible_glibc in order to reject any glibc value.

mattip · 2020-05-24T05:27:14Z

Tests are passing with monkeypatch and raising=False

brettcannon · 2020-05-26T19:05:28Z

@pradyunsg did you want to do another review, or should I go ahead and merge this?

pradyunsg · 2020-05-26T22:08:42Z

Go ahead. I'm tied up with college reports and submissions for the next few days.

brettcannon · 2020-05-27T18:38:37Z

Thanks for all the work on this, @mattip !

mattip force-pushed the pep-600 branch 2 times, most recently from 958576a to 372650d Compare April 8, 2020 14:09

brettcannon added the packaging.tags label Apr 8, 2020

brettcannon self-requested a review April 8, 2020 18:28

mattip force-pushed the pep-600 branch from 59231fe to f432bcd Compare April 10, 2020 07:45

mattip mentioned this pull request Apr 16, 2020

Tracking issue for PEP 600 Perennial manylinux rollout pypa/manylinux#542

Closed

18 tasks

brettcannon closed this Apr 16, 2020

brettcannon reopened this Apr 16, 2020

brettcannon requested changes Apr 17, 2020

View reviewed changes

mattip force-pushed the pep-600 branch 2 times, most recently from c7b2582 to 3e8aa45 Compare April 19, 2020 20:23

pradyunsg reviewed Apr 28, 2020

View reviewed changes

packaging/tags.py Show resolved Hide resolved

tests/test_tags.py Outdated Show resolved Hide resolved

packaging/tags.py Show resolved Hide resolved

packaging/tags.py Outdated Show resolved Hide resolved

pradyunsg closed this Apr 28, 2020

pradyunsg reopened this Apr 28, 2020

mattip closed this Apr 28, 2020

mattip reopened this Apr 28, 2020

refactor for _manylinux module changes

32293aa

mattip force-pushed the pep-600 branch from 538d9f5 to 32293aa Compare May 3, 2020 21:59

mattip commented May 3, 2020

View reviewed changes

parametrize tests for coverage

002af44

mattip mentioned this pull request May 5, 2020

ARM builds? pypa/manylinux#84

Closed

brettcannon requested review from brettcannon and pradyunsg May 5, 2020 20:27

pradyunsg closed this May 7, 2020

pradyunsg reopened this May 7, 2020

mayeut mentioned this pull request May 8, 2020

Allow manylinux_x_y wheel upload (PEP600) pypi/warehouse#7853

Merged

brettcannon requested changes May 22, 2020

View reviewed changes

brettcannon requested changes May 23, 2020

View reviewed changes

mattip added 2 commits May 23, 2020 23:40

fixes from review

1f97b89

blackify

dab14e3

brettcannon self-requested a review May 25, 2020 02:05

Merge branch 'master' into pep-600

fda98fc

brettcannon approved these changes May 26, 2020

View reviewed changes

brettcannon merged commit 28a2e2b into pypa:master May 27, 2020

di mentioned this pull request Oct 21, 2020

Release 20.5 #345

Closed

xrogaan mentioned this pull request Jan 19, 2021

Upgrade version of vendored package 'packaging' to support PEP 600 pypa/pipenv#4591

Closed

mattip mentioned this pull request Feb 28, 2021

Manylinux 2 24 MacPython/numpy-wheels#108

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support PEP 600 tags #293

Support PEP 600 tags #293

mattip commented Apr 8, 2020 •

edited

mattip commented Apr 8, 2020

brettcannon commented Apr 8, 2020

mattip commented Apr 9, 2020

brettcannon commented Apr 9, 2020

mattip commented Apr 10, 2020 •

edited

mattip commented Apr 16, 2020

brettcannon commented Apr 16, 2020

brettcannon left a comment

brettcannon Apr 17, 2020

mattip Apr 19, 2020

mattip commented Apr 20, 2020

brettcannon commented Apr 21, 2020

mattip commented Apr 28, 2020

mattip commented Apr 28, 2020

pradyunsg commented Apr 28, 2020

pradyunsg commented Apr 28, 2020

mattip commented Apr 28, 2020

mattip left a comment

mattip commented May 3, 2020

mattip commented May 15, 2020

brettcannon commented May 15, 2020

brettcannon left a comment

brettcannon May 22, 2020

mattip May 23, 2020

brettcannon May 23, 2020

mattip commented May 23, 2020

brettcannon May 23, 2020

brettcannon commented May 23, 2020

mattip commented May 24, 2020

brettcannon commented May 26, 2020

pradyunsg commented May 26, 2020

brettcannon commented May 27, 2020

Support PEP 600 tags #293

Support PEP 600 tags #293

Conversation

mattip commented Apr 8, 2020 • edited

mattip commented Apr 8, 2020

brettcannon commented Apr 8, 2020

mattip commented Apr 9, 2020

brettcannon commented Apr 9, 2020

mattip commented Apr 10, 2020 • edited

mattip commented Apr 16, 2020

brettcannon commented Apr 16, 2020

brettcannon left a comment

Choose a reason for hiding this comment

brettcannon Apr 17, 2020

Choose a reason for hiding this comment

mattip Apr 19, 2020

Choose a reason for hiding this comment

mattip commented Apr 20, 2020

brettcannon commented Apr 21, 2020

mattip commented Apr 28, 2020

mattip commented Apr 28, 2020

pradyunsg commented Apr 28, 2020

pradyunsg commented Apr 28, 2020

mattip commented Apr 28, 2020

mattip left a comment

Choose a reason for hiding this comment

mattip commented May 3, 2020

mattip commented May 15, 2020

brettcannon commented May 15, 2020

brettcannon left a comment

Choose a reason for hiding this comment

brettcannon May 22, 2020

Choose a reason for hiding this comment

mattip May 23, 2020

Choose a reason for hiding this comment

brettcannon May 23, 2020

Choose a reason for hiding this comment

mattip commented May 23, 2020

brettcannon May 23, 2020

Choose a reason for hiding this comment

brettcannon commented May 23, 2020

mattip commented May 24, 2020

brettcannon commented May 26, 2020

pradyunsg commented May 26, 2020

brettcannon commented May 27, 2020

mattip commented Apr 8, 2020 •

edited

mattip commented Apr 10, 2020 •

edited