SOABI support for Python 2.X and PyPy #3075

natefoo · 2015-09-03T16:42:01Z

Additionally, fix the version portion of the Python tag on wheels built with PyPy that use the Python API. It will now be the Python major version concatenated with the PyPy major and minor versions.

Fixes #2671, #2882.

xrefs:

dstufft · 2015-09-03T19:20:01Z

pip/pep425tags.py

+    impl = get_abbr_impl()
+    if not soabi and impl in ('cp', 'pp'):
+        d = 'd' if hasattr(sys, 'pydebug') and sys.pydebug else ''
+        m = 'm' if impl == 'cp' else ''


Isn't this inaccurate? This doesn't tell us if Python was compiled with --pymalloc or not does it?

Maybe using something like sysconfig.get_config_var('WITH_PYMALLOC') would let us inspect it.

Sorry, that bit came from Nick's comments here. sysconfig.get_config_var('WITH_PYMALLOC') does indeed seem to work, updating shortly.

natefoo · 2015-09-04T04:02:26Z

I verified the ABI flags with all subsets of dmu on Pythons 2.6 and 2.7, and dm on 3.2, 3.3, 3.4, and 3.5.

dstufft · 2015-09-11T11:59:12Z

Hmm, I wonder what the effect of merging this would be. If we merge it in pip install wheels made by a wheel without this patch applied? What about the inverse, if we merge it in wheel, will pip without this patch install those wheels?

qwcode · 2015-09-11T17:05:46Z

ditto what @dstufft said, and also would be nice to add tests for this.

the duplication of this pep425tags module in pip and wheel is pretty odd.

maybe move it to https://github.com/pypa/packaging , which pip vendors, and then wheel could depend on packaging.

natefoo · 2015-09-15T19:10:54Z

@dstufft wheels built with an unmodified wheel can be installed with the modified pip. Wheel will generate e.g. SQLAlchemy-1.0.4-cp27-none-linux_x86_64.whl, pip will prefer e.g. SQLAlchemy-1.0.4-cp27-cp27mu-linux_x86_64.whl but ABI none will be second in the list of supported tags.

Default wheels built with a modified wheel cannot be installed with the unmodified pip since on 2.x, unmodified pip will not match any wheel with an ABI tag other than none. Other than defaulting wheel to produce none ABI wheels (and doing that seems counterproductive), I am not sure how to prevent this.

@qwcode Agree regarding the duplication, and there will be more when the enhanced platform detection for Linux lands. The two pep425tags versions are not exactly the same but near enough that they could be recombined. I'll work on adding some tests.

dstufft · 2015-09-15T19:17:17Z

@natefoo That's fine, it just means we should probably make sure that pip is released prior to wheel.

dholth · 2015-09-17T14:45:10Z

pep425tags was sortof supposed to be a thing vendored into wheel and into pip, but it has diverged. https://bitbucket.org/dholth/pep425tags

natefoo · 2015-09-17T15:10:36Z

I have changed the way wide unicode detection works because it was doing the wrong thing on Python >= 3.3 (in the unlikely event that the SOABI config var is unset).

I also added some unit tests for the manual ABI flag detection.

anthrotype · 2015-10-09T07:29:11Z

👍

dstufft · 2015-10-11T23:17:26Z

Can you merge in develop into this or rebase it please?

Additionally, fix the version portion of the Python tag on wheels built with PyPy that use the Python API. It will now be the Python major version concatenated with the PyPy major and minor versions. Fixes pypa#2671, pypa#2882.

- Catch IOError wherever sysconfig is used (pypa#1074) - Reapply pypa#2915 Also, be sure to normalize the SOABI

…d `d` ABI flags.

Pythons with PEP 393 Flexible String Representation (so >= 3.3). Granted, on these Pythons, the SOABI config var should always be set, but the manual SOABI code path should still try to do the right thing.

natefoo · 2015-10-13T14:40:24Z

@dstufft Ok, I rebased it.

dstufft · 2015-10-13T15:38:57Z

Hmm, I'm not sure why that test is failing. It doesn't seem related to this PR, but it's passing on develops last build on Travis it appears.

Another thing I thought of though while reviewing this again. It's currently not handling the case where 2.x has unicode disabled. Right now this will just be treated as a narrow unicode build. I wonder if this is a case that we should care about or not? It's not handled by PEP3149 because that's specific to Python 3.x and 3.x doesn't allow disabling unicode.

The other thing I noticed, it's going to treat a missing config variable the same as not having a feature enabled. Is that the right behavior? Will it return False typically and only None if something is wrong? If it's only None if something is wrong, should we log a warning incase someone gets something that is not compatible?

natefoo · 2015-10-13T16:05:39Z

I'm getting the same error if I run the tests locally against develop.

Unfortunately there's no flag for "no unicode", so adding one would require a standards update, correct?. FWIW I have never encountered a no-unicode Python in the wild. Perhaps to avoid the standards issue we could drop the ABI tag back to "none" for no-unicode builds?

For the config variable issue, I could warn and also fall back to the "old" logic for detecting the flags, as laid out here and as implemented in the earlier commits in this PR.

dstufft · 2015-10-13T16:11:49Z

It would probably require a standards update yes, going back to none might be reasonable in that case (it's certainly no worse then where we are today anyways). Warning and falling back is probably reasonable there too.

natefoo · 2015-10-13T18:05:31Z

Okay, about done with the changes. Two things:

I can't find any way to detect a --with-pymalloc build other than checking the WITH_PYMALLOC config var, so the fallback code will just assume it is, as Nick suggested (as long as the interpreter is CPython). Hopefully the warning RuntimeWarning: Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect will be enough guidance here.
In trying to figure out how to detect a no-unicode build, I realized I actually don't know of a way to create a no-unicode Python 2 build. Is this possible? --disable-unicode does not work, and configuring without --enable-unicode* simply creates a UCS-2 build.

dstufft · 2015-10-13T18:11:21Z

Hmm, I thought --disable-unicode was the way to do it, maybe I'm wrong (or maybe it broke and nobody noticed).

natefoo · 2015-10-13T18:16:52Z

Ahh, looks like it works for 2.7.10. 2.6.9's ./configure fails on: checking what type to use for unicode... configure: error: invalid value for --enable-unicode. Use either ucs2 or ucs4 (lowercase).

dstufft · 2015-10-13T18:42:53Z

Awesome.

Assuming pymalloc with a warning is probably fine. Ideally this case shouldn't get hit.

Sent from my iPhone

On Oct 13, 2015, at 2:16 PM, Nate Coraor notifications@github.com wrote:

Ahh, looks like works for 2.7.10. 2.6.9's ./configure fails on: checking what type to use for unicode... configure: error: invalid value for --enable-unicode. Use either ucs2 or ucs4 (lowercase).

—
Reply to this email directly or view it on GitHub.

are unavailable, but issue a warning if this is used. 2. Explicitly handle the case where the unicode detection finds wide unicode but this is a 3.3+ build (necessary due to #1) 3. Fix tests broken due to pypa#2.

natefoo · 2015-10-13T20:02:48Z

Ok, I pushed a new change that creates a fallback if config vars are unavailable, and forces a none tag if sys.maxunicode does not exist.

For the record, a --disable-unicode install of 2.7.10 is very broken - make install fails and even once you work around that, standard modules don't import due to all the missing attributes under sys. Maybe it worked better with earlier Pythons. But get_abi_tag() will nonetheless do the right thing when sys.maxunicode does not exist.

dstufft · 2015-10-13T20:04:35Z

Ha. Probably it would have been fine. I've never had a Unicode-less build, I just knew it was an option. Thanks for making those changes. I'll give it another review when I get home.

Sent from my iPhone

On Oct 13, 2015, at 4:02 PM, Nate Coraor notifications@github.com wrote:

Ok, I pushed a new change that creates a fallback if config vars are unavailable, and forces a none tag if sys.maxunicode does not exist.

For the record, a --disable-unicode install of 2.7.10 is very broken - make install fails and even once you work around that, standard modules don't import due to all the missing attributes under sys. Maybe it worked better with earlier Pythons. But get_abi_tag() will nonetheless do the right thing when sys.maxunicode does not exist.

—
Reply to this email directly or view it on GitHub.

SOABI support for Python 2.X and PyPy

dstufft · 2015-10-14T21:59:23Z

Ok merged. Wheel should wait to pull this change in until pip 8 is released.

dstufft · 2015-10-14T21:59:40Z

Or unless we release it in another version I guess.

natefoo · 2015-10-14T22:52:16Z

Thanks @dstufft!

dstufft reviewed Sep 3, 2015
View reviewed changes

natefoo added 5 commits October 13, 2015 10:38

SOABI support for Python 2.X and PyPy

0b9afea

Additionally, fix the version portion of the Python tag on wheels built with PyPy that use the Python API. It will now be the Python major version concatenated with the PyPy major and minor versions. Fixes pypa#2671, pypa#2882.

Fix unit tests I broke:

a70e1dd

- Catch IOError wherever sysconfig is used (pypa#1074) - Reapply pypa#2915 Also, be sure to normalize the SOABI

Use the WITH_PYMALLOC and Py_DEBUG config vars for setting the m an…

62e265e

…d `d` ABI flags.

Setting the u ABI flag based on sys.maxunicode is incorrect on recent

a55fa5a

Pythons with PEP 393 Flexible String Representation (so >= 3.3). Granted, on these Pythons, the SOABI config var should always be set, but the manual SOABI code path should still try to do the right thing.

Add unit tests for manual SOABI detection

1b5b4ac

natefoo force-pushed the py2-soabi branch from c79ff25 to 1b5b4ac Compare October 13, 2015 14:39

natefoo mentioned this pull request Oct 13, 2015

Replace eggs with wheels galaxyproject/galaxy#428

Merged

17 tasks

1. Provide a fallback mechanism for determining ABI flags if config vars

63d239f

are unavailable, but issue a warning if this is used. 2. Explicitly handle the case where the unicode detection finds wide unicode but this is a 3.3+ build (necessary due to #1) 3. Fix tests broken due to pypa#2.

dstufft added a commit that referenced this pull request Oct 14, 2015

Merge pull request #3075 from natefoo/py2-soabi

461b7aa

SOABI support for Python 2.X and PyPy

dstufft merged commit 461b7aa into pypa:develop Oct 14, 2015

lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 3, 2019

lock bot locked as resolved and limited conversation to collaborators Jun 3, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SOABI support for Python 2.X and PyPy #3075

SOABI support for Python 2.X and PyPy #3075

natefoo commented Sep 3, 2015

dstufft Sep 3, 2015

dstufft Sep 3, 2015

natefoo Sep 4, 2015

natefoo commented Sep 4, 2015

dstufft commented Sep 11, 2015

qwcode commented Sep 11, 2015

natefoo commented Sep 15, 2015

dstufft commented Sep 15, 2015

dholth commented Sep 17, 2015

natefoo commented Sep 17, 2015

anthrotype commented Oct 9, 2015

dstufft commented Oct 11, 2015

natefoo commented Oct 13, 2015

dstufft commented Oct 13, 2015

natefoo commented Oct 13, 2015

dstufft commented Oct 13, 2015

natefoo commented Oct 13, 2015

dstufft commented Oct 13, 2015

natefoo commented Oct 13, 2015

dstufft commented Oct 13, 2015

natefoo commented Oct 13, 2015

dstufft commented Oct 13, 2015

dstufft commented Oct 14, 2015

dstufft commented Oct 14, 2015

natefoo commented Oct 14, 2015

SOABI support for Python 2.X and PyPy #3075

SOABI support for Python 2.X and PyPy #3075

Conversation

natefoo commented Sep 3, 2015

dstufft Sep 3, 2015

Choose a reason for hiding this comment

dstufft Sep 3, 2015

Choose a reason for hiding this comment

natefoo Sep 4, 2015

Choose a reason for hiding this comment

natefoo commented Sep 4, 2015

dstufft commented Sep 11, 2015

qwcode commented Sep 11, 2015

natefoo commented Sep 15, 2015

dstufft commented Sep 15, 2015

dholth commented Sep 17, 2015

natefoo commented Sep 17, 2015

anthrotype commented Oct 9, 2015

dstufft commented Oct 11, 2015

natefoo commented Oct 13, 2015

dstufft commented Oct 13, 2015

natefoo commented Oct 13, 2015

dstufft commented Oct 13, 2015

natefoo commented Oct 13, 2015

dstufft commented Oct 13, 2015

natefoo commented Oct 13, 2015

dstufft commented Oct 13, 2015

natefoo commented Oct 13, 2015

dstufft commented Oct 13, 2015

dstufft commented Oct 14, 2015

dstufft commented Oct 14, 2015

natefoo commented Oct 14, 2015