Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
Specify ABI for pantsbuild.pants wheel and build with both UCS2 and UCS4 #7235
We should be marking the ABI (application binary interface) for the
In particular, in Python 2, Python may be installed with either UCS2 (UTF-16) or UCS4 (UTF-8). We should be marking the wheel as either
As a result of marking the ABI, we must now produce more wheels. macOS defaults to UCS2. For Linux, "ucs4 is much more widespread among Linux CPython distributions." We do not want to rely on these assumptions, however, when releasing, as some users may not have these default unicode settings. So, instead we must release
At a high level, this PR does two things:
To setup the new shards, we use Pyenv to install new versions of Python 2 with the appropriate unicode settings, thanks to the env var
Because both the OSX UCS4 shard and Linux UCS2 shard already have Python 2.7 installed, we must install a Python 2.7.x version different than what is already there, and use
Specifically, we make these changes to achieve these two high level goals:
Ensuring the correct abi is used
We need to ensure the
An even better test would test the result of
We now properly mark the ABI for Python 2. Beyond the new script
In addition to correctness for Python 2, this unblocks releasing Python 3 wheels (#7197).
Note this should have no significant impact on the end user, as Pip will resolve to the current ABI for their interpreter. It will change the name of our
Downside: wheel building explosion
We currently are building more wheels than necessary. For wheels that are universal / platform-independent, we only need them to be built once, but we build them every time. See #7258.
This PR adds two new shards so adds ~30 unnecessary core wheels we build, in addition to 3rd party wheels that are universal / platform-independent.
I think we're all a bit light on the details here. Suffice it to say that a 2.7 interpreter built with ucs2 still handles encoding and decoding utf-8 strings, at least for simple test cases. I'm pretty sure we need to support both. The need comes from the fact that we are a bad python citizen. You cannot grab our sdist and build, we have to have burned the shared library you need or you're out of luck and are forced to clone our repo and hand-build.
Yes, agreed with @jsirois that UCS2 surprisingly works even though we assume UTF-8 everywhere. We know this because Python 2 on macOS defaults to UCS2 when using system Python or installed via Pyenv—the user must go out of their way to set
referenced this pull request
Feb 19, 2019
cosmicexplorer left a comment
Can there be a test verifying that all of these wheels we're building can actually be resolved and used correctly with all of the abis? If it requires having a python interpreter with that abi, is that something we can set up another docker image(s) to do?