Newly compile pyenv on macos broken with: Expected in: flat namespace

[ssbarnea]

I am on High Sierra and I recently discovered that some precompiled wheels stopped working with the newer pyenv versions. Initially I found the errors only with py37 but after upgradinf the other py35,py36 versions it started to reproduce on them too.

I am almost sure that this is caused by the way pyenv compiles python so it endsup being binary incompatible with manylinux wheels.

How to reproduce bug:

pip3 install pycrytpo
python3 -c "from Crypto.Util import _counter"

>>> Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: dlopen(/Users/ssbarnea/.pyenv/versions/3.7.1/lib/python3.7/site-packages/Crypto/Util/_counter.cpython-37m-darwin.so, 2): Symbol not found: _PyInt_FromLong
Referenced from: /Users/ssbarnea/.pyenv/versions/3.7.1/lib/python3.7/site-packages/Crypto/Util/_counter.cpython-37m-darwin.so
Expected in: flat namespace
in /Users/ssbarnea/.pyenv/versions/3.7.1/lib/python3.7/site-packages/Crypto/Util/_counter.cpython-37m-darwin.so

Please note that this has nothing to do with pycrypto package itself as I found the same kind of error on other libraries.

It may work mentioning some environment variables defined on the system:

CFLAGS='-I/usr/local/include -L/usr/local/lib -I/usr/local/opt/openssl/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include'
CPPFLAGS='-I/usr/local/opt/zlib/include -I/usr/local/opt/openssl/include -I/usr/local/opt/nss/include -I/usr/local/opt/icu4c/include  -I/usr/local/opt/libarchive/include -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/ -I/usr/local/opt/sqlite/include'
LDFLAGS='-L/usr/local/opt/zlib/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/nss/lib  -L/usr/local/opt/libarchive/lib -L/usr/local/opt/icu4c/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/sqlite/lib'

As I tried to investigate the issue myself I came across https://stackoverflow.com/a/51384999/99834 which sounds interesting.

Any ideas?
otool-pycrypto.log

[ssbarnea]

I created a script that tests the bug and also dumps python compile time paramerters, so we can compare them and find out what is causing the breakage. See https://gist.github.com/ssbarnea/6308a1ac61662bb22f5389afe89476ba

Because I didn't see too many similar reports I assume that problem is limited to local configuration like compiler flags and libraries, some of them being installed via brew, as recommended.

[ssbarnea]

I found the problem and is not pyenv itself is in fact caused by pip caching which would compile the package first time you install it and keep a compiled version. Later you install a new python interpreter, maybe even a minor version upgrage and start using it. On the new interpreter once you try to install the package pip will take "advantage" of the chache and install the precompiled wheel.... and surprise you get something that may not work with your python.

While I will raise this issue with pip and ask them to assure that caching is isolated per interpreter compile hash -- one that should use sysconfig.get_config_var(), I think that we need to do something about pyenv too as pyenv is far more likely to cause this problem that other environments.

I woud propose to remove pip cache on each pyenv installation. It may perform some slowdowns but limited ones as this operation is not done very often, better to have a correct slower cache than a fast and bad one.

[native-api]

Pip seems to have addressed better wheel tagging or smth like that in pypa/pip#7319 .