Issue in macOS binary wheel for 4.6.4/3.8: Illegal Instruction (my fault, but needs a change to prevent in the future)

[jamadden]

I'm working on moving RelStorage from Travis CI to GitHub actions. I got to the point of being able to import RelStorage, and the process crashed importing persistent with an Illegal Instruction error.

Fatal Python error: Illegal instruction

Current thread 0x000000011119edc0 (most recent call first):
...
  File "//site-packages/persistent/_compat.py", line 81 in _c_optimizations_available
  File "//site-packages/persistent/_compat.py", line 182 in find_impl
  File "//site-packages/persistent/_compat.py", line 190 in use_c_impl
  File "//site-packages/persistent/interfaces.py", line 27 in <module>
...
  File "//site-packages/persistent/__init__.py", line 33 in <module>
...
  File "//relstorage/src/relstorage/adapters/adapter.py", line 26 in <module>

I've seen this previously with other binary wheels built on macOS when the binary was built on a newer computer than the destination and using -march=native or similar optimizations.

And it turns out that there are two wheels for Python 3.8 published on PyPI for 4.6.4, one that zest.releaser built and uploaded for me (cp38-10.15), and one that came from Travis (cp38-10.9). My environment includes -march=native in CFLAGS et al.

I will yank the bad wheel, solving the immediate problem.

In other projects (gevent), I've used a zest.releaser hook to check the environment for that setting and forbid uploading a binary wheel built that way.

But IIRC zest.releaser now also has a way to say "just upload the source, never a binary", and that might be the better thing to do here, if we can continue relying on CI to build wheels.

[icemac]

Could it be that the Windows wheel also has a problem? See https://ci.appveyor.com/project/zopefoundation/zope/builds/37667794/job/tor6448qgs9sqgll where I got an Access is denied while installing it.

[icemac]

Hm, when using the AppVeyor config and tox.ini on Zope master Python 3.8 builds fine: https://ci.appveyor.com/project/zopefoundation/zope/builds/37679866/job/nunxuv3hiy0scvg9

[jamadden]

Could it be that the Windows wheel also has a problem?

I don't think so. The error here was a miscompilation. The 'access denied' error is from Windows not letting you delete a file that's in use while building BTrees from source — it didn't even try to use the binary wheel.

Installed c:\users\appveyor\appdata\local\temp\1\easy_install-cra246i4\btrees-4.7.2\.eggs\persistent-4.6.4-py3.8-win32.egg
error: [WinError 5] Access is denied: 'C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\easy_install-cra246i4\\BTrees-4.7.2\\temp\\easy_install-_cpkd18a\\persistent-4.6.4\\.eggs\\cffi-1.14.4-py3.8-win32.egg\\_cffi_backend.cp38-win32.pyd'

Here, BTrees has an install_requires= that specifies it depends on persistent, which in turn depends on CFFI. So the process running setuptools installed those things in the local .eggs directory and went about doing its thing to build BTrees. It looks like when this was done it went to clean up the .eggs directory, but because the dynamic library files were already loaded into the process, it couldn't do that.

I think one solution would be to have those dependencies installed already. Another solution might be to have BTrees move away from install_requires to pyproject.toml (because pip uses a bunch of different processes to manage that), but that would break buildout.

I don't know why the earlier job had to build BTrees from scratch, while the latter build apparently got the BTrees binary wheel from PyPI.

[mgedmin]

I'm pretty sure ./.eggs gets used for setup_requires rather than install_requires (and BTrees has a setup_requires=['persistent']).

Ideally we would use pyproject.toml's introspectable build requirements instead of setup_requires. Or in addition to. I don't think older versions of zc.buildout support pyproject.toml, and I'm mostly only hoping that zc.buildout 3.0 will due to using pip as the install backend.