dogpile.cache 0.9.1 introduced pep517 regression

[ssbarnea]

dogpile.cache 0.9.1 introduced pep517 regression which prevented its installation on some setups:

Collecting dogpile.cache>=0.6.5
  Downloading http://mirror.gra1.ovh.openstack.org/pypifiles/packages/e6/65/fa4a947926034cb4af5f469c376a21728b7fd8de4668fed757c3586f0eef/dogpile.cache-0.9.1.tar.gz (329 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'

:stderr:   ERROR: Command errored out with exit status 1:
   command: /home/zuul/test-python/bin/python /home/zuul/test-python/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpx9zntfdw
       cwd: /tmp/pip-install-9ep3va87/dogpile.cache
  Complete output (10 lines):
  Traceback (most recent call last):
    File "/home/zuul/test-python/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 257, in <module>
      main()
    File "/home/zuul/test-python/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 240, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/zuul/test-python/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 85, in get_requires_for_build_wheel
      backend = _build_backend()
    File "/home/zuul/test-python/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 76, in _build_backend
      obj = getattr(obj, path_part)
  AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /home/zuul/test-python/bin/python /home/zuul/test-python/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpx9zntfdw Check the logs for full command output.
WARNING: You are using pip version 20.0.2; however, version 20.1 is available.

Full report at https://zuul.opendev.org/t/openstack/build/0a2a5ca6414f4e56b9508322cf01c9f4

[zzzeek]

hi there -

there's been extended discussion about the pep517 issue here: sqlalchemy/sqlalchemy#5207

short answer is that dogpile.cache has added pyproject.toml as well, which turns on pip's pep517 mode, which then has all kinds of issues that overall will have to be resolved on the pip side and/or the community at large. however there is no documented incompatibility about including a pyproject.toml file alongside setup.py .

we can seek to remove pyproject.toml from the build but for now, however this is only a workaround. it is likely that there is something in your build environment that's stale or incompatible with pep517, try using a fresh install environment. If you run pip with --no-use-pep517 that would allow the install to proceed.

[ssbarnea]

@zzzeek Thanks for the quick reply. Lets keep issue open until we find a better solution. Use of pyproject.toml should not break installation, for example I already use it in projects like https://github.com/ansible-community/molecule/blob/master/pyproject.toml or https://github.com/ansible/ansible-lint/blob/master/pyproject.toml and they are not failing to install, on the same setups as dogpile.cache.

I think we may need bit of tuning to make both worlds happy.

[zzzeek]

on the SQLAlchemy side we simply removed pyproject.toml from the MANIFEST file so that it's not in the build. SQLAlchemy however builds C extensions so has a much more complicated situation.

for dogpile, there's nothing unusual in setup.py, it's about as vanilla as it gets. What sort of "tunings" did you have in mind?

[CaselIT]

I think that the problem may be an old version of setuptools.
Probably removing it like in alembic/sqlalchemy for now should be ok.

Another option is to use pep517 specifying in the pyproject.toml

[build-system]	
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

Likely also specifying a min version for setuptools.

[CaselIT]

But I'm not sure if the bug on debian regarding the broken pep517 package would still happen with that configuration.

All things considered it's probably best to sit out until it's in general use, since each time it was enabled, we got multiple issues regarding it, so it's clearly not ready for prime time yet.

[ssbarnea]

I am already testing patch that adds required missing bits to pyproject.toml from the #179 at https://review.opendev.org/#/c/724691/ -- we will know soon.

It is a very common mistake to forget to add [build-system] when adding this file and we do know that missing these produces lots of failures. The proposed values where taken from molecule, as the packaging installation is tested across a wide range of distros.

[zzzeek]

I dont understand why this is needed when pip states they have set up a sensible default for this: https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support is their default broken ?

[CaselIT]

the problem here is that I think that the sensible default does not install an updated version of setuptools. ops I guess

[zzzeek]

e.g., just like last time, this REALLY looks like a bug in pip/ setuptools / etc. they have documented a default, and it seems to break all over the place. if their default is unworkable, they should change it, or fix setuptools, or whatever.

[zzzeek]

but their default includes a minimum version of setuptools "setuptools>=40.8.0" - is that wrong ? did they mean to update it?

[ssbarnea]

@zzzeek I did not test this project with older setuptools, I only copy/pasted values that I needed on other projects (which where raised due to specific bugs).

So, shortly it may work file even with older versions of both pip and setuptools. Still, who has time to test all possible combinations? Considering that these are not runtime requirements, it should be safer to use more recent versions when building the wheel.

Just to be clear for everyone: these requirements will not alter your system pip/setuptools when you build or install the package.

[zzzeek]

as mentioned on the PR safest approach as we did in SQLAlchemy seems to omit this file from the builds. I just remain pretty frustrated that this ongoing issue is not considered to be a bug in pip.

[FedericoRessi]

I created this workaround for OpenStack projects: https://review.opendev.org/725209

[zzzeek]

the irony is dogpile.cache is actually CI tested in centos8 containers, so no intrinsic incompatibility there. ill just release without the pyproject.toml in the dist.

[sqla-tester]

Mike Bayer has proposed a fix for this issue in the master branch:

Omit pyproject.toml from installation https://gerrit.sqlalchemy.org/c/sqlalchemy/dogpile.cache/+/1941

[FedericoRessi]

I can confirm the problem is also happening when installing dogpile.cache on RHEL 8

[zzzeek]

0.9.2 is released which should not include this file