Build dependencies don't appear to support environment markers

[pv]

• Pip version: 10.0.0
• Python version: 3.6.5
• Operating system: Linux/Fedora

Description:

The requires build dependency listings in pyproject.toml doesn't appear to support PEP 508 environment markers.

Several projects appear to have expected pip would support this: pandas, scipy, pywt,

What I've run:

The following reproducer shell script

set -e -x

rm -rf mytest
mkdir mytest
cd mytest

python3 -mvenv env
./env/bin/pip install --upgrade 'pip>=10'

echo "assert False" > setup.py
cat <<EOF > pyproject.toml
[build-system]
requires = [
    "wheel",
    "setuptools",
    "numpy==1.12.1;python_version<'3'",
    "numpy==1.14.1;python_version>='3'",
]
EOF

./env/bin/pip wheel .

This prints

+ rm -rf mytest
+ mkdir mytest
+ cd mytest
+ python3 -mvenv env
+ ./env/bin/pip install --upgrade 'pip>=10'
Collecting pip>=10
  Using cached pip-10.0.0-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 9.0.3
    Uninstalling pip-9.0.3:
      Successfully uninstalled pip-9.0.3
Successfully installed pip-10.0.0
+ echo 'assert False'
+ cat
+ ./env/bin/pip wheel .
Processing /home/pauli/tmp/foo2/mytest
  Installing build dependencies ... error
  Complete output from command /home/pauli/tmp/foo2/mytest/env/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-5bwq_74_ https://pypi.python.org/packages/1b/d2/22cde5ea9af055f81814f9f2545f5ed8a053eb749c08d186b369959189a8/wheel-0.31.0-py2.py3-none-any.whl#md5=240d714477a715bcd90e94cb2c44f28c https://pypi.python.org/packages/20/d7/04a0b689d3035143e2ff288f4b9ee4bf6ed80585cc121c90bfd85a1a8c2e/setuptools-39.0.1-py2.py3-none-any.whl#md5=ca299c7acd13a72e1171a3697f2b99bc https://pypi.python.org/packages/b1/e2/884cfbfd4f21b2313210d1d2ea72ecc381b98826d1b7e6606929ac6c0a08/numpy-1.12.1-cp36-cp36m-manylinux1_x86_64.whl#md5=fbebdc68b7698e00c07bf4ddae0fb717 https://pypi.python.org/packages/de/7d/348c5d8d44443656e76285aa97b828b6dbd9c10e5b9c0f7f98eff0ff70e4/numpy-1.14.1-cp36-cp36m-manylinux1_x86_64.whl#md5=dd2321ea4590ec05d825d8c9a64fd64b:
  Double requirement given: numpy==1.14.1 from https://pypi.python.org/packages/de/7d/348c5d8d44443656e76285aa97b828b6dbd9c10e5b9c0f7f98eff0ff70e4/numpy-1.14.1-cp36-cp36m-manylinux1_x86_64.whl#md5=dd2321ea4590ec05d825d8c9a64fd64b (already in numpy==1.12.1 from https://pypi.python.org/packages/b1/e2/884cfbfd4f21b2313210d1d2ea72ecc381b98826d1b7e6606929ac6c0a08/numpy-1.12.1-cp36-cp36m-manylinux1_x86_64.whl#md5=fbebdc68b7698e00c07bf4ddae0fb717, name='numpy')
  
  ----------------------------------------
Command "/home/pauli/tmp/foo2/mytest/env/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-5bwq_74_ https://pypi.python.org/packages/1b/d2/22cde5ea9af055f81814f9f2545f5ed8a053eb749c08d186b369959189a8/wheel-0.31.0-py2.py3-none-any.whl#md5=240d714477a715bcd90e94cb2c44f28c https://pypi.python.org/packages/20/d7/04a0b689d3035143e2ff288f4b9ee4bf6ed80585cc121c90bfd85a1a8c2e/setuptools-39.0.1-py2.py3-none-any.whl#md5=ca299c7acd13a72e1171a3697f2b99bc https://pypi.python.org/packages/b1/e2/884cfbfd4f21b2313210d1d2ea72ecc381b98826d1b7e6606929ac6c0a08/numpy-1.12.1-cp36-cp36m-manylinux1_x86_64.whl#md5=fbebdc68b7698e00c07bf4ddae0fb717 https://pypi.python.org/packages/de/7d/348c5d8d44443656e76285aa97b828b6dbd9c10e5b9c0f7f98eff0ff70e4/numpy-1.14.1-cp36-cp36m-manylinux1_x86_64.whl#md5=dd2321ea4590ec05d825d8c9a64fd64b" failed with error code 1 in None

I would have expected it picks the a single appropriate numpy version based on the requirements markers, instead of trying to install two different versions and failing. (The test script will ultimately fail when it tries to run the setup.py --- in my real use I of course have a working setup.py but the behavior is the same).

[benoit-pierre]

I have a patch for this, I'll PR it later, extras are not supported either.

[pfmoore]

@dstufft @pradyunsg Are we sure this should be a release blocker targeted on 10.0.1? IMO this comes under "PEP 518 support is currently incomplete" and could be aimed at 10.1. There's currently a serious distlib bug (#5223) which I think warrants an emergency 10.0.1 fix to go into Python 3.7, but I'm not sure this does. Surely a simple workaround is for these projects to not rush into using PEP 518 features that we don't support yet?

OTOH, I've no objection to including extra fixes in 10.0.1 if they are low-impact, so depending on the complexity of the PR @benoit-pierre is preparing, my concern may be for nothing :-)

[benoit-pierre]

How do you specify extras when installing a wheel from an URL?

[pradyunsg]

Format it into a valid PEP 508 specifier? That should work.

[pradyunsg]

name[extras]@url ; markers

[benoit-pierre]

With a tarball, this work:

> (cd tests/data/src/requires_simple_extra && python setup.py sdist)
> rm -rf tmp && PYTHONPATH=$PWD/src python -m pip install -v -t tmp "file://$PWD/tests/data/src/requires_simple_extra/dist/requires_simple_extra-0.1.tar.gz#egg=requires_simple_extra[extra]"

But your suggestion does not work:

> rm -rf tmp && PYTHONPATH=$PWD/src python -m pip install -v -t tmp "requires_simple_extra[extra]@file://$PWD/tests/data/src/requires_simple_extra/dist/requires_simple_extra-0.1.tar.gz"

And with a wheel, this does not work either:

> rm -rf tmp && PYTHONPATH=$PWD/src python -m pip install -v -t tmp "requires_simple_extra[extra]@file://$PWD/tests/data/packages/requires_simple_extra-0.1-py2.py3-none-any.whl"

Additionally the fact that pip prints a warning if an extra is invalid (instead of erroring out) is going to be problematic when a build requirement is invalid.

[pfmoore]

@benoit-pierre requested that #5245 be deferred till 10.1, so I'm removing the release blocker status from this issue. It's simply part of "PEP 518 support is incomplete at the moment"

[pradyunsg]

@pfmoore I just realized this should be mentioned in the docs -- the part talking about PEP 518. I can create a PR now, if it's not too late.

[pfmoore]

@pradyunsg It's not too late. I'm not doing the release right now, just tidying stuff up in preparation.

[apoliakov]

Hey all just in case - I was looking for a workaround and looks like this did the trick:

pip install pandas --no-build-isolation

It sat in compilation for a while but eventually cleared out and appears to work now. Maybe this helps the next affected :)

[pradyunsg]

On Sat, 21 Apr 2018, 05:01 Alex Poliakov, ***@***.***> wrote: Hey guys is there a recommended workaround for this?

One workaround is to pass --no-build-isolation and manually installing the build time requirements of the package.

…

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5230 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADH7SXzPykYwx4z1cB_oNEKGgIfh1fkyks5tqm-2gaJpZM4TVTC3> .

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.