Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PEP 518 support: setup.py is called before build requirements have been installed #4647

Closed
benoit-pierre opened this issue Aug 5, 2017 · 11 comments · Fixed by #4799
Closed
Labels
auto-locked Outdated issues that have been locked by automation type: bug A confirmed bug or unintended behavior
Milestone

Comments

@benoit-pierre
Copy link
Member

benoit-pierre commented Aug 5, 2017

  • Pip version: 9.0.1-464-g573a5011
  • Python version: Python 3.6.2
  • Operating system: Arch Linux x86_64

Description:

I'd like to use pip' support for PEP 518, particularly to solve those 2 cases mentioned in the PEP:

  • While setuptools itself will install anything listed in this, they won't be installed until during the execution of the setup() function, which means that the only way to actually use anything added here is through increasingly complex machinations that delay the import and usage of these modules until later on in the execution of the setup() function.
  • This cannot include setuptools itself nor can it include a replacement to setuptools , which means that projects such as numpy.distutils are largely incapable of utilizing it and projects cannot take advantage of newer setuptools features until their users naturally upgrade the version of setuptools to a newer one.

For this to work I expect build requirements to be installed before any attempt at running setup.py.

What I've run:

> tmpdir="$(mktemp -d)"
> trap "cd - && rm -rf $tmpdir" EXIT
> cd "$tmpdir"
> python -m venv venv
> export PATH="$PWD/venv/bin:$PATH"
> pip install -U setuptools wheel https://github.com/pypa/pip/archive/573a50118148ff4646ae9310730f5bf6c4e8784f.zip python-xlib
Collecting https://github.com/pypa/pip/archive/573a50118148ff4646ae9310730f5bf6c4e8784f.zip
  Downloading https://github.com/pypa/pip/archive/573a50118148ff4646ae9310730f5bf6c4e8784f.zip (6.5MB)
    100% |████████████████████████████████| 6.5MB 269kB/s
Collecting setuptools
  Using cached setuptools-36.2.7-py2.py3-none-any.whl
Collecting wheel
  Using cached wheel-0.29.0-py2.py3-none-any.whl
Collecting python-xlib
  Using cached python_xlib-0.19-py2.py3-none-any.whl
Collecting six>=1.10.0 (from python-xlib)
  Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: setuptools, wheel, six, python-xlib, pip
  Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:
      Successfully uninstalled setuptools-28.8.0
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
  Running setup.py install for pip ... done
Successfully installed pip-10.0.0.dev0 python-xlib-0.19 setuptools-36.2.7 six-1.10.0 wheel-0.29.0
> pip freeze --all
pip==10.0.0.dev0
python-xlib==0.19
setuptools==36.2.7
six==1.10.0
wheel==0.29.0
> cat >setup.py <<\EOF
from setuptools import setup
import Xlib
setup(name='project', version='0.1')
EOF
> cat >pyproject.toml <<\EOF
[build-system]
requires = ["python-xlib"]
EOF
> cat >MANIFEST.in <<\EOF
include pyproject.toml
EOF
> python setup.py sdist
running sdist
running egg_info
creating project.egg-info
writing project.egg-info/PKG-INFO
writing dependency_links to project.egg-info/dependency_links.txt
writing top-level names to project.egg-info/top_level.txt
writing manifest file 'project.egg-info/SOURCES.txt'
reading manifest file 'project.egg-info/SOURCES.txt'
writing manifest file 'project.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt

running check
warning: check: missing required meta-data: url

warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied

creating project-0.1
creating project-0.1/project.egg-info
copying files to project-0.1...
copying setup.py -> project-0.1
copying project.egg-info/PKG-INFO -> project-0.1/project.egg-info
copying project.egg-info/SOURCES.txt -> project-0.1/project.egg-info
copying project.egg-info/dependency_links.txt -> project-0.1/project.egg-info
copying project.egg-info/top_level.txt -> project-0.1/project.egg-info
Writing project-0.1/setup.cfg
creating dist
Creating tar archive
removing 'project-0.1' (and everything under it)
> pip uninstall -y python-xlib
Uninstalling python-xlib-0.19:
  Successfully uninstalled python-xlib-0.19
> pip install dist/project-0.1.tar.gz
Processing ./dist/project-0.1.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-c16oxefg/setup.py", line 2, in <module>
        import Xlib
    ModuleNotFoundError: No module named 'Xlib'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-c16oxefg/
@RonnyPfannschmidt
Copy link
Contributor

pep518 is implemented in #4144 which will be released in pip 10, so currently there is no support in a released version

@pradyunsg pradyunsg added the resolution: invalid Invalid issue/PR label Aug 6, 2017
@pradyunsg
Copy link
Member

pradyunsg commented Aug 6, 2017

You seem to be using the latest master. The issue you're facing (PEP 518 build requirements not being satisfied when installing from local sdists) would be fixed once a rebased #4562 merges.

On that note, I'll close this issue. :)

@benoit-pierre
Copy link
Member Author

benoit-pierre commented Aug 6, 2017

Correct me if I am wrong, but #4562 is about extending support to VCS link / directory installs. There's already support for installing from sdists. Case in point:

> tmpdir="$(mktemp -d)"
> trap "cd - && rm -rf $tmpdir" EXIT
> cd "$tmpdir"
> python -m venv venv
> export PATH="$PWD/venv/bin:$PATH"
> pip -q install -U setuptools wheel python-xlib https://github.com/pypa/pip/archive/573a50118148ff4646ae9310730f5bf6c4e8784f.zip
> pip freeze --all
pip==10.0.0.dev0
python-xlib==0.19
setuptools==36.2.7
six==1.10.0
wheel==0.29.0
> >setup.py <<\EOF
from setuptools import setup
from pkg_resources import working_set
print('in setup.py, available distributions:\n' +
      "\n".join(sorted(working_set.by_key.keys())))
setup(name='project', version='0.1')
EOF
> >pyproject.toml <<\EOF
[build-system]
requires = ["python-xlib"]
EOF
> >MANIFEST.in <<\EOF
include pyproject.toml
EOF
> python setup.py -q sdist
in setup.py, available distributions:
pip
python-xlib
setuptools
six
wheel
warning: sdist: standard file not found: should have one of README, README.rst, README.txt

warning: check: missing required meta-data: url

warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied

> pip uninstall -y python-xlib
> pip install -v dist/project-0.1.tar.gz
Created temporary directory: /tmp/pip-install-eorc13vf
Processing ./dist/project-0.1.tar.gz
  Created temporary directory: /tmp/pip-req-build-054cpvg6
  Running command python -c "import setuptools"
  Running setup.py (path:/tmp/pip-req-build-054cpvg6/setup.py) egg_info for package from file:///tmp/tmp.TVomHQ96sm/dist/project-0.1.tar.gz
    Running command python -c "import setuptools"
    Running command python setup.py egg_info
    in setup.py, available distributions:
    pip
    project
    setuptools
    six
    wheel
    running egg_info
    creating pip-egg-info/project.egg-info
    writing pip-egg-info/project.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/project.egg-info/dependency_links.txt
    writing top-level names to pip-egg-info/project.egg-info/top_level.txt
    writing manifest file 'pip-egg-info/project.egg-info/SOURCES.txt'
    reading manifest file 'pip-egg-info/project.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'pip-egg-info/project.egg-info/SOURCES.txt'
  Source in /tmp/pip-req-build-054cpvg6 has version 0.1, which satisfies requirement project==0.1 from file:///tmp/tmp.TVomHQ96sm/dist/project-0.1.tar.gz
Building wheels for collected packages: project
  This version of pip does not implement PEP 516, so it cannot build a wheel without setuptools. You may need to upgrade to a newer version of pip.
  Created temporary directory: /tmp/pip-build-env-07bvfv7t
  1 location(s) to search for versions of python-xlib:
  * https://pypi.python.org/simple/python-xlib/
  Getting page https://pypi.python.org/simple/python-xlib/
  Looking up "https://pypi.python.org/simple/python-xlib/" in the cache
  Current age based on date: 441
  Freshness lifetime from max-age: 600
  Freshness lifetime from request max-age: 600
  The response is "fresh", returning cached response
  600 > 441
  Analyzing links from page https://pypi.python.org/simple/python-xlib/
    Found link https://pypi.python.org/packages/30/0a/ce07aa90f5086a643f70407769989162959147d14c871cba0e3822c8bc7a/python_xlib-0.19-py2.py3-none-any.whl#md5=eca5a5410a7090ab91263d11ac1340dc (from https://pypi.python.org/simple/python-xlib/), version: 0.19
    Found link https://pypi.python.org/packages/48/9a/bcf41728a0a81b3d76e2a873296c2912300c653f9e6453c760a50bd2ef93/python-xlib-0.19.tar.bz2#md5=81f26828fc4ccdcfae8e95de6409d129 (from https://pypi.python.org/simple/python-xlib/), version: 0.19
    Found link https://pypi.python.org/packages/5e/f6/1235775fe2a5cc0d788787b97b5911b208d2e093cc03d15a38175d53788b/python_xlib-0.17-py3-none-any.whl#md5=babd8a1dd7ded2ce08050dd9c50d3cf0 (from https://pypi.python.org/simple/python-xlib/), version: 0.17
    Skipping link https://pypi.python.org/packages/71/ca/0ecaa43b4403b9e56595190c6f2b93aae583fa189bb9acb61b57e75644c6/python_xlib-0.17-py2-none-any.whl#md5=c741c77e760abeb08873b70780fac3e9 (from https://pypi.python.org/simple/python-xlib/); it is not compatible with this Python
    Skipping link https://pypi.python.org/packages/bc/7a/ecdca237146993fbf988e78b0c7408d3b2bd8578496e9982c9cba2603e68/python_xlib-0.16-py2-none-any.whl#md5=9dd3f88d354536025161ccc902070770 (from https://pypi.python.org/simple/python-xlib/); it is not compatible with this Python
    Found link https://pypi.python.org/packages/d4/1b/a1ed5a5b47a6b4a1de029a157b61acf784e2fe8cf165fb6e32c627caa790/python-xlib-0.17.tar.bz2#md5=8b392bac1e634f605aa4dac3128dab95 (from https://pypi.python.org/simple/python-xlib/), version: 0.17
    Found link https://pypi.python.org/packages/d9/b3/189b3c786f41d416fddd90195b31d4b0031468bfd7b2392e5a3a9ccce393/python_xlib-0.18-py2.py3-none-any.whl#md5=d642624c44b0db855b611b1740bb8eb9 (from https://pypi.python.org/simple/python-xlib/), version: 0.18
    Found link https://pypi.python.org/packages/f6/11/a31b6ff171065f4e700914289036a1154dd8bfcb6b6e4c06a56c4052367f/python_xlib-0.16-py3-none-any.whl#md5=379107824aff037322b220fc444cf9ff (from https://pypi.python.org/simple/python-xlib/), version: 0.16
    Found link https://pypi.python.org/packages/fa/13/30ba501a3a92ceb35155798aa08f94cab15ca89a28deebcbbbc950c7c732/python-xlib-0.18.tar.bz2#md5=b54349fd0f3f6b9cbabf3a3303743758 (from https://pypi.python.org/simple/python-xlib/), version: 0.18
    Found link https://pypi.python.org/packages/ff/8e/0b63c8a086ca6bbbf0eafae1aa8e194ee63203c3f655f6e6c4ed24228646/python-xlib-0.16.tar.gz#md5=c60db93fd6d535d24a2bb4cc3e39e9f2 (from https://pypi.python.org/simple/python-xlib/), version: 0.16
  Using version 0.19 (newest of versions: 0.16, 0.17, 0.18, 0.19)
  Installing build dependencies ...   Running command /tmp/tmp.TVomHQ96sm/venv/bin/python -m pip install --ignore-installed --prefix /tmp/pip-build-env-07bvfv7t https://pypi.python.org/packages/30/0a/ce07aa90f5086a643f70407769989162959147d14c871cba0e3822c8bc7a/python_xlib-0.19-py2.py3-none-any.whl#md5=eca5a5410a7090ab91263d11ac1340dc
  Collecting python-xlib==0.19 from https://pypi.python.org/packages/30/0a/ce07aa90f5086a643f70407769989162959147d14c871cba0e3822c8bc7a/python_xlib-0.19-py2.py3-none-any.whl#md5=eca5a5410a7090ab91263d11ac1340dc
    Using cached python_xlib-0.19-py2.py3-none-any.whl
  Collecting six>=1.10.0 (from python-xlib==0.19)
    Using cached six-1.10.0-py2.py3-none-any.whl
  Installing collected packages: six, python-xlib
  Successfully installed python-xlib six
done
  Created temporary directory: /tmp/pip-wheel-us438joy
  Running command python -c "import setuptools"
  Running setup.py bdist_wheel for project ...   Destination directory: /tmp/pip-wheel-us438joy
  Running command /tmp/tmp.TVomHQ96sm/venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-054cpvg6/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-us438joy --python-tag cp36
  in setup.py, available distributions:
  pip
  project
  python-xlib
  setuptools
  six
  wheel
  running bdist_wheel
  running build
  installing to build/bdist.linux-x86_64/wheel
  running install
  running install_egg_info
  running egg_info
  writing project.egg-info/PKG-INFO
  writing dependency_links to project.egg-info/dependency_links.txt
  writing top-level names to project.egg-info/top_level.txt
  reading manifest file 'project.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file 'project.egg-info/SOURCES.txt'
  Copying project.egg-info to build/bdist.linux-x86_64/wheel/project-0.1-py3.6.egg-info
  running install_scripts
  creating build/bdist.linux-x86_64/wheel/project-0.1.dist-info/WHEEL
done
  Stored in directory: /home/bpierre/.cache/pip/wheels/2a/11/de/0c48087b5fbc5db1558414343195e81b473471967bde202c35
  Removing source in /tmp/pip-req-build-054cpvg6
Successfully built project
Installing collected packages: project

Successfully installed project-0.1
Cleaning up...

So build requirements are installed, it's just that setup.py has already been run once before...

I think my issue still stands.

@pradyunsg pradyunsg reopened this Aug 6, 2017
@pradyunsg
Copy link
Member

My bad - I guess I read something incorrectly. 😅

@benoit-pierre
Copy link
Member Author

Thanks for reopening, don't forget to remove the invalid label too. ;)

@benoit-pierre
Copy link
Member Author

Also note that my initial example was flawed, as pyproject.toml was not automatically included by setuptools in the sdist...

@pradyunsg pradyunsg added type: bug A confirmed bug or unintended behavior and removed resolution: invalid Invalid issue/PR labels Aug 6, 2017
@ghost ghost mentioned this issue Sep 3, 2017
@ghost
Copy link

ghost commented Oct 15, 2017

@benoit-pierre There is another subtle issue that I discovered while working on PEP 517 support. The existing "build environment" context manager doesn't load the entry_points from the temporary build environment. Does anyone know how pkg_resources's entry_points work, and more specifically, how to override the search location?

Ref: pypa/setuptools#1174.

@pradyunsg pradyunsg added this to the 10.0 milestone Oct 20, 2017
@pradyunsg pradyunsg added the !release blocker Hold a release until this is resolved label Oct 20, 2017
@pfmoore
Copy link
Member

pfmoore commented Jan 24, 2018

Note that the pyproject.toml in this example is not valid, as it doesn't include setuptools and wheel. Without PEP 517 support, pip can only build wheels using setuptools and wheel, and if you specify your own build requirements in a pyproject.toml, you need to include them alongside any other requirements you add.

I don't believe that this makes any difference to the validity of this issue, though.

@benoit-pierre
Copy link
Member Author

I assumed that in the absence of specifying a build-backend, since setuptools/wheel is assumed to be used, those are automatically added to requires. It would make sense, IMHO. But I guess this could be problematic if a specific version of those is needed.

@pradyunsg
Copy link
Member

pradyunsg commented Jan 25, 2018 via email

@pradyunsg pradyunsg removed !release blocker Hold a release until this is resolved labels Feb 8, 2019
@lock
Copy link

lock bot commented May 29, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot added the auto-locked Outdated issues that have been locked by automation label May 29, 2019
@lock lock bot locked as resolved and limited conversation to collaborators May 29, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
auto-locked Outdated issues that have been locked by automation type: bug A confirmed bug or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants