New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

setup_requires and install_requires don't play nicely together #209

Closed
bb-migration opened this Issue May 23, 2014 · 23 comments

Comments

Projects
None yet
1 participant
@bb-migration

bb-migration commented May 23, 2014

Originally reported by: msabramo (Bitbucket: msabramo, GitHub: msabramo)


What's the latest thinking on how to deal with a package that has setup_requires=['foo'] and install_requires=['foo']?

My observation is that this causes setuptools to download a foo.egg file to the current directory, install_requires then decides it doesn't need to install the package, and then the software craps out when it tries to import it because it can't import from the egg :-(

For example, I work on a project that uses pbr. The setup.py looks like this:

import setuptools

setuptools.setup(
    setup_requires=['pbr'],
    install_requires=['pbr'],
    pbr=True)

Now starting with a clean virtualenv and no .egg files in the current directory, here's what I see (with pip 1.5.5 and setuptools 3.4.4):

$ python setup.py install

Installed /Users/marca/dev/git-repos/jenkins-job-builder/pbr-0.8.0-py2.7.egg
running install
...
Requirement already satisfied (use --upgrade to upgrade): pbr>=0.5.21,<1.0 in ./pbr-0.8.0-py2.7.egg
...

$ ls -ld *.egg
drwxr-xr-x  4 marca  staff  136 May 23 08:56 pbr-0.8.0-py2.7.egg/

OK, fine. pbr wasn't available in the virtualenv so setuptools downloaded a .egg so that it could use it to do setupy things.

Not so fine is that because the .egg file was already present because it was downloaded in the setup_requires phase, the setuptools install_requires logic decided that it didn't have to install pbr in the virtualenv (I guess at this point the .egg file is on the PYTHONPATH so it can be imported?). This is not good because code outside of setuptools doesn't know how to do anything with that .egg:

$ python -c 'import jenkins_jobs.version'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "jenkins_jobs/version.py", line 18, in <module>
    from pbr.version import VersionInfo
ImportError: No module named pbr.version

So this is not good.

Ideas:

  1. Maybe setuptools puts the .egg file on the PYTHONPATH so that it can use it while creating metadata and such. Maybe when it's done, it can remove the .egg from the PYTHONPATH so that the install_requires code doesn't see the package as importable and then it would install the package in the virtualenv? A variation on the idea is to leave the PYTHONPATH but delete the .egg file (or download the egg into a temp directory that gets cleaned up as soon as it's not needed). Drawback here is that we download the package twice -- once for setup_requires and once for install_requires.
  2. Have a special case in the setup_requires logic so that if install_requires of the same dependency is present, instead of downloading the .egg, it simply installs the package in a more "permanent" place on the PYTHONPATH (site-packages)?

@jaraco: What do you think?


@bb-migration

This comment has been minimized.

bb-migration commented May 23, 2014

Original comment by msabramo (Bitbucket: msabramo, GitHub: msabramo):


This seems important to projects that use pbr, so want to add a few folks who I think have an interest in pbr:

@bb-migration

This comment has been minimized.

bb-migration commented May 23, 2014

Original comment by mordred (Bitbucket: mordred, GitHub: mordred):


You'll notice that if you do "pip install ." instead of "python setup.py install" you get the behavior you expect, which is that the depend that setuptools temporarily installs locally for the purposes for fulfilling the setup_requires does not prevent the install_requires from firing properly.

I kinda like 2 - except that it actually makes a behavioral change that could have knockon effects. Currently, other than setup_requires, in a pip installation, everything that gets listed in install_requires actually gets installed by pip and not by setuptools. This is important for situations where easy_install behaves poorly, like proxy-blocked places.

1 preseves the separation, but as you said means a double download. I wonder if easy_install could download the egg to a known location and then pass that info along to install_requires invocations so that install_requires could do the installation of the setup_requires downloaded thing. Then, if we wanted to get really clever, we could have that location be settable, so that in a pip-driven install, pip could request of setuptools that it puts downoaded setup_requires artifacts in $DIR, and then pip could consume them.

@dstufft does that sound extra crazy?

@bb-migration

This comment has been minimized.

bb-migration commented May 23, 2014

Original comment by dstufft (Bitbucket: dstufft, GitHub: dstufft):


Well 2 won't actually work because setup_requires runs before install_requires has been interpreted. That's the entire point of setup_requires. Otherwise pbr could never work because setuptools would have already interpreted the install_requires before it installed and loaded pbr.

Pip can't install from a .egg and setuptools prefers them. Trying to reuse artifacts is only going to work in certain situations and it honestly sounds like more work than it's worth. Especially since in the future I plan on stealing setup_requires from setuptools and using pip to install them into a temporary directory instead of curdir.

That being said not deleting the setup_requires files is an optimization to remove the need to download it every time you run setup.py. Probably it would make the most sense to just have a .setup-cache or something in curdir which setuptools will add to the PYTHONPATH during setup.py execution but which won't show up outside of that. It also would prevent setuptools from dropping little eggs all over the curdir and make it easier to clean them up.

@bb-migration

This comment has been minimized.

bb-migration commented May 25, 2014

Original comment by tsmithe (Bitbucket: tsmithe, GitHub: tsmithe):


I'd just like to record my experiences here, as I reported to distutils-sig: "I've got an extension that I build with distutils. It requires numpy both to build and to run, so I have numpy in both setup_requires and install_requires. Yet setup.py builds numpy twice -- once for the build stage, and then again on installation. This seems inefficient to me -- why not just build it once?"

This seems to be a slightly different problem to the egg issue discussed by Marc, though Donald's proposed caching solution would seem to fix both.

@bb-migration

This comment has been minimized.

bb-migration commented May 26, 2014

Original comment by dhellmann (Bitbucket: dhellmann, GitHub: dhellmann):


I like dstufft's idea of a cache dir, but does that fix the problem that the package needs to be installed on the real path?

@bb-migration

This comment has been minimized.

bb-migration commented May 27, 2014

Original comment by msabramo (Bitbucket: msabramo, GitHub: msabramo):


@dstufft's cache idea sounds nice to me. But it sounds more like a nicety to keep the current directory looking clean (which would be nice!). I was wondering if an even smaller change to fix the problem at hand is to simply make the egg part of the PYTHONPATH only during the setup_requires phase and then remove it from the PYTHONPATH immediately afterwards (maybe using a context manager?) That way the egg doesn't get looked at during install_requires. Make sense?

@bb-migration

This comment has been minimized.

bb-migration commented May 27, 2014

Original comment by mordred (Bitbucket: mordred, GitHub: mordred):


That sounds actually like the 'right' thing todo - since normally that
egg would not be in the path - so the setup_requires path manip seems
like a leak, rather than a feature.

@bb-migration

This comment has been minimized.

bb-migration commented May 27, 2014

Original comment by msabramo (Bitbucket: msabramo, GitHub: msabramo):


This is extremely simplistic and thus most probably wrong when considering a library with the history and complexity of setuptools, but perhaps it will simply move things a long bit by starting with some code to discuss:

--- a/setuptools/dist.py
+++ b/setuptools/dist.py
@@ -3,6 +3,7 @@ __all__ = ['Distribution']
 import re
 import os
 import sys
+import tempfile
 import warnings
 import distutils.log
 import distutils.core
@@ -326,7 +327,7 @@ class Distribution(_Distribution):
                     links = opts['find_links'][1].split() + links
                 opts['find_links'] = ('setup', links)
             cmd = easy_install(
-                dist, args=["x"], install_dir=os.curdir, exclude_scripts=True,
+                dist, args=["x"], install_dir=tempfile.gettempdir(), exclude_scripts=True,
                 always_copy=False, build_directory=None, editable=False,
                 upgrade=False, multi_version=True, no_report=True, user=False
             )

It seems to work for this particular case that I tested:

 marca@marca-mac2.local ⮀ git-repos/jenkins-job-builder ⮀ ⭠ tox_report_coverage s? ⮀ venv ⮀
 ❯ pip uninstall -y pbr; rm -rf pbr-0.8.0-py2.7.egg; python setup.py install
Uninstalling pbr:
  Successfully uninstalled pbr
running install
Requirement already satisfied (use --upgrade to upgrade): PyYAML in ./venv/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): python-jenkins in ./venv/lib/python2.7/site-packages
Downloading/unpacking pbr>=0.5.21,<1.0
  Using download cache from /Users/marca/.pip/download-cache/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fp%2Fpbr%2Fpbr-0.8.2.tar.gz
  Running setup.py (path:/Users/marca/dev/git-repos/jenkins-job-builder/venv/build/pbr/setup.py) egg_info for package pbr
    [pbr] Processing SOURCES.txt
    warning: LocalManifestMaker: standard file '-c' not found

    [pbr] In git context, generating filelist from git
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files found matching '.gitignore'
    warning: no previously-included files found matching '.gitreview'
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
Requirement already satisfied (use --upgrade to upgrade): pip in ./venv/lib/python2.7/site-packages (from pbr>=0.5.21,<1.0)
Installing collected packages: pbr
  Running setup.py install for pbr
    [pbr] Reusing existing SOURCES.txt
Successfully installed pbr
...

 marca@marca-mac2.local ⮀ git-repos/jenkins-job-builder ⮀ ⭠ tox_report_coverage s? ⮀ venv ⮀
 ❯ python -c 'import jenkins_jobs.version; print(jenkins_jobs.version.version_info)'
0.8.0.5
@bb-migration

This comment has been minimized.

bb-migration commented May 30, 2014

Original comment by dpursehouse (Bitbucket: dpursehouse, GitHub: dpursehouse):


I'm experiencing a similar problem; I'm not sure if it's exactly the same one as described in this issue.

I'm installing jenkins-job-builder in a virtualenv, from a requirements.txt file that also specifies pbr:

  pbr>=0.5.21,<1.0
  jenkins-job-builder==0.8.0


The PIP_INDEX_URL environment variable specifies a server on an internal network, and there is no access to the public internet (i.e. can't get to pypi.python.org).

During the installation, pbr is successfully installed in the virtualenv from the internal package server, but then when jjb is downloaded it tries to install pbr again, but from the external index server.

 Downloading/unpacking pbr>=0.5.21,<1.0 (from -r ../MYPROJECT/etc/requirements.txt (line 1))
  http://pypi.MYSITE.net/simple/pbr/ uses an insecure transport scheme (http). Consider using https if pypi.MYSITE.net has it available
  Running setup.py (path:/home/hudsonslave/root/workspace/MYJOB/MYPROJECT/.virtualenv/build/pbr/setup.py) egg_info for package pbr
    [pbr] Processing SOURCES.txt
    warning: LocalManifestMaker: standard file '-c' not found

    [pbr] In git context, generating filelist from git
    warning: no previously-included files found matching '.gitignore'
    warning: no previously-included files found matching '.gitreview'
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files found matching '.gitignore'
    warning: no previously-included files found matching '.gitreview'
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
 Downloading/unpacking jenkins-job-builder==0.8.0 (from -r ../MYPROJECT/etc/requirements.txt (line 2))
  http://pypi.MYSITE.net/simple/jenkins-job-builder/0.8.0 uses an insecure transport scheme (http). Consider using https if pypi.MYSITE.net has it available
  http://pypi.MYSITE.net/simple/jenkins-job-builder/ uses an insecure transport scheme (http). Consider using https if pypi.MYSITE.net has it available
  Running setup.py (path:/home/hudsonslave/root/workspace/MYJOB/MYPROJECT/.virtualenv/build/jenkins-job-builder/setup.py) egg_info for package jenkins-job-builder
    Download error on https://pypi.python.org/simple/pbr/: [Errno 110] Connection timed out -- Some packages may not be found!
    Couldn't find index page for 'pbr' (maybe misspelled?)
    Download error on https://pypi.python.org/simple/: [Errno 110] Connection timed out -- Some packages may not be found!
    No local packages or download links found for pbr
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/home/hudsonslave/root/workspace/MYJOB/MYPROJECT/.virtualenv/build/jenkins-job-builder/setup.py", line 19, in <module>
        pbr=True)
      File "/usr/lib/python2.7/distutils/core.py", line 112, in setup
        _setup_distribution = dist = klass(attrs)
      File "/home/hudsonslave/root/workspace/MYJOB/MYPROJECT/.virtualenv/local/lib/python2.7/site-packages/setuptools/dist.py", line 239, in __init__
        self.fetch_build_eggs(attrs.pop('setup_requires'))
      File "/home/hudsonslave/root/workspace/MYJOB/MYPROJECT/.virtualenv/local/lib/python2.7/site-packages/setuptools/dist.py", line 264, in fetch_build_eggs
        replace_conflicting=True
      File "/home/hudsonslave/root/workspace/MYJOB/MYPROJECT/.virtualenv/local/lib/python2.7/site-packages/pkg_resources.py", line 580, in resolve
        dist = best[req.key] = env.best_match(req, ws, installer)
      File "/home/hudsonslave/root/workspace/MYJOB/MYPROJECT/.virtualenv/local/lib/python2.7/site-packages/pkg_resources.py", line 818, in best_match
        return self.obtain(req, installer) # try and download/install
      File "/home/hudsonslave/root/workspace/MYJOB/MYPROJECT/.virtualenv/local/lib/python2.7/site-packages/pkg_resources.py", line 830, in obtain
        return installer(requirement)
      File "/home/hudsonslave/root/workspace/MYJOB/MYPROJECT/.virtualenv/local/lib/python2.7/site-packages/setuptools/dist.py", line 314, in fetch_build_egg
        return cmd.easy_install(req)
      File "/home/hudsonslave/root/workspace/MYJOB/MYPROJECT/.virtualenv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 587, in easy_install
        raise DistutilsError(msg)
    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pbr')
    Complete output from command python setup.py egg_info:
    Download error on https://pypi.python.org/simple/pbr/: [Errno 110] Connection timed out -- Some packages may not be found!

@bb-migration

This comment has been minimized.

bb-migration commented Jun 3, 2014

Original comment by msabramo (Bitbucket: msabramo, GitHub: msabramo):


Anyone try out my diff?

@bb-migration

This comment has been minimized.

bb-migration commented Jun 3, 2014

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):


Not yet. It's been a busy week, and I'm currently concerned about the failing tests in the tip with the new zipimport. Once that's settled, released, and stable, I'll revisit this issue. Feel free to ping periodically if you don't see any activity.

@bb-migration

This comment has been minimized.

bb-migration commented Sep 10, 2014

Original comment by rbuchholz (Bitbucket: rbuchholz, GitHub: rbuchholz):


@jaraco, ping?

@bb-migration

This comment has been minimized.

bb-migration commented Sep 10, 2014

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):


@msabramo Why does your patch work? I don't understand why installing the package to a temporary directory doesn't trigger the install_requires directive to be satisfied for pbr. Is it because '.' is on the sys.path, and thus setuptools will consider any eggs on sys.path as installed?

@bb-migration

This comment has been minimized.

bb-migration commented Sep 16, 2014

Original comment by msabramo (Bitbucket: msabramo, GitHub: msabramo):


I think it's what you said, but I'm not positive as my brain has swapped this out. :-)

@bb-migration

This comment has been minimized.

bb-migration commented Sep 27, 2014

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):


It seems unwise to me to change setuptools to use a global temp directory as a cache simply to address this issue. One concern is the possibility that an untrusted agent could write to that directory in advance, fulfilling the requirement with untrusted code.

Another option that's been proposed is to have the eggs created in a specially-named directory, such as "./.egg-cache". That also might address the issue, and would avoid the additional security risk (see #80).

There are additional issues too that I would like to see solved, such as the ability for setup_requires requirements to be more strict than system-installed packages (#141).

So the proposed patch doesn't address the issue directly and probably has unwanted consequences. I'd prefer a solution that addresses the issue directly by carefully keeping setup_requires and install_requires packages distinct, but I'd also accept a PR that bypasses the issue by fixing #80.

@bb-migration

This comment has been minimized.

bb-migration commented Sep 27, 2014

Original comment by dstufft (Bitbucket: dstufft, GitHub: dstufft):


I always assumed it would be a per invocation cache TBH created securely. Using a tempdir like that is only unsafe if you repeatedly re-use it.

@bb-migration

This comment has been minimized.

bb-migration commented Sep 27, 2014

Original comment by msabramo (Bitbucket: msabramo, GitHub: msabramo):


See https://bitbucket.org/pypa/setuptools/pull-request/84/put-setup_requires-eggs-in/diff

This implements @jaraco's idea using a specially named directory in the current directory (./.setuptools_egg_cache) instead of a temp directory.

@bb-migration

This comment has been minimized.

bb-migration commented Oct 10, 2014

Original comment by msabramo (Bitbucket: msabramo, GitHub: msabramo):


Bump

@bb-migration

This comment has been minimized.

bb-migration commented Jan 4, 2015

Original comment by syllog1sm (Bitbucket: syllog1sm, GitHub: syllog1sm):


As a workaround, I've created a separate package to install the headers:

https://github.com/syllog1sm/headers_workaround

I noticed that Issue #80, which seems to refer to this as well, is Closed: Resolved, but this still seems to be a problem for me.

I think this problem probably affects most extension modules, since it's very common to require numpy headers.

@bb-migration

This comment has been minimized.

bb-migration commented Jan 13, 2015

Original comment by bignose (Bitbucket: bignose, GitHub: bignose):


This is biting us with the ‘python-daemon’ distribution. As of version 2.0, the distribution needs ‘docutils’ available for the ‘setup.py egg_info’ command.

According to the Setuptools documentation for the setup_requires and install_requires arguments, specifying ‘docutils’ in both setup_requires and install_requires should satisfy this requirement.

That's not what happens though:

#!none

$ pip install python-daemon==2.0.2
Downloading/unpacking python-daemon==2.0.2
  Downloading python-daemon-2.0.2.tar.gz (67kB): 67kB downloaded
  Running setup.py (path:/tmp/pip-build-dBs22x/python-daemon/setup.py) egg_info for package python-daemon
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/tmp/pip-build-dBs22x/python-daemon/setup.py", line 27, in <module>
        import version
      File "version.py", line 51, in <module>
        import docutils.core
    ImportError: No module named docutils.core
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/tmp/pip-build-dBs22x/python-daemon/setup.py", line 27, in <module>

    import version

  File "version.py", line 51, in <module>

    import docutils.core

ImportError: No module named docutils.core

----------------------------------------

@bb-migration

This comment has been minimized.

bb-migration commented Jan 14, 2015

Original comment by bignose (Bitbucket: bignose, GitHub: bignose):


It turns out the ‘python-daemon’ setup failure was due to a circular import. The ‘docutils’ library was needed for ‘egg_info’, but was also imported before ‘setup()’ even had an opportunity to act. No dependency declaration could help there.

So that was not an instance of the behaviour in this bug report. Sorry for the noise!

@bb-migration

This comment has been minimized.

bb-migration commented Feb 27, 2015

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):


I believe this was fixed by Pull Request #97, released as 7.0.

@bb-migration

This comment has been minimized.

bb-migration commented Oct 7, 2015

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):


The core issue here was not fully resolved. We will continue to track the issue in #391.

metatoaster added a commit to flaviusb/kivy that referenced this issue Oct 22, 2018

Refactor the declaration and usage of Cython
- Use setuptools.setup by default if available for import.
- The objective for that complicated import/cython is commented.
- Also have more specific variable names, and have their usage be
  clarified.

- Remove Cython from install_requires as it already is declared under
  setup_requires, and that it conflicts with install_requires due to
  issues in setuptools.

  - pypa/setuptools#209
  - pypa/setuptools#391

- This commit goes back to breaking installation in environments without
  Cython, but should be rectified in the next commit.

flaviusb added a commit to flaviusb/kivy that referenced this issue Oct 22, 2018

Refactor the declaration and usage of Cython
- Use setuptools.setup by default if available for import.
- The objective for that complicated import/cython is commented.
- Also have more specific variable names, and have their usage be
  clarified.

- Remove Cython from install_requires as it already is declared under
  setup_requires, and that it conflicts with install_requires due to
  issues in setuptools.

  - pypa/setuptools#209
  - pypa/setuptools#391

- This commit goes back to breaking installation in environments without
  Cython, but should be rectified in the next commit.

flaviusb added a commit to flaviusb/kivy that referenced this issue Oct 23, 2018

Refactor the declaration and usage of Cython
- Use setuptools.setup by default if available for import.
- The objective for that complicated import/cython is commented.
- Also have more specific variable names, and have their usage be
  clarified.

- Remove Cython from install_requires as it already is declared under
  setup_requires, and that it conflicts with install_requires due to
  issues in setuptools.

  - pypa/setuptools#209
  - pypa/setuptools#391

- This commit goes back to breaking installation in environments without
  Cython, but should be rectified in the next commit.

flaviusb added a commit to flaviusb/kivy that referenced this issue Oct 23, 2018

Refactor the declaration and usage of Cython
- Use setuptools.setup by default if available for import.
- The objective for that complicated import/cython is commented.
- Also have more specific variable names, and have their usage be
  clarified.

- Remove Cython from install_requires as it already is declared under
  setup_requires, and that it conflicts with install_requires due to
  issues in setuptools.

  - pypa/setuptools#209
  - pypa/setuptools#391

- This commit goes back to breaking installation in environments without
  Cython, but should be rectified in the next commit.

flaviusb added a commit to flaviusb/kivy that referenced this issue Oct 23, 2018

Refactor the declaration and usage of Cython
- Use setuptools.setup by default if available for import.
- The objective for that complicated import/cython is commented.
- Also have more specific variable names, and have their usage be
  clarified.

- Remove Cython from install_requires as it already is declared under
  setup_requires, and that it conflicts with install_requires due to
  issues in setuptools.

  - pypa/setuptools#209
  - pypa/setuptools#391

- This commit goes back to breaking installation in environments without
  Cython, but should be rectified in the next commit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment