Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Put runtime dependencies in install_requires (fixes #308) #309

Merged
merged 1 commit into from Jun 6, 2018

Conversation

pdecat
Copy link
Contributor

@pdecat pdecat commented Jun 6, 2018

Fixes #308

Without this change:

# virtualenv test
New python executable in /home/pdecat/workspaces/thirdparty/pylxd/test/bin/python                                                                                                                              
Installing setuptools, pip, wheel...done.                                                                                                                                                                      
# ./test/bin/python setup.py --help

Installed /home/pdecat/workspaces/thirdparty/pylxd/.eggs/requests_unixsocket-0.1.5-py2.7.egg                                                                                                                   
Searching for requests!=2.8.0,>=2.5.2                                                                                                                                                                          
Reading https://pypi.org/simple/requests/                                                                                                                                                                      
Downloading https://files.pythonhosted.org/packages/49/df/50aa1999ab9bde74656c2919d9c0c085fd2b3775fd3eca826012bef76d8c/requests-2.18.4-py2.py3-none-any.whl#sha256=6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07
e895d1f594193a40a38b                                                                                                                                                                                           
Best match: requests 2.18.4                                                                                                                                                                                    
Processing requests-2.18.4-py2.py3-none-any.whl                                                                                                                                                                
Installing requests-2.18.4-py2.py3-none-any.whl to /home/pdecat/workspaces/thirdparty/pylxd/.eggs                                                                                                              
writing requirements to /home/pdecat/workspaces/thirdparty/pylxd/.eggs/requests-2.18.4-py2.7.egg/EGG-INFO/requires.txt                                                                                         
                                                                                                                                                                                                               
Installed /home/pdecat/workspaces/thirdparty/pylxd/.eggs/requests-2.18.4-py2.7.egg                                                                                                                             
Searching for pbr>=1.8                                                                                                                                                                                         
Reading https://pypi.org/simple/pbr/                                                                                                                                                                           
Downloading https://files.pythonhosted.org/packages/b3/5d/c196041ffdf3e34ba206db6d61d1f893a75e1f3435699ade9bd65e089a3d/pbr-4.0.4-py2.py3-none-any.whl#sha256=3747c6f017f2dc099986c325239661948f9f5176f6880d9fde
f164cb664cd665                                                                                                                                                                                                 
Best match: pbr 4.0.4                                                                                                                                                                                          
Processing pbr-4.0.4-py2.py3-none-any.whl                                                                                                                                                                      
Installing pbr-4.0.4-py2.py3-none-any.whl to /home/pdecat/workspaces/thirdparty/pylxd/.eggs                                                                                                                    
                                                                                                                                                                                                               
Installed /home/pdecat/workspaces/thirdparty/pylxd/.eggs/pbr-4.0.4-py2.7.egg                                                                                                                                   
Searching for urllib3>=1.8                                                                                                                                                                                     
Reading https://pypi.org/simple/urllib3/                                                                                                                                                                       
Downloading https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl#sha256=b5725a0bd4ba422ab0e66e89e030c806576753ea3ee0855
4382c14e685d117b5                                                                                                                                                                                              
Best match: urllib3 1.23                                                                                                                                                                                       
Processing urllib3-1.23-py2.py3-none-any.whl                                                                                                                                                                   
Installing urllib3-1.23-py2.py3-none-any.whl to /home/pdecat/workspaces/thirdparty/pylxd/.eggs                                                                                                                 
writing requirements to /home/pdecat/workspaces/thirdparty/pylxd/.eggs/urllib3-1.23-py2.7.egg/EGG-INFO/requires.txt                                                                                            
                                                                                                                                                                                                               
Installed /home/pdecat/workspaces/thirdparty/pylxd/.eggs/urllib3-1.23-py2.7.egg                                                                                                                                
Traceback (most recent call last):                                                                                                                                                                             
  File "setup.py", line 34, in <module>                                                                                                                                                                        
    pbr=True)                                                                                                                                                                                                  
  File "/home/pdecat/workspaces/thirdparty/pylxd/test/local/lib/python2.7/site-packages/setuptools/__init__.py", line 128, in setup
    _install_setup_requires(attrs)
  File "/home/pdecat/workspaces/thirdparty/pylxd/test/local/lib/python2.7/site-packages/setuptools/__init__.py", line 123, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/home/pdecat/workspaces/thirdparty/pylxd/test/local/lib/python2.7/site-packages/setuptools/dist.py", line 514, in fetch_build_eggs
    replace_conflicting=True,
  File "/home/pdecat/workspaces/thirdparty/pylxd/test/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 779, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (urllib3 1.23 (/home/pdecat/workspaces/thirdparty/pylxd/.eggs/urllib3-1.23-py2.7.egg), Requirement.parse('urllib3<1.23,>=1.21.1'), set(['requests']))

and

# ./test/bin/python setup.py egg_info
Traceback (most recent call last):
  File "setup.py", line 34, in <module>
    pbr=True)
  File "/home/pdecat/workspaces/thirdparty/pylxd/test/local/lib/python2.7/site-packages/setuptools/__init__.py", line 128, in setup
    _install_setup_requires(attrs)
  File "/home/pdecat/workspaces/thirdparty/pylxd/test/local/lib/python2.7/site-packages/setuptools/__init__.py", line 123, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/home/pdecat/workspaces/thirdparty/pylxd/test/local/lib/python2.7/site-packages/setuptools/dist.py", line 514, in fetch_build_eggs
    replace_conflicting=True,
  File "/home/pdecat/workspaces/thirdparty/pylxd/test/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 779, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (urllib3 1.23 (/home/pdecat/workspaces/thirdparty/pylxd/.eggs/urllib3-1.23-py2.7.egg), Requirement.parse('urllib3<1.23,>=1.21.1'), set(['requests']))

and

# ./test/bin/pip install .
Processing /home/pdecat/workspaces/thirdparty/pylxd
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-DK9Xjv/setup.py", line 34, in <module>
        pbr=True)
      File "/home/pdecat/workspaces/thirdparty/pylxd/test/local/lib/python2.7/site-packages/setuptools/__init__.py", line 128, in setup
        _install_setup_requires(attrs)
      File "/home/pdecat/workspaces/thirdparty/pylxd/test/local/lib/python2.7/site-packages/setuptools/__init__.py", line 123, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/home/pdecat/workspaces/thirdparty/pylxd/test/local/lib/python2.7/site-packages/setuptools/dist.py", line 514, in fetch_build_eggs
        replace_conflicting=True,
      File "/home/pdecat/workspaces/thirdparty/pylxd/test/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 779, in resolve
        raise VersionConflict(dist, req).with_context(dependent_req)
    pkg_resources.ContextualVersionConflict: (urllib3 1.23 (/tmp/pip-req-build-DK9Xjv/.eggs/urllib3-1.23-py2.7.egg), Requirement.parse('urllib3<1.23,>=1.21.1'), set(['requests']))
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-DK9Xjv/

With this change:

# virtualenv test
New python executable in /home/pdecat/workspaces/thirdparty/pylxd/test/bin/python                                                                                                                              
Installing setuptools, pip, wheel...done.                                                                                                                                                                      

# ./test/bin/python setup.py --help
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'pbr'
  warnings.warn(msg)
Common commands: (see '--help-commands' for more)

  setup.py build      will build the package underneath 'build/'
  setup.py install    will install the package

Global options:
  --verbose (-v)      run verbosely (default)
  --quiet (-q)        run quietly (turns verbosity off)
  --dry-run (-n)      don't actually do anything
  --help (-h)         show detailed help message
  --no-user-cfg       ignore pydistutils.cfg in your home directory
  --command-packages  list of packages that provide distutils commands

Information display options (just display information, ignore any commands)
  --help-commands     list all available commands
  --name              print package name
  --version (-V)      print package version
  --fullname          print <package name>-<version>
  --author            print the author's name
  --author-email      print the author's email address
  --maintainer        print the maintainer's name
  --maintainer-email  print the maintainer's email address
  --contact           print the maintainer's name if known, else the author's
  --contact-email     print the maintainer's email address if known, else the
                      author's
  --url               print the URL for this package
  --license           print the license of the package
  --licence           alias for --license
  --description       print the package description
  --long-description  print the long package description
  --platforms         print the list of platforms
  --classifiers       print the list of classifiers
  --keywords          print the list of keywords
  --provides          print the list of packages/modules provided
  --requires          print the list of packages/modules required
  --obsoletes         print the list of packages/modules made obsolete

usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

and

# ./test/bin/python setup.py egg_info
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'pbr'
  warnings.warn(msg)
running egg_info
writing requirements to pylxd.egg-info/requires.txt
writing pylxd.egg-info/PKG-INFO
writing top-level names to pylxd.egg-info/top_level.txt
writing dependency_links to pylxd.egg-info/dependency_links.txt
reading manifest file 'pylxd.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'AUTHORS'
warning: no files found matching 'ChangeLog'
warning: no previously-included files found matching '.gitignore'
warning: no previously-included files found matching '.gitreview'
warning: no previously-included files found matching 'contrib_testing'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
writing manifest file 'pylxd.egg-info/SOURCES.txt'

and

# ./test/bin/pip install .
Processing /home/pdecat/workspaces/thirdparty/pylxd
Collecting pbr>=1.8 (from pylxd==2.2.7)
  Using cached https://files.pythonhosted.org/packages/b3/5d/c196041ffdf3e34ba206db6d61d1f893a75e1f3435699ade9bd65e089a3d/pbr-4.0.4-py2.py3-none-any.whl
Collecting requests!=2.8.0,>=2.5.2 (from pylxd==2.2.7)
  Using cached https://files.pythonhosted.org/packages/49/df/50aa1999ab9bde74656c2919d9c0c085fd2b3775fd3eca826012bef76d8c/requests-2.18.4-py2.py3-none-any.whl
Collecting requests-unixsocket>=0.1.5 (from pylxd==2.2.7)
Collecting idna<2.7,>=2.5 (from requests!=2.8.0,>=2.5.2->pylxd==2.2.7)
  Using cached https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl
Collecting urllib3<1.23,>=1.21.1 (from requests!=2.8.0,>=2.5.2->pylxd==2.2.7)
  Using cached https://files.pythonhosted.org/packages/63/cb/6965947c13a94236f6d4b8223e21beb4d576dc72e8130bd7880f600839b8/urllib3-1.22-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests!=2.8.0,>=2.5.2->pylxd==2.2.7)
  Using cached https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests!=2.8.0,>=2.5.2->pylxd==2.2.7)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Building wheels for collected packages: pylxd
  Running setup.py bdist_wheel for pylxd ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-IugbUe/wheels/f3/08/e1/528257ce75fd2e041c3432977a12d4b658267ce1ab8c846d38
Successfully built pylxd
Installing collected packages: pbr, idna, urllib3, certifi, chardet, requests, requests-unixsocket, pylxd
Successfully installed certifi-2018.4.16 chardet-3.0.4 idna-2.6 pbr-4.0.4 pylxd-2.2.7 requests-2.18.4 requests-unixsocket-0.1.5 urllib3-1.22

Copy link

@Shr3ps Shr3ps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@pdecat pdecat force-pushed the GH-308_fix_setup_requires branch from 9d9d911 to 0b2e6b3 Compare June 6, 2018 08:25
@codecov-io
Copy link

codecov-io commented Jun 6, 2018

Codecov Report

Merging #309 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #309   +/-   ##
=======================================
  Coverage   96.31%   96.31%           
=======================================
  Files          11       11           
  Lines         814      814           
  Branches      100      100           
=======================================
  Hits          784      784           
  Misses         10       10           
  Partials       20       20

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3c8eeb8...b7d4ad3. Read the comment docs.

Signed-off-by: Patrick Decat <pdecat@gmail.com>
@pdecat pdecat force-pushed the GH-308_fix_setup_requires branch from 0b2e6b3 to b7d4ad3 Compare June 6, 2018 08:26
@pdecat
Copy link
Contributor Author

pdecat commented Jun 6, 2018

I guess the following comment is obsolete:

# git blame setup.py | grep "DO NOT EDIT"
^01aaf04 (Chuck Short   2015-05-08 11:35:42 -0400 16) # THIS FILE IS MANAGED BY THE GLOBAL REQUIREMENTS REPO - DO NOT EDIT

@pdecat
Copy link
Contributor Author

pdecat commented Jun 6, 2018

FWIW, same checks with python3 and venv.

Without this change:

# python3 -m venv .venv
# ./.venv/bin/python3 setup.py --help

Installed /home/pdecat/workspaces/thirdparty/pylxd/.eggs/requests_unixsocket-0.1.5-py3.6.egg
Searching for requests!=2.8.0,>=2.5.2
Reading https://pypi.python.org/simple/requests/
Downloading https://files.pythonhosted.org/packages/49/df/50aa1999ab9bde74656c2919d9c0c085fd2b3775fd3eca826012bef76d8c/requests-2.18.4-py2.py3-none-any.whl#sha256=6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b
Best match: requests 2.18.4
Processing requests-2.18.4-py2.py3-none-any.whl
Installing requests-2.18.4-py2.py3-none-any.whl to /home/pdecat/workspaces/thirdparty/pylxd/.eggs
writing requirements to /home/pdecat/workspaces/thirdparty/pylxd/.eggs/requests-2.18.4-py3.6.egg/EGG-INFO/requires.txt

Installed /home/pdecat/workspaces/thirdparty/pylxd/.eggs/requests-2.18.4-py3.6.egg
Searching for pbr>=1.8
Reading https://pypi.python.org/simple/pbr/
Downloading https://files.pythonhosted.org/packages/b3/5d/c196041ffdf3e34ba206db6d61d1f893a75e1f3435699ade9bd65e089a3d/pbr-4.0.4-py2.py3-none-any.whl#sha256=3747c6f017f2dc099986c325239661948f9f5176f6880d9fdef164cb664cd665
Best match: pbr 4.0.4
Processing pbr-4.0.4-py2.py3-none-any.whl
Installing pbr-4.0.4-py2.py3-none-any.whl to /home/pdecat/workspaces/thirdparty/pylxd/.eggs

Installed /home/pdecat/workspaces/thirdparty/pylxd/.eggs/pbr-4.0.4-py3.6.egg
Searching for urllib3>=1.8
Reading https://pypi.python.org/simple/urllib3/
Downloading https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl#sha256=b5725a0bd4ba422ab0e66e89e030c806576753ea3ee08554382c14e685d117b5
Best match: urllib3 1.23
Processing urllib3-1.23-py2.py3-none-any.whl
Installing urllib3-1.23-py2.py3-none-any.whl to /home/pdecat/workspaces/thirdparty/pylxd/.eggs
writing requirements to /home/pdecat/workspaces/thirdparty/pylxd/.eggs/urllib3-1.23-py3.6.egg/EGG-INFO/requires.txt

Installed /home/pdecat/workspaces/thirdparty/pylxd/.eggs/urllib3-1.23-py3.6.egg
Traceback (most recent call last):
  File "setup.py", line 34, in <module>
    pbr=True)
  File "/home/pdecat/workspaces/thirdparty/pylxd/.venv/lib/python3.6/site-packages/setuptools/__init__.py", line 128, in setup
    _install_setup_requires(attrs)
  File "/home/pdecat/workspaces/thirdparty/pylxd/.venv/lib/python3.6/site-packages/setuptools/__init__.py", line 123, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/home/pdecat/workspaces/thirdparty/pylxd/.venv/lib/python3.6/site-packages/setuptools/dist.py", line 513, in fetch_build_eggs
    replace_conflicting=True,
  File "/home/pdecat/workspaces/thirdparty/pylxd/.venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 783, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (urllib3 1.23 (/home/pdecat/workspaces/thirdparty/pylxd/.eggs/urllib3-1.23-py3.6.egg), Requirement.parse('urllib3<1.23,>=1.21.1'), {'requests'})

With this change:

# python3 -m venv .venv
# ./.venv/bin/python3 setup.py --help
/usr/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'pbr'
  warnings.warn(msg)
Common commands: (see '--help-commands' for more)

  setup.py build      will build the package underneath 'build/'
  setup.py install    will install the package

Global options:
  --verbose (-v)      run verbosely (default)
  --quiet (-q)        run quietly (turns verbosity off)
  --dry-run (-n)      don't actually do anything
  --help (-h)         show detailed help message
  --no-user-cfg       ignore pydistutils.cfg in your home directory
  --command-packages  list of packages that provide distutils commands

Information display options (just display information, ignore any commands)
  --help-commands     list all available commands
  --name              print package name
  --version (-V)      print package version
  --fullname          print <package name>-<version>
  --author            print the author's name
  --author-email      print the author's email address
  --maintainer        print the maintainer's name
  --maintainer-email  print the maintainer's email address
  --contact           print the maintainer's name if known, else the author's
  --contact-email     print the maintainer's email address if known, else the
                      author's
  --url               print the URL for this package
  --license           print the license of the package
  --licence           alias for --license
  --description       print the package description
  --long-description  print the long package description
  --platforms         print the list of platforms
  --classifiers       print the list of classifiers
  --keywords          print the list of keywords
  --provides          print the list of packages/modules provided
  --requires          print the list of packages/modules required
  --obsoletes         print the list of packages/modules made obsolete

usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

@ajkavanagh
Copy link
Contributor

Thanks very much for debugging this and providing a fix.

@ajkavanagh ajkavanagh merged commit fd4d999 into canonical:master Jun 6, 2018
Copy link

@blackboxsw blackboxsw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hrm doesn't this changeset just mean that you've pushed the ContextualversionConflict problem from setup to install time? From the virtualenv test run, it looks like the only reason we missed hitting this same problem was because of the inability to install pbr in the virtualenv per the warning

/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'pbr'
warnings.warn(msg)

If we had been able to install, wouldn't that have replicated the same error because pbr would have pulled in latest urllib3 @ 1.23?

@pdecat
Copy link
Contributor Author

pdecat commented Jun 7, 2018

At install time, pip will properly resolve runtime dependencies.

@pdecat
Copy link
Contributor Author

pdecat commented Jun 7, 2018

On second thought, perhaps the pbr dependency should have stayed in setup_requires.

@pdecat
Copy link
Contributor Author

pdecat commented Jun 8, 2018

Just tested that, it removes the pbr warning and still fix the issue.

I'm opening a second PR.

pdecat added a commit to pdecat/pylxd that referenced this pull request Jun 8, 2018
@ajkavanagh
Copy link
Contributor

@blackboxsw and @pdecat thanks for pursuing this to the relentless end. 👍

ajkavanagh pushed a commit that referenced this pull request Jun 8, 2018
Signed-off-by: Patrick Decat <pdecat@gmail.com>
@pdecat pdecat deleted the GH-308_fix_setup_requires branch June 8, 2018 13:54
lasizoillo pushed a commit to lasizoillo/pylxd that referenced this pull request Jul 13, 2018
lasizoillo pushed a commit to lasizoillo/pylxd that referenced this pull request Jul 13, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

setup.py should not put runtime dependencies in setup_requires as it breaks pip dependency management
5 participants