setuptools included in requirements.txt generation

[ipmb]

Describe your environment

1. MacOS 10.12.6
2. Python 3.6.3
3. pipenv, version 9.0.3

Expected result

The lock file should include all the dependencies of the installed package. Usually this isn't an issue because setuptools will already be installed, but it prevents me from using pip wheel against the generated requirements to create wheels.

Actual result

setuptools is excluded from the lock file and requirements which prevents the generation of wheels using pip wheel.

Steps to replicate

$ pipenv install python-json-logger
...
$ pipenv graph
python-json-logger==0.1.8
  - setuptools [required: Any, installed: 38.5.1]
$ grep setuptools Pipfile.lock || echo "not found"
not found
$ pipenv lock --requirements > requirements.txt
$ $ virtualenv .tmp
...
max:test pete$ ./tmp/bin/pip wheel -r requirements.txt
Collecting python-json-logger==0.1.8 (from -r requirements.txt (line 1))
  File was already downloaded /Users/pete/projects/test/python_json_logger-0.1.8-py2.py3-none-any.whl
Collecting setuptools (from python-json-logger==0.1.8->-r requirements.txt (line 1))
In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    setuptools from https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5 (from python-json-logger==0.1.8->-r requirements.txt (line 1))

[ipmb]

I understand why this is (it's in pip-tools UNSAFE_PACKAGES list).

My use case is compiling a set of wheels from a Pipfile.lock so they can be distributed to production environments. Is there a different way I can do this where I won't run into this issue?

[kennethreitz]

setuptools is a special package that should be provided by the base python installation (much like pip) and will always be excluded.

[uranusjr]

@ipmb What is the intention behind having a setuptools entry in your lock file (or requirements.txt)? You can’t install Setuptools from them anyway because you need to have Setuptools installed in order to install anything (with Pip) in the first place. The Setuptools distribution should be bundled in your Python distribution (you might need to install it separately when using certain package managers, e.g. APT), but in any case, it should not be part of the package requirements. It does not make sense (to me at least).

[ipmb]

It's a third party dependency that is including it. I understand what both you and @kennethreitz are saying and agree, but unfortunately there are lots of packages on PyPI that have setuptools as a requirement. IMO, this should work, but it doesn't (if setuptools is a dependency of one of your project's dependencies):

pipenv lock --requirements > requirements.txt
# ship requirements.txt to another location (a build server for example)
pip install -r requirements.txt

You'll get an error like this:

In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    setuptools from https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5 (from python-json-logger==0.1.8->-r /code/parts/web/src/requirements.txt (line 10))

The workaround I have in place is not so pretty:

pipenv lock --requirements > requirements.txt
# https://github.com/pypa/pipenv/issues/1417
pipenv graph | grep "\- setuptools \[" && echo "setuptools==38.5.1 --hash=sha256:6425484c08e99a98a42209c25c3d325f749230b55284d66192784f941a7e6628 --hash=sha256:7ffe771abfae419fd104f93400b61c935b5af10bfe4dfeec7a1bd495694eea35" >> requirements.txt

[kennethreitz]

ah, so that is a bug.

[kennethreitz]

fixed!

[ipmb]

This doesn't seem to be resolved in pipenv, version 11.0.2. A simple test is:

pipenv install pytest  # pytest depends on setuptools
pipenv lock -r | grep setuptools  # setuptools is not in the output