New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not exclude legitimately installed packages from pip freeze #1610

Closed
piotr-dobrogost opened this Issue Mar 2, 2014 · 5 comments

Comments

Projects
None yet
7 participants
@piotr-dobrogost

piotr-dobrogost commented Mar 2, 2014

Silently excluding legitimately installed packages like setuptools, distribute and pip is not ok. For instance, as requirements file can contain options for pip, someone might want to ensure that the proper version of pip (able to understand these options) would be available. Also, @qwcode gave rationale for not excluding setuptools in issue #1606:

with pip 1.5 able to install from wheel without setuptools, I can see a real case where it's meaningful for setuptools to be listed, i.e. when one of the packages needs setuptools due to an import of pkg_resources, but the install is just using pip (and no setuptools) to install from wheel.

Reasons for having setuptools as a requirement were given in answers to my recent question Is there any sense to put setuptools as a requirement? on distutils-sig mailing list.

There should be an option so that users could decide themselves what packages if any they want to exclude. In any case there should be information about excluded packages presented in the output of every command affected.

By legitimately installed I mean all outside of stdlib as excluding stdlib packages with metadata is another subject – see issue #1570.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Mar 2, 2014

Member

Putting pip inside of a requirements.txt file is almost never what you want. For instance, you stated:

For instance, as requirements file can contain options for pip, someone might want to ensure that the proper version of pip (able to understand these options) would be available.

However the problem is, if you have a requirements file like that, then an incompatible version of pip will bomb out trying to read that requirements.txt file before it even has a chance to upgrade pip. Even if everything goes without a hitch and you successfully upgrade/downgrade pip to a different version via the requirements.txt file it won't actually take any effect until the next invocation of pip (because the current version will have already been imported). If we want to handle the case where you modify the version of pip in a single invocation then we're going to need additional code to handle that.

In the past there was the same sort of problem with setuptools. You couldn't import pip without setuptools installed and you couldn't upgrade setuptools in the same process and have it take effect in the installation of the rest of the packages in that installation run. Now a days this is no longer the case, however I would still argue that it should be excluded by default because rarely does someone actually want to assert a particular version of setuptools (and in fact it can be harmful, especially on platforms like Heroku).

I do not believe we should be putting pip into the output of pip freeze by default, and I feel pretty strongly about that. setuptools/distribute being included by default I still don't think is generally applicable and can be harmful, but I could probably be convinced otherwise. One option is to include setuptools/distribute by default, but have it commented out so that it's obvious that it's there and what version was installed, but that it doesn't affect anything without end user work. Another option is to make the list of what to ignore configurable. If the user hasn't specified any ignores it uses our default list. If they configure any at all they are responsible for configuring them all.

Member

dstufft commented Mar 2, 2014

Putting pip inside of a requirements.txt file is almost never what you want. For instance, you stated:

For instance, as requirements file can contain options for pip, someone might want to ensure that the proper version of pip (able to understand these options) would be available.

However the problem is, if you have a requirements file like that, then an incompatible version of pip will bomb out trying to read that requirements.txt file before it even has a chance to upgrade pip. Even if everything goes without a hitch and you successfully upgrade/downgrade pip to a different version via the requirements.txt file it won't actually take any effect until the next invocation of pip (because the current version will have already been imported). If we want to handle the case where you modify the version of pip in a single invocation then we're going to need additional code to handle that.

In the past there was the same sort of problem with setuptools. You couldn't import pip without setuptools installed and you couldn't upgrade setuptools in the same process and have it take effect in the installation of the rest of the packages in that installation run. Now a days this is no longer the case, however I would still argue that it should be excluded by default because rarely does someone actually want to assert a particular version of setuptools (and in fact it can be harmful, especially on platforms like Heroku).

I do not believe we should be putting pip into the output of pip freeze by default, and I feel pretty strongly about that. setuptools/distribute being included by default I still don't think is generally applicable and can be harmful, but I could probably be convinced otherwise. One option is to include setuptools/distribute by default, but have it commented out so that it's obvious that it's there and what version was installed, but that it doesn't affect anything without end user work. Another option is to make the list of what to ignore configurable. If the user hasn't specified any ignores it uses our default list. If they configure any at all they are responsible for configuring them all.

@pfmoore

This comment has been minimized.

Show comment
Hide comment
@pfmoore

pfmoore Mar 2, 2014

Member

I don't have a strong opinion about setuptools, but I agree with @dstufft that including pip is wrong. And I do not want to see distribute included by default.

People can manually edit requirements files however they like - as far as I can see this is purely about using freeze and not reviewing or amending its output in any way. I think I'd need to see a specific, clear use case where the current behaviour causes an error and where manual editing of the requirements file is not practical, before I'd think this was worth it.

Member

pfmoore commented Mar 2, 2014

I don't have a strong opinion about setuptools, but I agree with @dstufft that including pip is wrong. And I do not want to see distribute included by default.

People can manually edit requirements files however they like - as far as I can see this is purely about using freeze and not reviewing or amending its output in any way. I think I'd need to see a specific, clear use case where the current behaviour causes an error and where manual editing of the requirements file is not practical, before I'd think this was worth it.

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Mar 2, 2014

Contributor

One option is to include setuptools/distribute by default,
but have it commented out so that it's obvious that it's there and what version was installed
Another option is to make the list of what to ignore configurable.
If the user hasn't specified any ignores it uses our default list.

I'd be ok with either or both of these.

Contributor

qwcode commented Mar 2, 2014

One option is to include setuptools/distribute by default,
but have it commented out so that it's obvious that it's there and what version was installed
Another option is to make the list of what to ignore configurable.
If the user hasn't specified any ignores it uses our default list.

I'd be ok with either or both of these.

@Ivoz

This comment has been minimized.

Show comment
Hide comment
@Ivoz

Ivoz Mar 19, 2014

Member

I think part of the issue is that pip and setuptools are pretty much "build environment" requirements, not regular "install" requirements; but requirements.txt' use case was only ever consciously designed for the latter.

If you separate these two concepts, you come up with a modicum solution:

pip install -r build_requirements.txt
pip install -r requirements.txt

Where build_requirements.txt had something like:

setuptools==<version>
pip==<version>
Member

Ivoz commented Mar 19, 2014

I think part of the issue is that pip and setuptools are pretty much "build environment" requirements, not regular "install" requirements; but requirements.txt' use case was only ever consciously designed for the latter.

If you separate these two concepts, you come up with a modicum solution:

pip install -r build_requirements.txt
pip install -r requirements.txt

Where build_requirements.txt had something like:

setuptools==<version>
pip==<version>
@taion

This comment has been minimized.

Show comment
Hide comment
@taion

taion Oct 8, 2015

As a reference point for how other package managers handle this, NPM explicitly splits out:

I think in general something like pip freeze that locks down the full set of dependencies and sub-dependencies is not the right way to handle development dependencies anyway.

taion commented Oct 8, 2015

As a reference point for how other package managers handle this, NPM explicitly splits out:

I think in general something like pip freeze that locks down the full set of dependencies and sub-dependencies is not the right way to handle development dependencies anyway.

xavfernandez added a commit to xavfernandez/pip that referenced this issue Oct 10, 2015

Do not skip anything
in case a skipped package is in the requirement file
Add an option to whitelist some packages - closes pypa#1610

xavfernandez added a commit to xavfernandez/pip that referenced this issue Oct 21, 2015

Do not skip anything
in case a skipped package is in the requirement file
Add an option to whitelist some packages - closes pypa#1610

xavfernandez added a commit to xavfernandez/pip that referenced this issue Oct 22, 2015

Do not skip anything
in case a skipped package is in the requirement file
Add an option to whitelist some packages - closes pypa#1610

xavfernandez added a commit to xavfernandez/pip that referenced this issue Nov 13, 2015

Do not skip anything
in case a skipped package is in the requirement file
Add an option to whitelist some packages - closes pypa#1610

xavfernandez added a commit to xavfernandez/pip that referenced this issue Nov 13, 2015

Do not skip anything
in case a skipped package is in the requirement file
Add an option to whitelist some packages - closes pypa#1610

xavfernandez added a commit to xavfernandez/pip that referenced this issue Nov 29, 2015

Do not skip anything
in case a skipped package is in the requirement file
Add an option to whitelist some packages - closes pypa#1610

xavfernandez added a commit to xavfernandez/pip that referenced this issue Dec 3, 2015

Do not skip anything
in case a skipped package is in the requirement file
Add an option to whitelist some packages - closes pypa#1610

xavfernandez added a commit to xavfernandez/pip that referenced this issue Jan 2, 2016

Do not skip anything
in case a skipped package is in the requirement file
Add an option to whitelist some packages - closes pypa#1610

xavfernandez added a commit to xavfernandez/pip that referenced this issue Jan 4, 2016

Do not skip anything
in case a skipped package is in the requirement file
Add an option to whitelist some packages - closes pypa#1610

xavfernandez added a commit to xavfernandez/pip that referenced this issue Jan 13, 2016

Do not skip anything
in case a skipped package is in the requirement file
Add an option to whitelist some packages - closes pypa#1610

xavfernandez added a commit to xavfernandez/pip that referenced this issue Jan 15, 2016

Do not skip anything
in case a skipped package is in the requirement file
Add an option to whitelist some packages - closes pypa#1610

xavfernandez added a commit to xavfernandez/pip that referenced this issue Jan 19, 2016

Do not skip anything
in case a skipped package is in the requirement file
Add an option to whitelist some packages - closes pypa#1610

xavfernandez added a commit to xavfernandez/pip that referenced this issue Jan 26, 2016

Do not skip anything
in case a skipped package is in the requirement file
Add an option to whitelist some packages - closes pypa#1610

xavfernandez added a commit to xavfernandez/pip that referenced this issue Jan 31, 2016

Do not skip anything
in case a skipped package is in the requirement file
Add an option to whitelist some packages - closes pypa#1610

xavfernandez added a commit to xavfernandez/pip that referenced this issue Feb 4, 2016

Do not skip anything
in case a skipped package is in the requirement file
Add an option to whitelist some packages - closes pypa#1610

xavfernandez added a commit to xavfernandez/pip that referenced this issue Feb 6, 2016

Add --all option to pip freeze
In order to include pip/setuptools/wheel in the freeze output
Closes pypa#1610

xavfernandez added a commit to xavfernandez/pip that referenced this issue Feb 17, 2016

Add --all option to pip freeze
In order to include pip/setuptools/wheel in the freeze output
Closes pypa#1610

@dstufft dstufft closed this in #3458 Mar 3, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment