pip freeze/list should filter out 'argparse' for python > 2.6 #1570

Closed
qwcode opened this Issue Feb 15, 2014 · 24 comments

Comments

Projects
None yet
9 participants
Contributor

qwcode commented Feb 15, 2014

No description provided.

@qwcode qwcode added bug labels Feb 15, 2014

Why should any package be filtered out?

Contributor

qwcode commented Feb 15, 2014

if it's in the stdlib. argparse is in the stdlib for python >2.6

Member

Ivoz commented Feb 16, 2014

Would the same argument apply to futures?

Contributor

qwcode commented Feb 16, 2014

It applies to anything in the stdlib, that has packaging metadata that pkg_resources detects, and therefore shows up in pip list/freeze.

@dstufft dstufft added this to the 1.5.3 milestone Feb 16, 2014

Do you mean some stdlib packages have packaging metadata and some do not? If so what are criteria for including them in some and not in the others? I have to admit I've never thought about metadata in the context of stdlib packages...

Contributor

qwcode commented Feb 16, 2014

I honestly don't know the thinking on why wsgiref and argparse have egg-info files in the stdlib.
it might be a policy when 3rd party projects (from pypi) move into the stdlib, so you don't install over it.

we can probably come up with something more general than simply adding these to a filter one by one.

Contributor

qwcode commented Feb 16, 2014

btw, filtering out wsgiref is only in the develop branch, so since this is for 1.5.X, need to handle that one too

Contributor

msabramo commented Feb 17, 2014

I guess we could try to do something smart, but a really dumb and robust way to solve the problem is to give users a knob to control it. E.g.: what if in ~/.pip/pip.conf I could specify something like:

ignore-packages =
    wsgiref
    argparse

That also allows people to opt out of it - it seems some folks don't want it.

Contributor

msabramo commented Feb 17, 2014

Although I can argue against my idea above:

  1. One can do that with grep.
  2. Something like argparse would be an external package in some virtualenvs and
    stdlib in others so dumb filtering isn't very useful.

So maybe smart detection of stdlib is better after all.

Contributor

msabramo commented Feb 17, 2014

Maybe could do some heuristic based on directory path - e.g.: site-packages means external; py?.? means stdlib. There is probably some packaging that breaks this (doesn't Ubuntu use /usr/lib/pyshared ?) but perhaps these could be captured also and/or there could be options to control it.

Contributor

qwcode commented Feb 18, 2014

for >= py27, I think it's whether the dist.location == sysconfig.get_paths()['stdlib']
but we have to deal with py26 as well.
thinking I might just hard code the 2 known cases, but at least factor it out better into the backwardcompat package.

Owner

dstufft commented Feb 18, 2014

Hardcoding the two known cases, especially for 1.5.3 seems most reasonable to me.

merwok commented Feb 18, 2014

Some history for the curious:

tl;dr: Daniel’s idea sounds good, but probably needs a refinement for argparse (see comment on PR).

wsgiref existed on PyPI before entering the stdlib with the same importable name. It was developed by PJE, who added the egg-info file so that projects already depending on wsgiref in their setup.py files would not have to make this dependency conditional on the Python version, nor see the module installed in site-packages in addition to the stdlib.

argparse.egg-info is different: it’s not in the upstream repo, so it looks like each OS/distro chooses to add it or not, probably for the same reason than wsgiref. I see the file in my Debian testing system, but not in http://patch-tracker.debian.org/package/python2.7/2.7.6-5

What about python.egg-info? I think the existence of this file was an unexpected side effect of using distutils to build CPython’s extension modules. http://bugs.python.org/issue10645 gives more info.

Owner

pfmoore commented Feb 18, 2014

Oddly enough, of the versions of Python I have installed (Windows, 2.6, 2.7, 3.2, 3.3 and 3.4) only 3.2 has any egg-info files, and that only wsgiref. I don't recall ever having seen an argparse egg-info.

But there's no harm in filtering out stuff that's known to be in the stdlib - worst that can happen is we filter it but it wasn't there anyway.

Contributor

qwcode commented Feb 18, 2014

ubuntu py2.7 has argparse.egg-info

@merwok
Thanks for providing context. What a mess…

@dstufft dstufft modified the milestones: 1.6, 1.5.3 Feb 20, 2014

Contributor

qwcode commented Oct 14, 2014

closing. merged in #1606

@qwcode qwcode closed this Oct 14, 2014

wjwwood commented Jan 22, 2015

This is a really troublesome development for our dependency tool. In my opinion it is logically problematic to be able to do pip install argparse, have it succeed, and then not be able to see that it is installed with pip freeze. I would say that if you cannot see argparse in pip freeze with Python 2.7, then you shouldn't be able to install it either.

The current situation for us (we're using pip version 6.0.6) is that our tool checks to see if argparse is installed, sees that it isn't, and then installs it. This is not really a problem once, but now every time you run the command to install dependencies it will try to install argparse and more troubling it will always report that not all dependencies are satisfied.

We can add an exception for argparse, I guess, but from the perspective of someone else using pip in an automated fashion, this is really inconsistent behavior.

The other idea, which I realize is more complicated, is that pip will omit argparse from the list when provided by Python's stdlib unless the user explicitly installs it. After which point it would be listed in pip freeze.

One last thing I'll suggest is less attractive to me, but maybe a compromise. If pip gave a way to get a list of explicitly ignored items for pip freeze, e.g. something like pip freeze-ignored, then our tool could combine the output from pip freeze and the list of ignored items to do our installed/not-installed filtering.

Honestly though, I think that if you can successfully install something (anything) with pip install it should then end up in pip freeze in some form. Anything else is really strange to me.

@wjwwood wjwwood referenced this issue in ros-infrastructure/rosdep Jan 22, 2015

Closed

rosdep cannot detect installation of [argparse] #368

Contributor

msabramo commented Jan 22, 2015

Can your tool use pip list instead of pip freeze? I believe the former doesn't do filtering, but can't check easily cuz I'm on a phone.

An alternative is to check for argparse by trying to import it -- then it wouldn't try to install it if it's in the stdlib. Unless you want it to install so that you're guaranteed to using the same on all versions.

I do tend to agree though. Ideally, it would not display argparse in Python >= 2.7 if you didn't pip install it but it would show up if you did pip install it.

wjwwood commented Jan 22, 2015

I believe it does filter, based on my tests and the title of the issue:

william@dosa /tmp/rosdep_test_ws
% sudo -H pip install -U argparse
Password:
Requirement already up-to-date: argparse in /Library/Python/2.7/site-packages

william@dosa /tmp/rosdep_test_ws
% sudo -H pip list | grep argparse

wjwwood commented Jan 22, 2015

An alternative is to check for argparse by trying to import it -- then it wouldn't try to install it if it's in the stdlib. Unless you want it to install so that you're guaranteed to using the same on all versions.

I think we might do this in the short term, but that makes the assumption that our tool (which only happens to be written in Python) is running in the same Python environment it is installing for. We could, for example, run the tool from a Python3.x virtualenv, but have it use a pip from the System's Python2.7.

The other problem with this is that we need to do it for every item you guys decide to add to the list in the future. That's why I suggested exposing the exclusion list through another command.

Contributor

xavfernandez commented Jan 23, 2015

You can also use the returncode of pip show to check if the package is installed:

(test) pip show argparse -q
(test) echo $?
0
(test) pip show argparse2 -q
(test) echo $?
1

wjwwood commented Jan 23, 2015

@xavfernandez Thanks that does work for me.

However, that seems even more inconsistent to me, that argparse doesn't show up in the output of just list but it does give a return code 0 when passed as an argument. In fact based on the discussion in this thread I think this behavior would be considered a bug.

wjwwood commented Jan 23, 2015

Sorry, list and show are different, for some reason I parsed that as pip list argparse -q.

Is it the intention to leave this behavior for pip show argparse? If so I can just use that as a solution for our tool.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment