Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip freeze does case sensitive comparsion #5716

Closed
abs51295 opened this issue Aug 19, 2018 · 15 comments · Fixed by #6857
Closed

pip freeze does case sensitive comparsion #5716

abs51295 opened this issue Aug 19, 2018 · 15 comments · Fixed by #6857

Comments

@abs51295
Copy link
Contributor

@abs51295 abs51295 commented Aug 19, 2018

Environment

  • pip version: 18.0
  • Python version: 3.6
  • OS: Mac OS X

Description

Running pip freeze -l -r requirements.txt does a case sensitive comparison with dependencies mentioned in requirements.txt.

Expected behavior
Running pip freeze -l -r requirements.txt should do case insensitive comparison with dependencies mentioned in requirements.txt or at least allow that via a command line option.

How to Reproduce

  1. Create a virtualenv.
  2. Create a requirements.txt file.
  3. Run pip install -r requirements.txt.
  4. Run pip freeze -l -r requirements.txt.

Output

#
# This file is autogenerated by pip-compile
# To update, run:
#    pip-compile --output-file requirements.txt requirements.in
asn1crypto==0.24.0
attrs==18.1.0
certifi==2018.4.16
cffi==1.11.5
chardet==3.0.4
click==6.7
colorama==0.3.9
coverage==4.5.1
cryptography==2.2.2
flake8-polyfill==1.0.2
flake8==3.5.0
Requirement file [requirements.txt] contains flask-cors==3.0.4, but that package is not installed
Requirement file [requirements.txt] contains flask==1.0.2, but that package is not installed
gevent==1.3.1
greenlet==0.4.13
gunicorn==19.8.1
idna==2.6
itsdangerous==0.24
Requirement file [requirements.txt] contains jinja2==2.10, but that package is not installed
mando==0.6.4
Requirement file [requirements.txt] contains markupsafe==1.0, but that package is not installed
mccabe==0.6.1
more-itertools==4.1.0
pluggy==0.6.0
psycopg2==2.7.4
py==1.5.3
pycodestyle==2.3.1
pycparser==2.18
pycrypto==2.6.1
pyflakes==1.6.0
Requirement file [requirements.txt] contains pyjwt==1.6.3, but that package is not installed
pytest-cov==2.5.1
pytest-mock==1.10.0
pytest==3.5.1
radon==2.2.0
requests-futures==0.9.7
requests==2.18.4
semantic-version==2.6.0
six==1.11.0
Requirement file [requirements.txt] contains sqlalchemy==1.2.7, but that package is not installed
urllib3==1.22
Requirement file [requirements.txt] contains werkzeug==0.14.1, but that package is not installed
## The following requirements were added by pip freeze:
Flask==1.0.2
Flask-Cors==3.0.4
Jinja2==2.10
MarkupSafe==1.0
PyJWT==1.6.3
SQLAlchemy==1.2.7
Werkzeug==0.14.1
@cjerdonek cjerdonek added the C: freeze label Aug 19, 2018
@abs51295

This comment has been minimized.

Copy link
Contributor Author

@abs51295 abs51295 commented Aug 19, 2018

@cjerdonek I will like to work on this issue. Let me know, when we have classified this as a bug or a feature request.

@cjerdonek cjerdonek added the triage label Aug 19, 2018
@pradyunsg pradyunsg added type: bug and removed triage labels Aug 20, 2018
@pradyunsg

This comment has been minimized.

Copy link
Member

@pradyunsg pradyunsg commented Aug 20, 2018

Hey @abs51295!

I'd consider this a bug since this is essentially a case where pip is not normalizing the given package names correctly.

@abs51295

This comment has been minimized.

Copy link
Contributor Author

@abs51295 abs51295 commented Aug 20, 2018

Thanks @pradyunsg I will send a fix soon! 😄

@abs51295

This comment has been minimized.

Copy link
Contributor Author

@abs51295 abs51295 commented Aug 21, 2018

@pradyunsg how should the fix look like? Should we change the dependency names to lower case explicitly?

@pradyunsg

This comment has been minimized.

Copy link
Member

@pradyunsg pradyunsg commented Aug 22, 2018

There's a normalize_name or a canonical_name (or similarly named) function in pip's codebase, which should be used to normalize names.

(sorry I missed you on IRC - I slept off with the client open)

@abs51295

This comment has been minimized.

Copy link
Contributor Author

@abs51295 abs51295 commented Aug 25, 2018

Thanks @pradyunsg will look into this.

( I don't see you on #pypa-dev anymore on IRC)

@abs51295

This comment has been minimized.

Copy link
Contributor Author

@abs51295 abs51295 commented Aug 25, 2018

@pradyunsg

This comment has been minimized.

Copy link
Member

@pradyunsg pradyunsg commented Aug 27, 2018

Hey @abs51295! I was away from a PC for most of the time over the weekend.

I've not looked carefully but you'd want to start looking here: https://github.com/pypa/pip/blob/b13cee6388adea5ddaf1ddb982beeec37d07ffda/src/pip/_internal/commands/freeze.py

Basically, whenever pip's getting information from the user/distlib/pkg_resources, you'd want to normalize the name using pip._vendor.packaging.utils.canonicalize_name.

I found this: /src/pip/_vendor/packaging/requirements.py@master#L86-L89.

That's a vendored library (packaging) and while I do think that change needs discussion, the change in pip doesn't and they're fairly separate.

@abs51295

This comment has been minimized.

Copy link
Contributor Author

@abs51295 abs51295 commented Sep 4, 2018

Basically, whenever pip's getting information from the user/distlib/pkg_resources, you'd want to normalize the name using pip._vendor.packaging.utils.canonicalize_name

@pradyunsg I did that, but why I was referring to the https://github.com/pypa/pip/blob/master/src/pip/_vendor/packaging/requirements.py#L86-L89, is because of the following:

  1. Canonicalizing the name when pip's getting the information, which can be achieved by canonicalizing the name here, solves only half the problem.

  2. The other half of the problem lies when the requirement line is parsed from requirements.txt file and compared with existing installations as obtained in step 1.

  3. During the above comparsion, if we don't canonicalize the name obtained from the requirements.txt file, then we will again run into the same problem of case sensitive comparison. To mitigate that, we need to canonicalize the name of this requirement which leads to this and hence the comment.

@abs51295

This comment has been minimized.

Copy link
Contributor Author

@abs51295 abs51295 commented Sep 12, 2018

@pradyunsg any updates on this?

@pradyunsg

This comment has been minimized.

Copy link
Member

@pradyunsg pradyunsg commented Sep 14, 2018

Sorry, I haven't taken a deeper look yet. Real life has kept me more than busy lately.

I'll try to take a look today or this weekend. Feel free to ping me again in that time.

@abs51295

This comment has been minimized.

Copy link
Contributor Author

@abs51295 abs51295 commented Sep 14, 2018

Thanks. Will wait for your reply 🙂

@abs51295

This comment has been minimized.

Copy link
Contributor Author

@abs51295 abs51295 commented Oct 4, 2018

@pradyunsg any updates?

@chetabahana

This comment has been minimized.

Copy link

@chetabahana chetabahana commented May 6, 2019

any updates? hope it badly to regenerate requirements.txt
all the warned packages were actually been added by pip already

(virtual-env)$ pip --version
pip 19.1 from /env/lib/python3.7/site-packages/pip (python 3.7)

(virtual-env)$ pip install -r requirements.txt
... done
Successfully installed all

(virtual-env)$ pip install django-storages[google]
... done
Successfully installed 
cachetools-3.1.0 google-api-core-1.9.0 google-auth-1.6.3 google-cloud-core-0.29.1 
google-cloud-storage-1.15.0 google-resumable-media-0.3.2 googleapis-common-protos-1.5.9 
protobuf-3.7.1 pyasn1-0.4.5 pyasn1-modules-0.2.5 rsa-4.0

(virtual-env)$ pip freeze -r requirements.txt
-i https://pypi.python.org/simple
amqp==2.4.2
aniso8601==3.0.2
autopep8==1.4.4
WARNING: Requirement file [requirements.txt] contains babel==2.6.0, but package 'babel' is not installed
billiard==3.6.0.0
bleach==2.1.4
boto3==1.9.138
botocore==1.12.138
braintree==3.49.0
cairocffi==1.0.2
WARNING: Requirement file [requirements.txt] contains cairosvg==2.3.1, but package 'cairosvg' is not installed
celery==4.3.0
certifi==2019.3.9
cffi==1.12.3
chardet==3.0.4
cssselect2==0.2.1
defusedxml==0.6.0
dj-database-url==0.5.0
dj-email-url==0.2.0
django-appconf==1.0.3
django-babel==0.6.2
django-bootstrap4==0.0.8
django-cache-url==3.0.0
django-celery-results==1.0.4
django-countries==5.3.3
django-debug-toolbar-request-history==0.0.10
django-debug-toolbar==1.11
django-elasticsearch-dsl==0.5.1
django-filter==2.1.0
django-graphql-jwt==0.2.1
django-impersonate==1.4
django-js-asset==1.2.2
django-measurement==3.0.0
django-mptt==0.10.0
django-phonenumber-field==2.3.1
django-prices-openexchangerates==1.0.1
django-prices-vatlayer==1.0.2
django-prices==1.0.4
django-recaptcha==1.5.0
django-redis==4.10.0
django-render-block==0.5
django-silk==2.0.0
django-storages==1.7.1
django-templated-email==2.3.0
django-versatileimagefield==1.10
django-webpack-loader==0.6.0
WARNING: Requirement file [requirements.txt] contains django==2.2, but package 'django' is not installed
docutils==0.14
elasticsearch-dsl==6.0.1
elasticsearch==6.3.1
WARNING: Requirement file [requirements.txt] contains faker==1.0.5, but package 'faker' is not installed
freezegun==0.3.11
google-i18n-address==2.3.4
google-measurement-protocol==1.0.0
gprof2dot==2016.10.13
graphene-django-optimizer==0.4.0
graphene-django==2.2.0
graphene==2.1.3
graphql-core==2.1
graphql-relay==0.4.5
gunicorn==19.9.0
html5lib==1.0.1
idna==2.8
ipaddress==1.0.22
WARNING: Requirement file [requirements.txt] contains jinja2==2.10.1, but package 'jinja2' is not installed
jmespath==0.9.4
jsonfield==2.0.2
kombu==4.5.0
WARNING: Requirement file [requirements.txt] contains markdown==3.1, but package 'markdown' is not installed
WARNING: Requirement file [requirements.txt] contains markupsafe==1.1.1, but package 'markupsafe' is not installed
maxminddb-geolite2==2018.703
maxminddb==1.4.1
measurement==2.0.1
mpmath==1.1.0
oauthlib==3.0.1
phonenumberslite==8.10.10
WARNING: Requirement file [requirements.txt] contains pillow==5.4.1, but package 'pillow' is not installed
prices==1.0.0
promise==2.2.1
psycopg2-binary==2.8.2
purl==1.5
pycodestyle==2.5.0
pycparser==2.19
WARNING: Requirement file [requirements.txt] contains pygments==2.3.1, but package 'pygments' is not installed
WARNING: Requirement file [requirements.txt] contains pyjwt==1.7.1, but package 'pyjwt' is not installed
WARNING: Requirement file [requirements.txt] contains pyphen==0.9.5, but package 'pyphen' is not installed
python-dateutil==2.8.0
python3-openid==3.1.0
pytz==2019.1
raven==6.9.0
razorpay==1.2.0
redis==3.2.1
requests-oauthlib==1.2.0
requests==2.21.0
WARNING: Requirement file [requirements.txt] contains rx==1.6.1, but package 'rx' is not installed
s3transfer==0.2.0
singledispatch==3.4.0.3
six==1.12.0
social-auth-app-django==3.1.0
social-auth-core==3.1.0
sqlparse==0.3.0
stripe==2.27.0
sympy==1.4
text-unidecode==1.2
tinycss2==1.0.2
typing==3.6.6
urllib3==1.24.2
WARNING: Requirement file [requirements.txt] contains uwsgi==2.0.18 ; platform_system != 'Windows', but package 'uwsgi' is not installed
vine==1.3.0
WARNING: Requirement file [requirements.txt] contains weasyprint==47, but package 'weasyprint' is not installed
webencodings==0.5.1

The following requirements were added by pip freeze:

astroid==2.2.5
Babel==2.6.0
cachetools==3.1.0
CairoSVG==2.3.1
configparser==3.7.4
Django==2.2
django-fsm==2.6.0
Faker==1.0.5
google-api-core==1.9.0
google-auth==1.6.3
google-cloud-core==0.29.1
google-cloud-storage==1.15.0
google-resumable-media==0.3.2
googleapis-common-protos==1.5.9
isort==4.3.17
Jinja2==2.10.1
lazy-object-proxy==1.3.1
Markdown==3.1
MarkupSafe==1.1.1
mccabe==0.6.1
Pillow==5.4.1
protobuf==3.7.1
pyasn1==0.4.5
pyasn1-modules==0.2.5
pyflakes==2.1.1
Pygments==2.3.1
PyJWT==1.7.1
pylint==2.3.1
Pyphen==0.9.5
rsa==4.0
Rx==1.6.1
typed-ast==1.3.5
uWSGI==2.0.18
WeasyPrint==47
wrapt==1.11.1
@abs51295

This comment has been minimized.

Copy link
Contributor Author

@abs51295 abs51295 commented Aug 11, 2019

@pradyunsg @cjerdonek I have created #6857 in respect to this. Please have a look. Thanks.

abs51295 added a commit to abs51295/pip that referenced this issue Aug 11, 2019
abs51295 added a commit to abs51295/pip that referenced this issue Aug 31, 2019
abs51295 added a commit to abs51295/pip that referenced this issue Aug 31, 2019
abs51295 added a commit to abs51295/pip that referenced this issue Sep 2, 2019
@lock lock bot added the S: auto-locked label Nov 17, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Nov 17, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

4 participants
You can’t perform that action at this time.