Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add 'pip cache' command #6391

Merged
merged 40 commits into from Apr 13, 2020
Merged

Add 'pip cache' command #6391

merged 40 commits into from Apr 13, 2020

Conversation

duckinator
Copy link
Contributor

@duckinator duckinator commented Apr 8, 2019

This PR adds pip cache command, as per #4685.

TODO:

  • Fix tests
  • General refactoring/cleanup
  • Documentation
  • Add news file fragment
  • Add test for pip cache list with an empty cache directory.
  • Add test for pip cache list that verifies the number of items in the cache is reported correctly.
  • Add test for pip cache purge which verifies that it (if called correctly) deletes everything in the wheel cache.
  • Add test for pip cache purge which verifies that it raises an error + doesn't delete anything if given too many arguments.
  • Convert repeated section to a fixture? See Add 'pip cache' command #6391 (comment).
  • Open issue about adding an option to control sorting of pip cache list (mainly, allow sorting by size).

(Thanks to @BrownTruck @xavfernandez and @tdsmith for their attempts, in #3734 and #3968 respectively, which I took a lot of inspiration/ideas from.)

@duckinator

This comment has been minimized.

@duckinator

This comment has been minimized.

@duckinator

This comment has been minimized.

@duckinator

This comment has been minimized.

@pradyunsg
Copy link
Member

pradyunsg commented Apr 9, 2019

Thanks for this PR @duckinator! :D

I think script.pip(...) is creating a subprocess, and modifies the environment when doing so.

Yep. I can confirm this.


@BrownTruck

That PR was by @xavfernandez that @BrownTruck (a bot) rebased. No worries on getting this detail wrong tho. :P

@pradyunsg pradyunsg added C: cache type: enhancement labels Apr 22, 2019
@duckinator

This comment has been minimized.

@pradyunsg
Copy link
Member

pradyunsg commented Apr 23, 2019

Hey @duckinator!

This is still on my radar; I just haven't had much time outside of RL to get to answering your question.

@duckinator
Copy link
Contributor Author

duckinator commented May 10, 2019

Rebased, but not made any changes.

@xavfernandez
Copy link
Member

xavfernandez commented May 11, 2019

Tests are using https://github.com/pypa/pip/blob/master/tests/conftest.py#L98-L166 which should help you find the correct cache directory.

@chrahunt
Copy link
Member

chrahunt commented Jul 13, 2019

The basic issue appears to be that changes to APPDATA and LOCALAPPDATA in the environment of the current process do not impact the returned value of ctypes.windll.shell32.SHGetFolderPathW. Only for sub-processes.

One option could be to take the current value of os.environ['LOCALAPPDATA'] and append \pip\Cache to it, duplicating the logic from appdirs.user_cache_dir.

Another option could be to get the cache dir from a sub-process itself, like script.run('python', '-c', 'from pip._internal.locations import USER_CACHE_DIR; print(USER_CACHE_DIR). This seems like more of a hack, but does avoid the duplication.

In either case, it may be good to make it a property of PipTestEnvironment so it's accessible from the required tests or any others that might need the cache directory.

@BrownTruck
Copy link
Contributor

BrownTruck commented Jul 28, 2019

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

@BrownTruck BrownTruck added the needs rebase or merge label Jul 28, 2019
@duckinator
Copy link
Contributor Author

duckinator commented Aug 14, 2019

I'll try to get around to rebasing this and updating it taking into account @chrahunt's comment soon. Sorry for the delay!

@pypa-bot pypa-bot removed the needs rebase or merge label Sep 25, 2019
@duckinator duckinator force-pushed the pip-cache branch 2 times, most recently from 3eecaef to bea48bc Compare Sep 25, 2019
@duckinator

This comment has been minimized.

@duckinator
Copy link
Contributor Author

duckinator commented Sep 25, 2019

I opened issue #7082 about accessing environment variables that are modified in subprocesses.

@duckinator
Copy link
Contributor Author

duckinator commented Sep 25, 2019

Okay, looks like I'm not going to be able to add a fix for the Windows tests tonight, but I rebased my changes, opened #7082, and have a rough idea of how I want to approach it.

src/pip/_internal/commands/cache.py Outdated Show resolved Hide resolved
src/pip/_internal/commands/cache.py Outdated Show resolved Hide resolved
duckinator and others added 16 commits Apr 1, 2020
Co-Authored-By: Pradyun Gedam <pradyunsg@gmail.com>
It bothered me that _whether or not a function had a certain argument_
dictated the contents of a directory. Pytest fixtures are inherently
kinda magical, but that was a bit much for me.
@duckinator
Copy link
Contributor Author

duckinator commented Apr 1, 2020

Rebased and resolved the merge conflict. /cc @pradyunsg

@sbidoul
Copy link
Member

sbidoul commented Apr 1, 2020

@pradyunsg it would be great to have this one 20.1 too.

@pradyunsg pradyunsg added this to the 20.1 milestone Apr 1, 2020
@pradyunsg
Copy link
Member

pradyunsg commented Apr 13, 2020

We're going to have a beta period for pip 20.1, and we can collect reports about how this works during the beta. Merging since I think it's very reasonable to have users try out this fairly new and big user facing chunk to our beta period and give us feedback. :)

Thanks for working on this @duckinator! This command has been a long time coming -- your patience and work on this is much appreciated! ^>^

@pradyunsg pradyunsg merged commit bdff935 into pypa:master Apr 13, 2020
36 checks passed
@pradyunsg
Copy link
Member

pradyunsg commented Apr 13, 2020

Alsssooo, thanks everyone who reviewed this or made suggestions here! Much appreciated! ^>^

@duckinator
Copy link
Contributor Author

duckinator commented Apr 13, 2020

@pradyunsg I'm excited that it's finally been merged! Do you know what the timing for releasing the 20.1 beta will be?

@pradyunsg
Copy link
Member

pradyunsg commented Apr 13, 2020

It's April 20th - #7951 is the tracking issue. :)

bors bot added a commit to duckinator/emanate that referenced this issue May 13, 2020
118: Update pip to 20.1 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.0.2** to **20.1**.



<details>
  <summary>Changelog</summary>
  
  
   ### 20.1
   ```
   =================

Process
-------

- Document that pip 21.0 will drop support for Python 2.7.

Features
--------

- Add ``pip cache dir`` to show the cache directory. (`7350 &lt;https://github.com/pypa/pip/issues/7350&gt;`_)

Bug Fixes
---------

- Abort pip cache commands early when cache is disabled. (`8124 &lt;https://github.com/pypa/pip/issues/8124&gt;`_)
- Correctly set permissions on metadata files during wheel installation,
  to permit non-privileged users to read from system site-packages. (`8139 &lt;https://github.com/pypa/pip/issues/8139&gt;`_)
   ```
   
  
  
   ### 20.1b1
   ```
   ===================

Deprecations and Removals
-------------------------

- Remove emails from AUTHORS.txt to prevent usage for spamming, and only populate names in AUTHORS.txt at time of release (`5979 &lt;https://github.com/pypa/pip/issues/5979&gt;`_)
- Remove deprecated ``--skip-requirements-regex`` option. (`7297 &lt;https://github.com/pypa/pip/issues/7297&gt;`_)
- Building of local directories is now done in place, instead of a temporary
  location containing a copy of the directory tree. (`7555 &lt;https://github.com/pypa/pip/issues/7555&gt;`_)
- Remove unused ``tests/scripts/test_all_pip.py`` test script and the ``tests/scripts`` folder. (`7680 &lt;https://github.com/pypa/pip/issues/7680&gt;`_)

Features
--------

- pip now implements PEP 610, so ``pip freeze`` has better fidelity
  in presence of distributions installed from Direct URL requirements. (`609 &lt;https://github.com/pypa/pip/issues/609&gt;`_)
- Add ``pip cache`` command for inspecting/managing pip&#39;s wheel cache. (`6391 &lt;https://github.com/pypa/pip/issues/6391&gt;`_)
- Raise error if ``--user`` and ``--target`` are used together in ``pip install`` (`7249 &lt;https://github.com/pypa/pip/issues/7249&gt;`_)
- Significantly improve performance when ``--find-links`` points to a very large HTML page. (`7729 &lt;https://github.com/pypa/pip/issues/7729&gt;`_)
- Indicate when wheel building is skipped, due to lack of the ``wheel`` package. (`7768 &lt;https://github.com/pypa/pip/issues/7768&gt;`_)
- Change default behaviour to always cache responses from trusted-host source. (`7847 &lt;https://github.com/pypa/pip/issues/7847&gt;`_)
- An alpha version of a new resolver is available via ``--unstable-feature=resolver``. (`988 &lt;https://github.com/pypa/pip/issues/988&gt;`_)

Bug Fixes
---------

- Correctly freeze a VCS editable package when it is nested inside another VCS repository. (`3988 &lt;https://github.com/pypa/pip/issues/3988&gt;`_)
- Correctly handle ``%2F`` in URL parameters to avoid accidentally unescape them
  into ``/``. (`6446 &lt;https://github.com/pypa/pip/issues/6446&gt;`_)
- Reject VCS URLs with an empty revision. (`7402 &lt;https://github.com/pypa/pip/issues/7402&gt;`_)
- Warn when an invalid URL is passed with ``--index-url`` (`7430 &lt;https://github.com/pypa/pip/issues/7430&gt;`_)
- Use better mechanism for handling temporary files, when recording metadata
  about installed files (RECORD) and the installer (INSTALLER). (`7699 &lt;https://github.com/pypa/pip/issues/7699&gt;`_)
- Correctly detect global site-packages availability of virtual environments
  created by PyPA’s virtualenv&gt;=20.0. (`7718 &lt;https://github.com/pypa/pip/issues/7718&gt;`_)
- Remove current directory from ``sys.path`` when invoked as ``python -m pip &lt;command&gt;`` (`7731 &lt;https://github.com/pypa/pip/issues/7731&gt;`_)
- Stop failing uninstallation, when trying to remove non-existent files. (`7856 &lt;https://github.com/pypa/pip/issues/7856&gt;`_)
- Prevent an infinite recursion with ``pip wheel`` when ``$TMPDIR`` is within the source directory. (`7872 &lt;https://github.com/pypa/pip/issues/7872&gt;`_)
- Significantly speedup ``pip list --outdated`` by parallelizing index interaction. (`7962 &lt;https://github.com/pypa/pip/issues/7962&gt;`_)
- Improve Windows compatibility when detecting writability in folder. (`8013 &lt;https://github.com/pypa/pip/issues/8013&gt;`_)

Vendored Libraries
------------------

- Update semi-supported debundling script to reflect that appdirs is vendored.
- Add ResolveLib as a vendored dependency.
- Upgrade certifi to 2020.04.05.1
- Upgrade contextlib2 to 0.6.0.post1
- Upgrade distro to 1.5.0.
- Upgrade idna to 2.9.
- Upgrade msgpack to 1.0.0.
- Upgrade packaging to 20.3.
- Upgrade pep517 to 0.8.2.
- Upgrade pyparsing to 2.4.7.
- Remove pytoml as a vendored dependency.
- Upgrade requests to 2.23.0.
- Add toml as a vendored dependency.
- Upgrade urllib3 to 1.25.8.

Improved Documentation
----------------------

- Emphasize that VCS URLs using git, git+git and git+http are insecure due to
  lack of authentication and encryption (`1983 &lt;https://github.com/pypa/pip/issues/1983&gt;`_)
- Clarify the usage of --no-binary command. (`3191 &lt;https://github.com/pypa/pip/issues/3191&gt;`_)
- Clarify the usage of freeze command in the example of Using pip in your program (`7008 &lt;https://github.com/pypa/pip/issues/7008&gt;`_)
- Add a &quot;Copyright&quot; page. (`7767 &lt;https://github.com/pypa/pip/issues/7767&gt;`_)
- Added example of defining multiple values for options which support them (`7803 &lt;https://github.com/pypa/pip/issues/7803&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
@lock lock bot added the auto-locked label May 20, 2020
@lock lock bot locked as resolved and limited conversation to collaborators May 20, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
auto-locked C: cache type: enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet