Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cross-platform command to return pip's cache directory #7350

Closed
hugovk opened this issue Nov 13, 2019 · 9 comments · Fixed by #8095
Closed

Cross-platform command to return pip's cache directory #7350

hugovk opened this issue Nov 13, 2019 · 9 comments · Fixed by #8095
Labels
state: needs discussion This needs some more discussion type: feature request Request for a new feature

Comments

@hugovk
Copy link
Contributor

hugovk commented Nov 13, 2019

What's the problem this feature will solve?

From the command-line, I'd like a cross-platform method to get pip's cache directory, which by default is different per OS.

There's currently no supported way to do this.

Describe the solution you'd like

PR #6391 is adding pip cache info to returns the wheels directory, plus some extra info:

$ pip cache info
Cache info:
  Location: /Users/hugo/Library/Caches/pip/wheels
  Packages: 471

So something like pip cache dir could be a simplified version of that:

$ pip cache dir
/Users/hugo/Library/Caches/pip

This would be useful for caching with GitHub Actions CI. Right now, the config needs repeating three times, once per OS, which is rather cumbersome (actions/cache#86):

      - name: Ubuntu cache
        uses: actions/cache@v1
        if: startsWith(matrix.os, 'ubuntu')
        with:
          path: ~/.cache/pip
          key:
            ${{ matrix.os }}-${{ matrix.python-version }}-${{ hashFiles('**/setup.py')
            }}
          restore-keys: |
            ${{ matrix.os }}-${{ matrix.python-version }}-

      - name: macOS cache
        uses: actions/cache@v1
        if: startsWith(matrix.os, 'macOS')
        with:
          path: ~/Library/Caches/pip
          key:
            ${{ matrix.os }}-${{ matrix.python-version }}-${{ hashFiles('**/setup.py')
            }}
          restore-keys: |
            ${{ matrix.os }}-${{ matrix.python-version }}-

      - name: Windows cache
        uses: actions/cache@v1
        if: startsWith(matrix.os, 'windows')
        with:
          path: ~\AppData\Local\pip\Cache
          key:
            ${{ matrix.os }}-${{ matrix.python-version }}-${{ hashFiles('**/setup.py')
            }}
          restore-keys: |
            ${{ matrix.os }}-${{ matrix.python-version }}-

Other people also want a cross-platform method:

Alternative Solutions

  • The wrong way: use pip's internal API

Use pip's private, internal API, which has changed in the past and may change in the future:

$ python -c "from pip._internal.locations import USER_CACHE_DIR; print(USER_CACHE_DIR)"
/Users/hugo/Library/Caches/pip
  • Another way: change pip's cache dir

Provide --cache-dir or set the PIP_CACHE_DIR environment variable to whatever path you like and cache that. Or use --user to install into the user directory, and cache from there.

However, ideally I'd like not to change pip's behaviour in any way.

We also test on other CIs, and locally, and I'd like pip to use its defaults as much as possible across the board, and have fewer differences across envs.

@triage-new-issues triage-new-issues bot added the S: needs triage Issues/PRs that need to be triaged label Nov 13, 2019
@chrahunt
Copy link
Member

chrahunt commented Nov 13, 2019

Alternatively, a --json/--format=json could be added to the upcoming pip cache info, then anyone can extract the desired fields using something like jq?

@chrahunt chrahunt added state: needs discussion This needs some more discussion type: feature request Request for a new feature labels Nov 13, 2019
@triage-new-issues triage-new-issues bot removed the S: needs triage Issues/PRs that need to be triaged label Nov 13, 2019
@hugovk
Copy link
Contributor Author

hugovk commented Nov 14, 2019

Yes, that's one option, and would allow the total number of packages to be consumed programatically, if anyone needed it.

Although I had suggested this, which doesn't include the wheels subdirectory:

$ pip cache dir
/Users/hugo/Library/Caches/pip

pip cache info includes the wheels subdir:

$ pip cache info
Cache info:
  Location: /Users/hugo/Library/Caches/pip/wheels
  Packages: 508

Would that be the wrong thing to cache on a CI? Should http/, selfcheck/ and selfcheck.json be included or skipped when caching?

$ ls -l /Users/hugo/Library/Caches/pip/
total 48
drwx------    6 hugo  staff   192B 13 Nov 12:09 .
drwx------+ 155 hugo  staff   4.8K 14 Nov 16:57 ..
drwx------   18 hugo  staff   576B  8 Nov  2017 http
drwxr-xr-x   46 hugo  staff   1.4K 14 Nov 12:20 selfcheck
-rw-r--r--    1 hugo  staff    22K 13 Nov 12:09 selfcheck.json
drwxr-xr-x  214 hugo  staff   6.7K 12 Nov 10:22 wheels

(That's on macOS, I've not checked the structure for Linux/Windows.)


Here's the 14 lines needed for pip cache dir, done before you suggested --json:

@chrahunt
Copy link
Member

chrahunt commented Nov 15, 2019

With unlimited space, fast cache upload, and a cache that gets re-uploaded after each run, all of the directories you mentioned (except maybe selfcheck, but it should be small) would probably be worth caching. If any of those aren't true, then it depends. Based on the limits described in the docs it's probably not worth thinking about until you hit the limit.

@pradyunsg pradyunsg mentioned this issue Nov 15, 2019
10 tasks
@hugovk
Copy link
Contributor Author

hugovk commented Nov 15, 2019

Here's an example of how pip cache info --json could be implemented:

$ pip cache info
Cache info:
  Root: /Users/hugo/Library/Caches/pip
  Wheels: /Users/hugo/Library/Caches/pip/wheels
  Packages: 508
$ pip cache info --json
{"root": "/Users/hugo/Library/Caches/pip", "wheels": "/Users/hugo/Library/Caches/pip/wheels", "packages": 508}

One thing that would potentially affect #6391 is including the root cache path in addition to the wheels path. And renaming the wheels one from "Location" to "Wheels" and adding "Root" (or other names).

@duckinator
Copy link
Contributor

duckinator commented Nov 17, 2019

I'm in favor of adding this, and think it makes sense as part of the pip cache command, but I think it'd probably be best as a follow-up PR to #6391.

@hugovk
Copy link
Contributor Author

hugovk commented Nov 17, 2019

I agree, I'd like to see #6391 land first and don't wish to delay it.

Just one thing to consider: perhaps rename "Location:" to "Wheels:" in #6391 to avoid doing it later here. But again, bikeshedding/naming also takes time so we can do it here to avoid delaying #6391 :)

And I've created #7372 for #6391 (comment).

@hugovk
Copy link
Contributor Author

hugovk commented Apr 15, 2020

#6391 has been merged! 🎉


Two options have been suggested here for this issue, my initial pip cache dir and @chrahunt suggested pip cache list --json/--format=json.

I don't have a preference, and have updated these branches for each option.

Which one is preferred? I'll open a PR for it and we can discuss specifics. Thank you!

@duckinator
Copy link
Contributor

duckinator commented Apr 20, 2020

Honestly, I think they're both useful in different contexts.

  • In general, having machine-readable versions of things (as the --json/--format=json flag provides) can be very useful,
  • However, it simultaneously feels a bit silly to require a separate tool (e.g. jq) to parse data out for what is likely the most commonly-wanted value (the cache directory).

@hugovk
Copy link
Contributor Author

hugovk commented Apr 20, 2020

Thanks for the comments! I agree.

To start things off, I've created PR #8095 for pip cache dir. It's simpler to implement, and is easier to use for my actual use case: checking the cache dir on a CI.

bors bot added a commit to duckinator/emanate that referenced this issue May 13, 2020
118: Update pip to 20.1 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.0.2** to **20.1**.



<details>
  <summary>Changelog</summary>
  
  
   ### 20.1
   ```
   =================

Process
-------

- Document that pip 21.0 will drop support for Python 2.7.

Features
--------

- Add ``pip cache dir`` to show the cache directory. (`7350 &lt;https://github.com/pypa/pip/issues/7350&gt;`_)

Bug Fixes
---------

- Abort pip cache commands early when cache is disabled. (`8124 &lt;https://github.com/pypa/pip/issues/8124&gt;`_)
- Correctly set permissions on metadata files during wheel installation,
  to permit non-privileged users to read from system site-packages. (`8139 &lt;https://github.com/pypa/pip/issues/8139&gt;`_)
   ```
   
  
  
   ### 20.1b1
   ```
   ===================

Deprecations and Removals
-------------------------

- Remove emails from AUTHORS.txt to prevent usage for spamming, and only populate names in AUTHORS.txt at time of release (`5979 &lt;https://github.com/pypa/pip/issues/5979&gt;`_)
- Remove deprecated ``--skip-requirements-regex`` option. (`7297 &lt;https://github.com/pypa/pip/issues/7297&gt;`_)
- Building of local directories is now done in place, instead of a temporary
  location containing a copy of the directory tree. (`7555 &lt;https://github.com/pypa/pip/issues/7555&gt;`_)
- Remove unused ``tests/scripts/test_all_pip.py`` test script and the ``tests/scripts`` folder. (`7680 &lt;https://github.com/pypa/pip/issues/7680&gt;`_)

Features
--------

- pip now implements PEP 610, so ``pip freeze`` has better fidelity
  in presence of distributions installed from Direct URL requirements. (`609 &lt;https://github.com/pypa/pip/issues/609&gt;`_)
- Add ``pip cache`` command for inspecting/managing pip&#39;s wheel cache. (`6391 &lt;https://github.com/pypa/pip/issues/6391&gt;`_)
- Raise error if ``--user`` and ``--target`` are used together in ``pip install`` (`7249 &lt;https://github.com/pypa/pip/issues/7249&gt;`_)
- Significantly improve performance when ``--find-links`` points to a very large HTML page. (`7729 &lt;https://github.com/pypa/pip/issues/7729&gt;`_)
- Indicate when wheel building is skipped, due to lack of the ``wheel`` package. (`7768 &lt;https://github.com/pypa/pip/issues/7768&gt;`_)
- Change default behaviour to always cache responses from trusted-host source. (`7847 &lt;https://github.com/pypa/pip/issues/7847&gt;`_)
- An alpha version of a new resolver is available via ``--unstable-feature=resolver``. (`988 &lt;https://github.com/pypa/pip/issues/988&gt;`_)

Bug Fixes
---------

- Correctly freeze a VCS editable package when it is nested inside another VCS repository. (`3988 &lt;https://github.com/pypa/pip/issues/3988&gt;`_)
- Correctly handle ``%2F`` in URL parameters to avoid accidentally unescape them
  into ``/``. (`6446 &lt;https://github.com/pypa/pip/issues/6446&gt;`_)
- Reject VCS URLs with an empty revision. (`7402 &lt;https://github.com/pypa/pip/issues/7402&gt;`_)
- Warn when an invalid URL is passed with ``--index-url`` (`7430 &lt;https://github.com/pypa/pip/issues/7430&gt;`_)
- Use better mechanism for handling temporary files, when recording metadata
  about installed files (RECORD) and the installer (INSTALLER). (`7699 &lt;https://github.com/pypa/pip/issues/7699&gt;`_)
- Correctly detect global site-packages availability of virtual environments
  created by PyPA’s virtualenv&gt;=20.0. (`7718 &lt;https://github.com/pypa/pip/issues/7718&gt;`_)
- Remove current directory from ``sys.path`` when invoked as ``python -m pip &lt;command&gt;`` (`7731 &lt;https://github.com/pypa/pip/issues/7731&gt;`_)
- Stop failing uninstallation, when trying to remove non-existent files. (`7856 &lt;https://github.com/pypa/pip/issues/7856&gt;`_)
- Prevent an infinite recursion with ``pip wheel`` when ``$TMPDIR`` is within the source directory. (`7872 &lt;https://github.com/pypa/pip/issues/7872&gt;`_)
- Significantly speedup ``pip list --outdated`` by parallelizing index interaction. (`7962 &lt;https://github.com/pypa/pip/issues/7962&gt;`_)
- Improve Windows compatibility when detecting writability in folder. (`8013 &lt;https://github.com/pypa/pip/issues/8013&gt;`_)

Vendored Libraries
------------------

- Update semi-supported debundling script to reflect that appdirs is vendored.
- Add ResolveLib as a vendored dependency.
- Upgrade certifi to 2020.04.05.1
- Upgrade contextlib2 to 0.6.0.post1
- Upgrade distro to 1.5.0.
- Upgrade idna to 2.9.
- Upgrade msgpack to 1.0.0.
- Upgrade packaging to 20.3.
- Upgrade pep517 to 0.8.2.
- Upgrade pyparsing to 2.4.7.
- Remove pytoml as a vendored dependency.
- Upgrade requests to 2.23.0.
- Add toml as a vendored dependency.
- Upgrade urllib3 to 1.25.8.

Improved Documentation
----------------------

- Emphasize that VCS URLs using git, git+git and git+http are insecure due to
  lack of authentication and encryption (`1983 &lt;https://github.com/pypa/pip/issues/1983&gt;`_)
- Clarify the usage of --no-binary command. (`3191 &lt;https://github.com/pypa/pip/issues/3191&gt;`_)
- Clarify the usage of freeze command in the example of Using pip in your program (`7008 &lt;https://github.com/pypa/pip/issues/7008&gt;`_)
- Add a &quot;Copyright&quot; page. (`7767 &lt;https://github.com/pypa/pip/issues/7767&gt;`_)
- Added example of defining multiple values for options which support them (`7803 &lt;https://github.com/pypa/pip/issues/7803&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 15, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
state: needs discussion This needs some more discussion type: feature request Request for a new feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants