Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trusted Hosts Aren't Cached #7847

Closed
natedogith1 opened this issue Mar 11, 2020 · 5 comments · Fixed by #7885
Closed

Trusted Hosts Aren't Cached #7847

natedogith1 opened this issue Mar 11, 2020 · 5 comments · Fixed by #7885
Labels
C: cache type: enhancement

Comments

@natedogith1
Copy link

@natedogith1 natedogith1 commented Mar 11, 2020

Environment

  • pip version: 20.0.2
  • Python version: 3.7.3
  • OS: CentOS Linux release 7.4.1708
  • Virtualenv version: 16.6.1 (ran as "virtualenv --python=3.7 build/python_env")

Description
If a host is added as a trusted-host, it won't use the cache.

Expected behavior
trusted-hosts use the cache.

How to Reproduce

  1. pip config --site set global.index-url https://example.com/api/pypi/pypi/simple/
  2. pip config --site set global.trusted-host example.com
  3. pip install -r pip_requirements.txt

Output
Line from broken output:

Collecting requests==2.23.0
  Downloading https://artifactory.rnclab.us.alcatel-lucent.com/artifactory/api/pypi/pypi/packages/1a/70/1935c770cb3be6e3a8b78ced23d7e0f3b187f5cbfab4749523ed65d7c9b1/requests-2.23.0-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 3.0 MB/s

Line from non-broken output (not configuring trusted-host):

Collecting requests==2.23.0
  Using cached https://example.com/artifactory/api/pypi/pypi/packages/1a/70/1935c770cb3be6e3a8b78ced23d7e0f3b187f5cbfab4749523ed65d7c9b1/requests-2.23.0-py2.py3-none-any.whl (58 kB)
@triage-new-issues triage-new-issues bot added the S: needs triage label Mar 11, 2020
@NoahGorny
Copy link
Contributor

@NoahGorny NoahGorny commented Mar 23, 2020

As you can see in commit 5bab65d, a change was introduced that did not cache responses for any "unsafe" origin, such as http, or invalid https, even with the "trusted-host" option.

I also encounter this behavior as in my workplace I must use "trusted-host" as well, and it is pretty annoying that it does not cache any responses at all, so I have 2 options to offer:

  1. Change this behavior to allow caching from any "trusted-host" source. Makes sense as you specifically said the host should be trusted. Maybe refrase the "trusted-host" docs to specify that. This option however changes default behavior and more drastic.
  2. New command line option called cache-trusted-host that will do just that, then you can specify this option in pip config etc...

Out of these two option I prefer the first, but I think we should have a discussion about it.
In any way this is not a lot of diff, I can open a PR if needed :)

NoahGorny added a commit to NoahGorny/pip that referenced this issue Mar 23, 2020
Allows users to cache results even from unsafe yet trusted hosts
Resolves pypa#7847
NoahGorny added a commit to NoahGorny/pip that referenced this issue Mar 23, 2020
Allows users to cache results even from unsafe yet trusted hosts
Resolves pypa#7847
@NoahGorny NoahGorny mentioned this issue Mar 23, 2020
2 tasks
@NoahGorny
Copy link
Contributor

@NoahGorny NoahGorny commented Mar 23, 2020

hey @pradyunsg. can you please take a look at this and the PR I opened to solve it? 😄

@d3dave
Copy link

@d3dave d3dave commented Mar 23, 2020

The change discussed for not caching packages from insecure origins was introduced alongside the --trusted-host flag in pip version 6. Then, installing from insecure origins was deprecated but not disabled (pip reported a warning, but still fetched the packages). As a result, the cache could still be poisoned by untrusted origins.

Since version 7, pip does not allow fetching packages from insecure origins unless they are explicitly marked trusted. As a result, the original change for not caching packages loses merit, because the cache cannot be poisoned anymore by untrusted origins (pip refuses to fetch the packages).

In light of the above, I believe the correct solution here is option 1.

@sbidoul
Copy link
Member

@sbidoul sbidoul commented Mar 25, 2020

Could it be a third option to use --cert with a custom certificate bundle, as an alternative way to trust that host?

@NoahGorny
Copy link
Contributor

@NoahGorny NoahGorny commented Mar 25, 2020

In my workplace some certs are expired, some use weak algorithms that are no longer supported, etc...
It would take years for me to convince the IT department to do anything about it, and I do not think that I (or my workplace) are unique in this regard.
I am fine with caching "insecure" sites when I have no other option to do otherwise

@pradyunsg pradyunsg added the C: cache label Apr 21, 2020
@triage-new-issues triage-new-issues bot removed the S: needs triage label Apr 21, 2020
@pradyunsg pradyunsg added the type: enhancement label Apr 21, 2020
bors bot added a commit to duckinator/emanate that referenced this issue May 13, 2020
118: Update pip to 20.1 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.0.2** to **20.1**.



<details>
  <summary>Changelog</summary>
  
  
   ### 20.1
   ```
   =================

Process
-------

- Document that pip 21.0 will drop support for Python 2.7.

Features
--------

- Add ``pip cache dir`` to show the cache directory. (`7350 &lt;https://github.com/pypa/pip/issues/7350&gt;`_)

Bug Fixes
---------

- Abort pip cache commands early when cache is disabled. (`8124 &lt;https://github.com/pypa/pip/issues/8124&gt;`_)
- Correctly set permissions on metadata files during wheel installation,
  to permit non-privileged users to read from system site-packages. (`8139 &lt;https://github.com/pypa/pip/issues/8139&gt;`_)
   ```
   
  
  
   ### 20.1b1
   ```
   ===================

Deprecations and Removals
-------------------------

- Remove emails from AUTHORS.txt to prevent usage for spamming, and only populate names in AUTHORS.txt at time of release (`5979 &lt;https://github.com/pypa/pip/issues/5979&gt;`_)
- Remove deprecated ``--skip-requirements-regex`` option. (`7297 &lt;https://github.com/pypa/pip/issues/7297&gt;`_)
- Building of local directories is now done in place, instead of a temporary
  location containing a copy of the directory tree. (`7555 &lt;https://github.com/pypa/pip/issues/7555&gt;`_)
- Remove unused ``tests/scripts/test_all_pip.py`` test script and the ``tests/scripts`` folder. (`7680 &lt;https://github.com/pypa/pip/issues/7680&gt;`_)

Features
--------

- pip now implements PEP 610, so ``pip freeze`` has better fidelity
  in presence of distributions installed from Direct URL requirements. (`609 &lt;https://github.com/pypa/pip/issues/609&gt;`_)
- Add ``pip cache`` command for inspecting/managing pip&#39;s wheel cache. (`6391 &lt;https://github.com/pypa/pip/issues/6391&gt;`_)
- Raise error if ``--user`` and ``--target`` are used together in ``pip install`` (`7249 &lt;https://github.com/pypa/pip/issues/7249&gt;`_)
- Significantly improve performance when ``--find-links`` points to a very large HTML page. (`7729 &lt;https://github.com/pypa/pip/issues/7729&gt;`_)
- Indicate when wheel building is skipped, due to lack of the ``wheel`` package. (`7768 &lt;https://github.com/pypa/pip/issues/7768&gt;`_)
- Change default behaviour to always cache responses from trusted-host source. (`7847 &lt;https://github.com/pypa/pip/issues/7847&gt;`_)
- An alpha version of a new resolver is available via ``--unstable-feature=resolver``. (`988 &lt;https://github.com/pypa/pip/issues/988&gt;`_)

Bug Fixes
---------

- Correctly freeze a VCS editable package when it is nested inside another VCS repository. (`3988 &lt;https://github.com/pypa/pip/issues/3988&gt;`_)
- Correctly handle ``%2F`` in URL parameters to avoid accidentally unescape them
  into ``/``. (`6446 &lt;https://github.com/pypa/pip/issues/6446&gt;`_)
- Reject VCS URLs with an empty revision. (`7402 &lt;https://github.com/pypa/pip/issues/7402&gt;`_)
- Warn when an invalid URL is passed with ``--index-url`` (`7430 &lt;https://github.com/pypa/pip/issues/7430&gt;`_)
- Use better mechanism for handling temporary files, when recording metadata
  about installed files (RECORD) and the installer (INSTALLER). (`7699 &lt;https://github.com/pypa/pip/issues/7699&gt;`_)
- Correctly detect global site-packages availability of virtual environments
  created by PyPA’s virtualenv&gt;=20.0. (`7718 &lt;https://github.com/pypa/pip/issues/7718&gt;`_)
- Remove current directory from ``sys.path`` when invoked as ``python -m pip &lt;command&gt;`` (`7731 &lt;https://github.com/pypa/pip/issues/7731&gt;`_)
- Stop failing uninstallation, when trying to remove non-existent files. (`7856 &lt;https://github.com/pypa/pip/issues/7856&gt;`_)
- Prevent an infinite recursion with ``pip wheel`` when ``$TMPDIR`` is within the source directory. (`7872 &lt;https://github.com/pypa/pip/issues/7872&gt;`_)
- Significantly speedup ``pip list --outdated`` by parallelizing index interaction. (`7962 &lt;https://github.com/pypa/pip/issues/7962&gt;`_)
- Improve Windows compatibility when detecting writability in folder. (`8013 &lt;https://github.com/pypa/pip/issues/8013&gt;`_)

Vendored Libraries
------------------

- Update semi-supported debundling script to reflect that appdirs is vendored.
- Add ResolveLib as a vendored dependency.
- Upgrade certifi to 2020.04.05.1
- Upgrade contextlib2 to 0.6.0.post1
- Upgrade distro to 1.5.0.
- Upgrade idna to 2.9.
- Upgrade msgpack to 1.0.0.
- Upgrade packaging to 20.3.
- Upgrade pep517 to 0.8.2.
- Upgrade pyparsing to 2.4.7.
- Remove pytoml as a vendored dependency.
- Upgrade requests to 2.23.0.
- Add toml as a vendored dependency.
- Upgrade urllib3 to 1.25.8.

Improved Documentation
----------------------

- Emphasize that VCS URLs using git, git+git and git+http are insecure due to
  lack of authentication and encryption (`1983 &lt;https://github.com/pypa/pip/issues/1983&gt;`_)
- Clarify the usage of --no-binary command. (`3191 &lt;https://github.com/pypa/pip/issues/3191&gt;`_)
- Clarify the usage of freeze command in the example of Using pip in your program (`7008 &lt;https://github.com/pypa/pip/issues/7008&gt;`_)
- Add a &quot;Copyright&quot; page. (`7767 &lt;https://github.com/pypa/pip/issues/7767&gt;`_)
- Added example of defining multiple values for options which support them (`7803 &lt;https://github.com/pypa/pip/issues/7803&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 15, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
C: cache type: enhancement
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants