session: Cache http requests of trusted hosts

[NoahGorny]

As a follow-up to #7885, I want to also cache http responses if the host is specified as trusted. Back in the day, I was not sure how to do it, and I am still unsure whether my solution here works or not. I am welcoming a discussion about this change 😄

[uranusjr]

The idea does not sound unreasonable to me, but I’m a bit worried about security. The cache is global (to the user at least), while --trusted-host is per-command, so ideally we should only be able to hit the cache when --trusted-host is again given on a subsequent command. I’m not sure if that’s even possible though.

[NoahGorny]

The idea does not sound unreasonable to me, but I’m a bit worried about security. The cache is global (to the user at least), while --trusted-host is per-command, so ideally we should only be able to hit the cache when --trusted-host is again given on a subsequent command. I’m not sure if that’s even possible though.

If you cache a package from a host you specified as trusted- it should be trusted just any other package. When people abuse this flag and misuse it- it's a different problem, but it should not hurt the valid users, which do not get any caching whatsoever now.

@uranusjr I am also unsure whether this solution works at all, and I am not quite sure how can I test it

[uranusjr]

When people abuse this flag and misuse it- it's a different problem, but it should not hurt the valid users

The problem is exactly that misuse may hurt valid users. If something (say a script) calls pip under the hood, it can pollute the cache and affect all subsequent pip calls without the user knowing.

[NoahGorny]

When people abuse this flag and misuse it- it's a different problem, but it should not hurt the valid users

The problem is exactly that misuse may hurt valid users. If something (say a script) calls pip under the hood, it can pollute the cache and affect all subsequent pip calls without the user knowing.

I do not think that scripts should ever use trusted-host unless created by the same user that is using the script. I think that we can maybe add a warning about the dangers of trusted-host when invoking pip, and this way we can notify the user in case a script is secretly using it

[NoahGorny]

Hi @uranusjr, I have updated relevant tests, and I will be glad if you can take another look on this PR 😄

[pradyunsg]

Do we cache any other http protocol responses already?

[uranusjr]

We already cache HTTPS with a valid certificate. This adds caching to domains explicitly trusted by the user (via --trust-host).

[pradyunsg]

ACK. I have no blocking concerns based on a cursory look.

I think I'm ambivalent on this and, thus, I'll defer to someone else to click the merge button. :)

[NoahGorny]

I had troubles testing this..
I tried to set up a local pypi, and disabled the default "secure" status for localhost.
However, I saw that if we build the package from source, we cache the resulting wheel regardless of my change.
If we do not build from source, and use a wheel, the package is not cached with/without this PR, even when the adapter is the correct InsecureCacheControlAdapter. The package is just not cached..

Steps to reproduce:

# from source
pip install example
pip cache list | grep example
pip cache remove example; pip uninstall example

# set up local pypi with the tar, remove it from SECURE_ORIGINS
pip install example --index-url http://localhost:8080 --trusted-host localhost example
# still works, regardless of this PR or not
pip cache list | grep example
pip cache remove example; pip uninstall example
# Save the created wheel file and put in the pypi's packages dir
pip install example --index-url http://localhost:8080 --trusted-host localhost example
# Nothing is cached
pip cache list | grep example

Any help on this would be appreciated @uranusjr @pradyunsg

[uranusjr]

I think you’re checking the wrong cache. pip has an HTTP cache that caches all incoming HTTP responses in $cache/http, and a wheel cache that stores built wheels in $cache/wheels. You’re adding into the former, but pip cache is querying the latter.

[NoahGorny]

I think you’re checking the wrong cache. pip has an HTTP cache that caches all incoming HTTP responses in $cache/http, and a wheel cache that stores built wheels in $cache/wheels. You’re adding into the former, but pip cache is querying the latter.

Oh, okay
Is there any way to interact with the http cache?

[uranusjr]

The information is not exposed to end users, but for tests you can use SafeFileCache from pip._internal.network.cache, or implement your own with CacheControl.

[NoahGorny]

Howdy!
Sorry for the big delay @uranusjr @pradyunsg
I tested this and it seems like the wanted solution!

Let me know what you think 😄

[pradyunsg]

I think I'm ambivalent on this and, thus, I'll defer to someone else to click the merge button. :)

This is still true.

[uranusjr]

Personally I don’t want anyone to ever need to use --trusted-host, but if you do, you get what you get.

[notatallshaw]

Personally I don’t want anyone to ever need to use --trusted-host, but if you do, you get what you get.

Not sure if OP works in a large enterprise like myself, but just wanted to add I really appreciate this PR!

Unfortunately without --trusted-host there is currently no way obvious way for pip to work with our default authentication stack, which is a more general Python ecosystem issue (or a more general enterprise / awkward Microsoft authentication stack issue).

[uranusjr]

Given that Microsoft employees never seem to have filed any complaints against this, I’m inclined to think this is an issue with your company’s internal network admin staff.

[notatallshaw]

Given that Microsoft employees never seem to have filed any complaints against this, I’m inclined to think this is an issue with your company’s internal network admin staff.

The impression I get is Microsoft does not dog food the authentication stack that they create. Especially when it comes to using them with languages and ecosystems they did not create, e.g. Python.

Unfortunately there is no supported Python library that I'm aware of that features being able to pass client certificates from the Windows certificate store for authentication with end points or proxies (or both).

There are COM and CLR APIs, but patching tools like pip to replace them with a custom request back-end based on something like pythonnet is certainly a non-trivial exercise.

[uranusjr]

I’m pretty sure there is a Python library for this :p And swapping out the HTTP request backend is not that hard either. This deserves its own discussion though; iirc there is already a thread about providing a hook for plugging in custom request backends.

[notatallshaw]

I’m pretty sure there is a Python library for this :p

Well I and many others have been searching for 7+ years and have yet to find one.

We have an internal one that hacks together OpenSSL's CAPI (Microsoft CAPI not Python C-API) Engine with a pyopenssl context, but CAPI is deprecated and so doesn't support sites that require TLS 1.2+, and more bad news OpenSSL have deprecated Engines all together so the whole thing will eventually break.

iirc there is already a thread about providing a hook for plugging in custom request backends

That would be great.

Also, apologies for the off-topic spam, is there a way on GitHub to split off to a non-issue discussion?

[pradyunsg]

Also, apologies for the off-topic spam, is there a way on GitHub to split off to a non-issue discussion?

Not at this time, no. There's off-GitHub ways like IRC but nothing on-GitHub.

IMO, we're already buckling under the load of issue-related discussions and it's tricky to open up to communication channels that are more support-oriented, because 3-5 active pip maintainers can't possibly deal with the flood of questions that we'd get if we have something like that on GitHub. :)