Wheel cache link look up in the new resolver

[uranusjr]

Very much a work in progress. I managed to fix test_pip_wheel_build_cache, but not test_install_git_sha_cached. I’m not even sure if the latter one is actually related to the wheel cache now.

I’m also having trouble implementing PEP 610 (equivalent of #7612). test_install_find_links_no_direct_url does not pass, and I can’t figure out if my original_link_is_in_wheel_cache is sane. Are these two the same thing? I have no idea :(

[uranusjr]

Alright, I got test_install_git_sha_cached() sorted. Turns out it is indeed related to the wheel cache, and I was simply not passing the correct information to it.

@sbidoul Would you mind clarifying what original_link_is_in_wheel_cache means, and what kind of input sets it True? I get it’s False most of the time. The only usage I find it makes a difference is in direct_url_from_link(), when link is VCS. The only place modifying it is the resolver’s _populate_link(), when there is a persistent cache match. What are the expected values of req.link and req.original_link here, and what does it mean when they point to the same Link?

---

Edit: I’m failing test_install_vcs_non_editable_direct_url, which seems related.

[sbidoul]

@uranusjr original_link is the link available when the requirement is initialized. link and original_link start identical. Over the build/install process link may become different from original_link, e.g. when it is unpacked to a local directory, or when it is replaced by a cache entry.

In _populate_link, this

pip/src/pip/_internal/resolution/legacy/resolver.py

Lines 290 to 291 in 6f7d5b9

	if req.link is req.original_link and cache_entry.persistent:
	req.original_link_is_in_wheel_cache = True

is meant to record if the original link was found in cache. As you noted, this only necessary so far to know that the VCS link contained a commit id without actually doing a VCS checkout.

'hope this helps.

I've tried to look into this PR but I'm still struggling with the concept of creating an InstallRequirement from a parent InstallRequirement and a link.

[uranusjr]

In _populate_link, this […] is meant to record if the original link was found in cache. As you noted, this only necessary so far to know that the VCS link contained a commit id without actually doing a VCS checkout.

So what are the implications of req.link is req.original_link? Or to put it another way, in what cases would req.link be changed at this point? Because if I understand correctly, these two would always be identical if req was constructed with a link; the unpack/build process are yet to happen at this point (and the cache assignment is the next line so also not happened yet).

---

The name parent definitely can be confusing here (in fact @pradyunsg was wondering about renaming it just yesterday). The idea behind it is that we don’t want to keep mutating InstallRequirement instances (because it makes things difficult for backtracking). We instead “clone” the InstallRequirement backing a requirement to construct a candidate, so we can still have the unbuilt InstallRequirement to work with on backtracking. parent is the InstallRequirement to clone from.

[uranusjr]

Interlinking #8005 (comment)

[uranusjr]

I think it’s better to pause work on this until #8005 goes in. The internal link state in InstallRequirement changes too much with it I have trouble getting my head straight what this change would affect.

[uranusjr]

Also need to consider #8123 because wheel cache lookup loses us the yanked information.

[BrownTruck]

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

[uranusjr]

Yay, I think I’ve managed to solve the PEP 610 related logic. We still need to deal with the problem that a wheel cache lookup looses the link.is_yanked context, but I think it’s better to think of a way on top of this implementation. The yanked issue is now tracked in #8202, and will be worked on after this goes in.

[BrownTruck]

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

[pradyunsg]

Assuming the answer to the TODO is "yes", LGTM!

[pfmoore]

Other than one typo and a couple of minor comments, this LGTM

[pradyunsg]

I'm faster than you @BrownTruck!

[pradyunsg]

Python 2 is always going to be a PITA. :)

[uranusjr]

The “Linux” job is stale from before the CI change. It will stay failed because now I can’t trigger it to pass anymore (without readding the job).

[pfmoore]

The “Linux” job is stale from before the CI change.

Is that an Azure thing? Why isn't it possible to rerun the job? I have a "Re-run" link on the check results page, I'll click that in case it helps.

[pfmoore]

Ah yes, I see, that was one of the renamings @pradyunsg did.

OK, so I guess the only options is to rebase on master. Merging with failing checks isn't allowed.

[pradyunsg]

You won this race Paul, but I won't say that was graceful. I'd taken a break for dinner. :P

[sbidoul]

@uranusjr

I managed to fix test_pip_wheel_build_cache, but not test_install_git_sha_cached. I’m not even sure if the latter one is actually related to the wheel cache now.

test_install_git_sha_cached is definitely related to the wheel cache.

When installing a cached VCS requirement, I can see that the cache lookup from this PR does correctly find the cache entry. However when reaching prepare_linked_requirement I see that req.link is the VCS link instead of a link to the cache entry (and since is_wheel is false for that link, it tries to git clone it).

[pradyunsg]

OOoo. So, I just noticed that this breaks the presentation logic in RequirementPreparer.prepare_linked_requirement. :(

Processing /Users/pradyunsg/Library/Caches/pip/wheels/da/f1/6f/450a8e3042772f552f5316cfbd51552bb13995077e4502ba2f/rackspace_novaclient-2.1-py3-none-any.whl
Processing /Users/pradyunsg/Library/Caches/pip/wheels/45/42/fc/37502e2e6b8c5e745d2ec318c6d1ee9328ead3d704b5584797/ip_associations_python_novaclient_ext-0.2-py3-none-any.whl
Processing /Users/pradyunsg/Library/Caches/pip/wheels/b8/0c/c5/f4087714c89b18994a27c42c01bd9a8ce3bfcccf5d72ff978e/rackspace_auth_openstack-1.3-py3-none-any.whl
Processing /Users/pradyunsg/Library/Caches/pip/wheels/bc/eb/b2/2d27f5c5d3d8caecebbf87ccb272f60ee278d968c1c3a10c85/os_diskconfig_python_novaclient_ext-0.1.3-py3-none-any.whl
Processing /Users/pradyunsg/Library/Caches/pip/wheels/25/01/b7/d60cc1e3f2b4f9e5b359ce1b7fe03f65084020bfb0d4edcca2/os_networksv2_python_novaclient_ext-0.26-py3-none-any.whl
Processing /Users/pradyunsg/Library/Caches/pip/wheels/64/ce/0e/e407508c11ef0d63567a51a86302b44de81b78f602d7e8dc82/rax_scheduled_images_python_novaclient_ext-0.3.1-py3-none-any.whl
Processing /Users/pradyunsg/Library/Caches/pip/wheels/1d/b7/c2/8988a8a46e720cced366730c323604d1dba6644886f9a666aa/rax_default_network_flags_python_novaclient_ext-0.4.0-py3-none-any.whl
Processing /Users/pradyunsg/Library/Caches/pip/wheels/89/7c/51/371c863bd836d32c94f518678e612386ea757e4b195a080c78/os_virtual_interfacesv2_python_novaclient_ext-0.20-py3-none-any.whl
Collecting python-novaclient
  Using cached python_novaclient-17.2.1-py3-none-any.whl (319 kB)
Processing /Users/pradyunsg/Library/Caches/pip/wheels/48/6d/77/9517cb933af254f51a446f1a5ec9c2be3e45f17384940bce68/prettytable-0.7.2-py3-none-any.whl
Collecting iso8601>=0.1.11
  Using cached iso8601-0.1.12-py3-none-any.whl (12 kB)
Collecting stevedore>=2.0.1
  Using cached stevedore-3.2.1-py3-none-any.whl (42 kB)
Collecting oslo.i18n>=3.15.3
  Using cached oslo.i18n-5.0.0-py3-none-any.whl (46 kB)
Collecting six>=1.10.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting oslo.serialization!=2.19.1,>=2.18.0
  Using cached oslo.serialization-4.0.0-py3-none-any.whl (25 kB)
Collecting msgpack>=0.5.2
  Using cached msgpack-1.0.0-cp38-cp38-macosx_10_13_x86_64.whl (78 kB)
Collecting simplejson>=3.5.1
  Using cached simplejson-3.17.2-cp38-cp38-macosx_10_14_x86_64.whl (74 kB)
Collecting keystoneauth1>=3.5.0
  Using cached keystoneauth1-4.2.1-py3-none-any.whl (308 kB)

Notice the initial wheel files that are from the cache, printed in full.