Wheel cache is not forward/backward compatible between 22.2.x and 22.3

[cboylan]

Description

Installing packages and having the resulting wheels go into a cache dir with pip 22.3 produces a cache that appears to be ignored by 22.2.2 forcing 22.2.2 to install from sdists. The same thing seems to happen if you create the wheel cache with 22.2.2 and then try to install packages with that cache using 22.3.

We noticed this because we've got some multistage docker image builds that will install all build requirements (like gcc) and create wheels for all deps in a throwaway image then copy only the wheels over to the final image and pip install what we need there using the wheel cache to avoid needing build requirements like gcc. This broke when the pip versions on either side of that got out of sync.

I don't know if this was an expected change (it hasn't been a problem in the past and I strongly suspect that we've gotten out of sync previously). This was also not called out in the release notes. Just want to make sure this gets reported even if ultimately this was expected.

Expected behavior

I expected that the cache would be forward compatible at least. It seems a little odd to force users to rebuild all new wheels when they update pip versions. Old pip recognizing newer pips wheel cache and using it is maybe a bit less expected (though would've been nice in our situation).

That said I'm not entirely sure there is a bug here. I'm mostly interested in calling the behavior change out to double check that this was expected. And to see if this sort of behavior change is worthy of a release note.

pip version

22.3

Python version

3.10

OS

Debian Bullseye, OpenSUSE Tumbleweed

How to Reproduce

A working cache with the same pip version across environments:

> python3 -m venv venv1
> venv1/bin/python3 --version
Python 3.10.7
> venv1/bin/pip --version
pip 22.2.2 from /home/clark/tmp/venv1/lib64/python3.10/site-packages/pip (python 3.10)
> venv1/bin/pip install build wheel
Collecting build
  Using cached build-0.8.0-py3-none-any.whl (17 kB) 
Collecting wheel
  Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB) 
Collecting packaging>=19.0
  Using cached packaging-21.3-py3-none-any.whl (40 kB) 
Collecting pep517>=0.9.1
  Using cached pep517-0.13.0-py3-none-any.whl (18 kB) 
Collecting tomli>=1.0.0
  Using cached tomli-2.0.1-py3-none-any.whl (12 kB) 
Collecting pyparsing!=3.0.5,>=2.0.2
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB) 
Installing collected packages: wheel, tomli, pyparsing, pep517, packaging, build
Successfully installed build-0.8.0 packaging-21.3 pep517-0.13.0 pyparsing-3.0.9 tomli-2.0.1 wheel-0.37.1

[notice] A new release of pip available: 22.2.2 -> 22.3
[notice] To update, run: python3 -m pip install --upgrade pip 
> venv1/bin/pip install --cache-dir ./pip-testing/wheels netifaces
Collecting netifaces
  Downloading netifaces-0.11.0.tar.gz (30 kB) 
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: netifaces
  Building wheel for netifaces (setup.py) ... done
  Created wheel for netifaces: filename=netifaces-0.11.0-cp310-cp310-linux_x86_64.whl size=35306 sha256=0bc5bfaf024c86104588db92a0124cf6d181d45c1935bccd75ec0812b7327644
  Stored in directory: /home/clark/tmp/pip-testing/wheels/wheels/48/65/b3/4c4cc6038b81ff21cc9df69f2b6774f5f52e23d3c275ed15aa
Successfully built netifaces
Installing collected packages: netifaces
Successfully installed netifaces-0.11.0

[notice] A new release of pip available: 22.2.2 -> 22.3
[notice] To update, run: python3 -m pip install --upgrade pip 
> python3 -m venv venv2
> venv2/bin/python --version
Python 3.10.7
> venv2/bin/pip --version
pip 22.2.2 from /home/clark/tmp/venv2/lib64/python3.10/site-packages/pip (python 3.10)
> venv2/bin/pip install --cache-dir ./pip-testing/wheels netifaces
Collecting netifaces
  Using cached netifaces-0.11.0-cp310-cp310-linux_x86_64.whl
Installing collected packages: netifaces
Successfully installed netifaces-0.11.0

[notice] A new release of pip available: 22.2.2 -> 22.3
[notice] To update, run: python3 -m pip install --upgrade pip

And now using the same wheel cache but with a different virtualenv that has newer pip installed it fallsback to building from the sdist:

> python3 -m venv venv2
> venv2/bin/python --version
Python 3.10.7
> venv2/bin/pip --version
pip 22.3 from /home/clark/tmp/venv2/lib64/python3.10/site-packages/pip (python 3.10)
> venv2/bin/pip install --cache-dir ./pip-testing/wheels netifaces
Collecting netifaces
  Using cached netifaces-0.11.0.tar.gz (30 kB)
  Preparing metadata (setup.py) ... done
Installing collected packages: netifaces
  DEPRECATION: netifaces is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for netifaces ... done
Successfully installed netifaces-0.11.0

Note netifaces was chosen as an example here because it does not publish python3.10 wheels on pypi and its install is quick and simple. I don't expect this to be a package specific issue, but I suppose that is possible as well.

Output

Output was provided inline with the reproduction steps above.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

The format of the cache isn't public, and you shouldn't really be using the cache like this. I don't know if it was accidental or deliberate that the cache format changed - I would expect that we'd keep it stable when we can, simply because it makes it work better as a cache, but we don't guarantee that.

For the use case you describe, you probably shouldn't be doing this anyway. It seems like you'd be better using pip wheel to download/build the wheels you want, and then copying those wheels to the target and using --find-links so that pip install will find the wheels when installing on the target. This is likely to be a lot more robust than your current approach.

[fungi]

So, just to be clear, there's no expectation that a new version of pip will be able to use a wheel cache created by the previous version, and such behavior changes are neither considered a regression nor do they rise to the level of requiring a release note.

Should pip's documentation state that, and perhaps also recommend clearing your wheel cache as a part of each upgrade?

[pfmoore]

That's a lot stronger than what I said, which was that we don't make any particular promises, but we won't change the cache format without good reason.

Also, it's a cache, intended purely to improve performance by avoiding unnecessary rebuilds. If you want to guarantee pip won't try to build on the target, you should be using pip wheel and --find-links as I said.

[fungi]

Setting the reported use case where this was discovered aside entirely, I'm just wondering more generally if cache layout/format changes for pip are something which should be mentioned in release notes so that people upgrading know when they should blow away the cache because its contents have been rendered useless and now it's only full of cruft, or if it's simply good hygiene to do that at every pip upgrade on the basis that it may have changed.

[cboylan]

I did more digging into why this is happening and have learned some things. I think this may actually be a bug in bad03ef. The wheel caching behavior simply acted as a canary. In particular that commit updated the Link class's handling of the hashes received from the simple API. It added Link.from_json() which is called by parse_links() to set the valid hashes on the Link object.

But the same commit updated Link.hash and Link.hash_name to only consult self.link_hash which is not set by Link.from_json(). I'm not familiar enough with pip to say one way or another, but is it possible that this means the files downloaded from pypi are no longer having their index listed hashes double checked locally to ensure there aren't any bit flips (or worse)?

The reason this affects the wheel cache behavior is that the wheel caching mechanism hashes Link object information to determine where to read and write the cached wheels. In 22.2.2 this included hash information for the packages but in 22.3 it does not.

22.2.2 wheel cache path hash input:

{"interpreter_name":"cp","interpreter_version":"310","sha256":"043a79146eb2907edf439899f262b3dfe41717d34124298ed281139a8b93ca32","url":"https://files.pythonhosted.org/packages/a6/91/86a6eac449ddfae239e93ffc1918cf33fd9bab35c04d1e963b311e347a73/netifaces-0.11.0.tar.gz"}

22.3.0 wheel cache path hash input:

{"interpreter_name":"cp","interpreter_version":"310","url":"https://files.pythonhosted.org/packages/a6/91/86a6eac449ddfae239e93ffc1918cf33fd9bab35c04d1e963b311e347a73/netifaces-0.11.0.tar.gz"}

This patch against 22.3 seems to fix the wheel cache issues we have with the wheel cache, but I also think this ensures Link objects created by parse_links() using the simple json api have a valid link_hash set which I believe may be important. If the hash information from the simple json API should be set on the Link objects some other way then it might be a good idea to clean up Link.hashes to reduce confusion and update parse_links() to set the values correctly.

diff --git a/src/pip/_internal/models/link.py b/src/pip/_internal/models/link.py
index c792d128b..03cc34b42 100644
--- a/src/pip/_internal/models/link.py
+++ b/src/pip/_internal/models/link.py
@@ -248,6 +248,11 @@ class Link(KeyBasedCompareMixin):
         yanked_reason = file_data.get("yanked")
         dist_info_metadata = file_data.get("dist-info-metadata")
         hashes = file_data.get("hashes", {})
+        link_hash = None
+        for hash_name, hash_value in hashes.items():
+            if hash_name in _SUPPORTED_HASHES:
+                link_hash = LinkHash(name=hash_name, value=hash_value)
+                break
 
         # The Link.yanked_reason expects an empty string instead of a boolean.
         if yanked_reason and not isinstance(yanked_reason, str):
@@ -262,6 +267,7 @@ class Link(KeyBasedCompareMixin):
             requires_python=pyrequire,
             yanked_reason=yanked_reason,
             hashes=hashes,
+            link_hash=link_hash,
             dist_info_metadata=dist_info_metadata,
         )
 

Finally, while I agree that our image build process can probably be made more robust, I think we shouldn't fixate too strongly on that specific use case here. I will work on improving our image builds. But at the same time I also expect that if I build perfectly valid wheels with 22.2.2 then upgrade to 22.3 that pip will continue to effectively use its wheel cache and check the hashes of its downloads.

[sbidoul]

@cboylan I just finished bissecting and found the same commit. I've no time right now to investigate your proposed fix but there is definitely some bug to fix here, and your explanation makes a lot of sense at first sight.

[pradyunsg]

FWIW, I'll note that in #11143 -- the fact that it requires changing the cache format and the lack of a migration path for the same, was treated as a blocking concern.

if cache layout/format changes for pip are something which should be mentioned in release notes

I don't think we intentionally changed the cache format here. It's probably reasonable to treat this as a bug/regression IMO. :)

[pradyunsg]

A PR fixing this, with a test to validate the caching key would be welcome!

[cboylan]

I've managed to convert our Docker image builds over to using pip wheel --wheel-dir /some/path and pip install --find-links /some/path and it seems to be mostly working. One behavior change is that when we pip install it seems to only install the wheels it needed to build locally out of the --find-links path. Wheels that are available on PyPI are redownloaded. When we used --cache-dir instead everything was populated out of there.

This behavior does seem to be intentional as --no-index notes Ignore package index (only looking at --find-links URLs instead). I haven't tested that yet but suspect it will restore the previous behavior of using local wheels only. Making note of this here should the google indexes land anyone else with similar needs on this issue.

[potiuk]

@cboylan - just a comment on rhe use case of yours. We are using a different (and IMHO better) approach in multi-staging image of Airflow. Rather than relying on building and copying either wheels or cache between the stages, we are using --user§ flag in the build stage and then we copy the whole resulting .local directory between stages.

This has the advantage that it is guaranteed to work and it has smallest possible overhead for copying the files - it only copies what is needed and size of the final image is as small as possible.

The problem with building the .whl files in one segment and installing them in the next, is that they significantly increase the size of the final image anyway. I am not sure how your Dockerfile looks like, but If you copy the .whl to the final image in order to install them, the .whl files will remain in the layer of the docker image of yours and increase final size of the image even if you delete them in the next step. I am not aware if there is a better way of doing it (short of copying the whl files somewhere remotely and using them from there - or maybe some clever way of using buildkit and cache mounts) to avoid this overhead. The problem is that once you COPY something (.whl files) to the final image, docker creates a layer for those files and even if you delete them, the layer will remain. There is no way to COPY and RUN (and then delete) in the same step.

I recommend to take a look at your image with https://github.com/wagoodman/dive - fantastic tool to see whether your image is optimised.

This is the relevant line of the Airflow's Dockerfile: https://github.com/apache/airflow/blob/main/Dockerfile#L1337:

COPY --from=airflow-build-image --chown=airflow:0
"${AIRFLOW_USER_HOME_DIR}/.local" "${AIRFLOW_USER_HOME_DIR}/.local"

If you want tou can also use PYTHONUSERBASE variable if you want to create the "user" installation in different place than ~/.local .

In the example above "airflow-build-image" is the build segment and AIRFLOW_USER_HOME_DIR is the home dir of the Airflow user.

The "airflow-build-image' segment uses a bit more complex bash scripts (because it handles multiple ways of installing airlfow) but it boils down to (with AIRFLOW_USER_HOME_DIR home directory of the user):

pip install --user .....

This aproach not only makes the final image a lot smaller (no .whl files) but is also way faster (no need to install the .whl files again). And IMHO has virtually no drawbacks.

More about --user installs : https://pip.pypa.io/en/latest/user_guide/#user-installs

[cboylan]

@cboylan - just a comment on rhe use case of yours. We are using a different (and IMHO better) approach in multi-staging image of Airflow. Rather than relying on building and copying either wheels or cache between the stages, we are using --user§ flag in the build stage and then we copy the whole resulting .local directory between stages.

This has the advantage that it is guaranteed to work and it has smallest possible overhead for copying the files - it only copies what is needed and size of the final image is as small as possible.

Yup, I know that was some of the intent around the system we currently have. I'm not sure why it was built to copy wheels across and do another install there since as you mention that increases the image by the size of the wheels themselves. I'll have to investigate if we can shift to this method of using user installed packages/a virtualenv to reduce the image sizes further. At the very least, I think child images will need to update their ENTRYPOIN/CMD directives. Thank you for calling this out, I suspect that this would be an improvement for us.

[potiuk]

I suspect that this would be an improvement for us.

Would be great to hear back if you manage to test/try it and see the savings you could achieve.