Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Reading from cache archive failed while upgrading pytest #1571

Closed
harshil21 opened this issue Feb 17, 2024 · 6 comments · Fixed by #1707
Closed

BUG: Reading from cache archive failed while upgrading pytest #1571

harshil21 opened this issue Feb 17, 2024 · 6 comments · Fixed by #1707
Labels
bug Something isn't working cache Caching of packages and metadata

Comments

@harshil21
Copy link

Hi! Thanks for making uv! I just upgraded to uv 0.1.3 from 0.1.2 and wanted to upgrade pytest (from 8.0.0 to 8.0.1), so I ran uv pip install -U pytest, however that resulted in an error:

error: Reading from cache archive failed: check bytes error: check failed for tuple struct member 0: check failed for slice index 0: check failed for struct member files: check failed for struct member source_dists: check failed for slice index 0: check failed for struct member file: check failed for struct member requires_python: invalid tag for enum: 108

Here's the full error with --verbose:

Log
(.venv) ➜  python-telegram-bot git:(dependabot/pip/pytest-8.0.0) uv pip install -U pytest --verbose
 uv::requirements::from_source source=pytest
    0.002668s DEBUG uv_interpreter::virtual_env Found a virtualenv through VIRTUAL_ENV at: /home/hoppingturtles/Documents/pyprojects/python-telegram-bot/.venv
    0.002763s DEBUG uv_interpreter::interpreter Using cached markers for: /home/hoppingturtles/Documents/pyprojects/python-telegram-bot/.venv/bin/python
    0.002772s DEBUG uv::commands::pip_install Using Python 3.12.2 environment at /home/hoppingturtles/Documents/pyprojects/python-telegram-bot/.venv/bin/python
 uv_client::flat_index::from_entries 
 uv_resolver::resolver::solve 
      0.005313s   0ms DEBUG uv_resolver::resolver Solving with target Python version 3.12.2
   uv_resolver::resolver::choose_version package=root
   uv_resolver::resolver::get_dependencies package=root, version=0a0.dev0
        0.005372s   0ms DEBUG uv_resolver::resolver Adding direct dependency: pytest*
   uv_resolver::resolver::choose_version package=pytest
     uv_resolver::resolver::package_wait package_name=pytest
 uv_resolver::resolver::process_request request=Versions pytest
   uv_client::registry_client::simple_api package=pytest
     uv_client::cached_client::get_cacheable 
       uv_client::cached_client::read_and_parse_cache file=/home/hoppingturtles/.cache/uv/simple-v0/pypi/pytest.rkyv
 uv_resolver::resolver::process_request request=Prefetch pytest *
          0.006056s   0ms DEBUG uv_client::cached_client Found fresh response for: https://pypi.org/simple/pytest/
 uv_resolver::version_map::from_metadata 
   uv_distribution::distribution_database::get_or_build_wheel_metadata dist=pytest==8.0.1
     uv_client::registry_client::wheel_metadata built_dist=pytest==8.0.1
       uv_client::cached_client::get_serde 
         uv_client::cached_client::get_cacheable 
           uv_client::cached_client::read_and_parse_cache file=/home/hoppingturtles/.cache/uv/wheels-v0/pypi/pytest/pytest-8.0.1-py3-none-any.msgpack
        0.006598s   1ms DEBUG uv_resolver::resolver Searching for a compatible version of pytest (*)
        0.006609s   1ms DEBUG uv_resolver::resolver Selecting: pytest==8.0.1 (pytest-8.0.1-py3-none-any.whl)
   uv_resolver::resolver::get_dependencies package=pytest, version=8.0.1
     uv_resolver::resolver::distributions_wait package_id=pytest-8.0.1
              0.006748s   0ms DEBUG uv_client::cached_client Found fresh response for: https://files.pythonhosted.org/packages/2e/28/30125a808a2448d72fdba26d01ef2bec76a3c860c8694b636e6104e38713/pytest-8.0.1-py3-none-any.whl
        0.006828s   0ms DEBUG uv_resolver::resolver Adding transitive dependency: iniconfig*
        0.006840s   0ms DEBUG uv_resolver::resolver Adding transitive dependency: pluggy>=1.3.0, <2.0
        0.006848s   0ms DEBUG uv_resolver::resolver Adding transitive dependency: packaging*
   uv_resolver::resolver::choose_version package=iniconfig
     uv_resolver::resolver::package_wait package_name=iniconfig
 uv_resolver::resolver::process_request request=Versions iniconfig
   uv_client::registry_client::simple_api package=iniconfig
     uv_client::cached_client::get_cacheable 
       uv_client::cached_client::read_and_parse_cache file=/home/hoppingturtles/.cache/uv/simple-v0/pypi/iniconfig.rkyv
 uv_resolver::resolver::process_request request=Versions pluggy
   uv_client::registry_client::simple_api package=pluggy
     uv_client::cached_client::get_cacheable 
       uv_client::cached_client::read_and_parse_cache file=/home/hoppingturtles/.cache/uv/simple-v0/pypi/pluggy.rkyv
 uv_resolver::resolver::process_request request=Versions packaging
   uv_client::registry_client::simple_api package=packaging
     uv_client::cached_client::get_cacheable 
       uv_client::cached_client::read_and_parse_cache file=/home/hoppingturtles/.cache/uv/simple-v0/pypi/packaging.rkyv
 uv_resolver::resolver::process_request request=Prefetch packaging *
 uv_resolver::resolver::process_request request=Prefetch pluggy >=1.3.0, <2.0
 uv_resolver::resolver::process_request request=Prefetch iniconfig *
          0.007354s   0ms DEBUG uv_client::cached_client Found stale response for: https://pypi.org/simple/pluggy/
          0.007370s   0ms DEBUG uv_client::cached_client Sending revalidation request for: https://pypi.org/simple/pluggy/
       uv_client::cached_client::revalidation_request url="https://pypi.org/simple/pluggy/"
          0.007448s   0ms DEBUG uv_client::cached_client Found fresh response for: https://pypi.org/simple/iniconfig/
error: Reading from cache archive failed: check bytes error: check failed for tuple struct member 0: check failed for slice index 0: check failed for struct member files: check failed for struct member source_dists: check failed for slice index 0: check failed for struct member file: check failed for struct member requires_python: invalid tag for enum: 108

Other packages like typing-extensions seem to upgrade just fine, only pytest is causing this issue. Also reproducible on uv 0.1.2.

uv version: 0.1.3
Build: Ubuntu 23.10, Linux kernel 6.5.0 x86_64
Python: 3.12.2
Rust: 1.76.0

@charliermarsh
Copy link
Member

Will look into this (\cc @BurntSushi), but in the meantime, you can do uv clean (or even uv clean pytest).

@charliermarsh charliermarsh added the bug Something isn't working label Feb 17, 2024
@charliermarsh
Copy link
Member

(Thank you for the clear issue!)

@harshil21
Copy link
Author

Will look into this (\cc @BurntSushi), but in the meantime, you can do uv clean (or even uv clean pytest).

Thanks for the crazy fast response! This did indeed work for me, thanks!

@AlexWaygood
Copy link
Member

(I just opened #1609, which is possibly a duplicate of this issue)

@charliermarsh
Copy link
Member

Cross-posting: I believe I've confirmed locally that this was caused by #1556. I changed one of the cached structs, and that breaks the zero-copy representation. I should've bumped the cache version.

@charliermarsh
Copy link
Member

It might be worth adding test coverage for these kinds of bumps, or some other form of testing and enforcement. (\cc @BurntSushi just for visibility.)

BurntSushi added a commit that referenced this issue Feb 19, 2024
This PR introduces more robust cache healing when `uv` fails to
deserialize an existing cache entry.

("Cache healing" in this context means that if `uv` fails to
deserialize a cache entry, then it will automatically invalidate that
entry and re-generate the data. Typically by sending an HTTP request.)

Previous to some optimizations I made around deserialization, we were
already doing this. After those optimizations, deserializing a cache
policy and the payload were split into two steps. While deserializing
a cache policy retained its cache healing behavior, deserializing the
payload did not. This became an issue when #1556 landed, which changed
one of our `rkyv` data types. This in turn made our internal types
incompatible with existing cache entries. One could work-around this
by clearing `uv`'s cache with `uv clean`, but we should just do it
automatically on a cache entry by entry basis.

This does technically introduce a new cost by pessimistically cloning
the HTTP request so that we can re-send it if necessary (see the commit
messages for the knot pushing me toward this approach). So I re-ran my
favorite ad-hoc benchmark:

```
$ hyperfine -w10 --runs 50 "uv-main pip compile --cache-dir ~/astral/tmp/cache-main ~/astral/tmp/reqs/home-assistant-reduced.in -o /dev/null" "uv-test pip compile --cache-dir ~/astral/tmp/cache-test ~/astral/tmp/reqs/home-assistant-reduced.in -o /dev/null" ; A bart
Benchmark 1: uv-main pip compile --cache-dir ~/astral/tmp/cache-main ~/astral/tmp/reqs/home-assistant-reduced.in -o /dev/null
  Time (mean ± σ):     114.4 ms ±   3.2 ms    [User: 149.4 ms, System: 221.5 ms]
  Range (min … max):   106.7 ms … 122.0 ms    50 runs

Benchmark 2: uv-test pip compile --cache-dir ~/astral/tmp/cache-test ~/astral/tmp/reqs/home-assistant-reduced.in -o /dev/null
  Time (mean ± σ):     114.0 ms ±   3.0 ms    [User: 146.0 ms, System: 223.3 ms]
  Range (min … max):   105.3 ms … 121.4 ms    50 runs

Summary
  uv-test pip compile --cache-dir ~/astral/tmp/cache-test ~/astral/tmp/reqs/home-assistant-reduced.in -o /dev/null ran
    1.00 ± 0.04 times faster than uv-main pip compile --cache-dir ~/astral/tmp/cache-main ~/astral/tmp/reqs/home-assistant-reduced.in -o /dev/null
```

Which is about what I expected.

We should endeavor to have a better testing strategy for these kinds of
bugs, but I think it might be a little tricky to do. I created
#1699 to track that.

Fixes #1571
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working cache Caching of packages and metadata
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants