-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Locally Cache Resolved Packages #1892
Comments
I assume each cache file represents a (package + tags), maybe named with a scheme similar to PEP 427? If that is the case, each cache file would contain direct dependency information of that (package + tags) combination. |
+1 from me. I don't think the cache will make any current issues with indeterminism worse, and in some ways is likely to mitigate them - rerunning a lock on the same machine when there haven't been any new releases will always give the same answer, for example, even if some of the setup.py files involved are non-deterministic. |
What are the tradeoffs when distributing the cache across multiple files, is there anywhere in particular we should look for inspiration on the performance front? Is there an ideal format? |
The core trade-off I'm aware is that using multiple files makes it much easier to avoid conflicts when multiple processes are writing to the cache at the same time, but also uses up more inodes on the filesystem (which is typically only a potential problem on actual file servers, not development workstations). There are also differences in how the two models interact with filesystem caching in general, as well as what happens when the cache is actually stored on a network drive. Unfortunately, I've never had much luck looking for actual guides on this, since my search results always get swamped by links related to regular application caching. However, one use-case-specific consideration here is that it would be nice if the |
Caching more things would definitely be helpful (and I like the idea!) 👍 I had a couple thoughts: A couple points in favor of multiple files:
(but you could still go pull the cache together into one big file after the fact :) ) In terms of the lockfile, it'd be neat to be able to bias towards keeping packages as similar as possible.
Do we know which things really aren't going to change? For example, you can upload a new build with a different build number (for whatever reason, tho I assume it's in the case of a bad compilation or a bad setup.py), so it's not necessarily the case that asking for the same dependency range resolves to the same set of options. Perhaps it'd be better to key on URL-with-hash and still require at least one listing of PyPI to figure out whether package lists have changed when attempting to lock again. In the same vein, I'm not sure you can cache all resolved versions of a package, because you'd still need to hit PyPI to find out if new packages are available.
I tried reusing pip's caches in another PR and the internal API is simple, but the actual file layout is somewhat opaque (it looks like it's just the sha256 split into a couple pieces to reduce number of files in a specific directory). Might be nice if the final component of the directory tree included some human-readable name too. |
The key mapping that we want to cache is "Which dependencies does version X of project Y declare?", since that isn't a simple API query for Python packages - it requires downloading artifacts and looking inside them. However, the answer to that question is going to remain stable once calculated for a published artifact, since PyPI itself prohibits changing previously published artifacts. (Technically other index servers can allow artifact replacement, but I'm OK with viewing "We replaced a non-development artifact in our internal index server and changed its dependency declarations without bumping the version number" as a self-inflicted problem that folks will need to resolve by purging any affected caches). Caching dependency resolution is much harder (beyond what the lock file does), since that's an inherently time dependent question - new feature and maintenance releases may be published at any time. |
Right, I don't think there is any way to cache all versions of anything, or even to know that we have cached the current version. The only thing I want to cache is that we have resolved |
We don't want caching proxies like devpi/Artifactory/Nexus/Pulp et al to make |
The years is 2022 and |
As part of an effort to speed up the locking process, I think we can make some incremental gains by storing a local cache of the dependency graph of each package we resolve for each abi tag + python version.
Essentially I would suggest:
['name', 'resolved_version', 'specifiers', 'markers']
This saves us having to continuously hit pypi for information about things that are not going to change, and allows us to re-resolve and re-lock without re-calculating each setup file over and over again which is incredibly slow. While this may come at the risk of issues of indeterminism in
setup.py
execution, I think the risk is minimal and the gains are vast.Reference issues: #1785 #1866
/cc @ncoghlan @uranusjr @jtratner
The text was updated successfully, but these errors were encountered: