New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Merged by Bors] - feat: caching build files on Azure #1230
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should also upload some kind of manifest file for every upload, containing at least the list of referenced cache files and the git revision of mathlib4, so that we can do garbage collection.
Can we do garbage collection by detecting files that were created a long time ago? I believe anything older than 7 days can be discarded safely (it's easy to replace it anyways) |
Co-authored-by: Gabriel Ebner <gebner@gebner.org>
…es; also cache IR files
Co-authored-by: Gabriel Ebner <gebner@gebner.org>
There's still a couple TODOs here, but I think it's best to merge this now and iterate later. Please add copyright headers (just copy&paste from the Mathlib files). bors d+ |
✌️ arthurpaulino can now approve this pull request. To approve and merge a pull request, simply reply with |
bors r+ |
This PR implements caching for Mathlib build files. The basic API is: * `lake exe cache get` to download missing build files * `lake exe cache put` to upload files that are missing on the server * `lake exe cache` to print the help menu This caching method has a few enhancements on top of the solution we had for mathlib in Lean 3: 1. Source files are content-addressed and build files are referenced by such hashes separately 2. Minimized download traffic with queries that only pull files that are missing locally 3. Minimized upload traffic with queries that only push files that are missing on the server
Pull request successfully merged into master. Build succeeded:
|
This PR implements caching for Mathlib build files. The basic API is:
lake exe cache get
to download missing build fileslake exe cache put
to upload files that are missing on the serverlake exe cache
to print the help menuThis caching method has a few enhancements on top of the solution we had for mathlib in Lean 3: