cmd/go: remote build cache #42785
Open
cmd/go: remote build cache #42785
Labels
Milestone
Comments
Some prior art: Bazel supports remote caching over both HTTP and gRPC based protocols. Some of the folks that worked on Bazel RBE prototyped this in the Implementing remote caching and execution adds a lot of complexity to the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What version of Go are you using (
go version
)?What are you suggesting?
Go's build cache (GOCACHE) provides a hash-driven mechanism for storing intermediate compilation results. This hash takes into consideration a number of different variables (including Go versions, architecture, flags, paths, etc). As such, it should be possible to share the cache across different concurrent build jobs (both for developers and CI jobs).
There are a lot of different ways to share thousands of small files across machines, but for most people this will involve NFS (AWS EFS, GCP Filestore, etc). NFS is suboptimal for this use case (provisioning IO throughput across many machines is problematic, builds should not be punished for NFS outages, file interactions need to be atomic, metadata (stat) lookups are rarely cached, etc, etc).
Instead, this is a proposal to support URLs in GOCACHE. This would allow CI servers to have a simple, central HTTP server which would accept GETs for cache lookups and PUT/POST for cache writes. For distributed systems, local proxies off the central HTTP server would provide an additional speedup.
Why would it be valuable?
As anyone who has had to interact with NFS in their career would agree, this is a much simpler configuration than trying to mount a shared filesystem.
This would also lead to the possibility of using memcache/etc URLs in the future.
What are you looking for?
What is the process to officially propose this idea and have it accepted? If it was accepted, I would like some design constraints. I could take it from there and put together a prototype.
The text was updated successfully, but these errors were encountered: