Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: remote build cache #42785

idcmp opened this issue Nov 23, 2020 · 6 comments

cmd/go: remote build cache #42785

idcmp opened this issue Nov 23, 2020 · 6 comments
GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.


Copy link

idcmp commented Nov 23, 2020

What version of Go are you using (go version)?

$ go version
go version go1.15.3 darwin/amd64

What are you suggesting?

Go's build cache (GOCACHE) provides a hash-driven mechanism for storing intermediate compilation results. This hash takes into consideration a number of different variables (including Go versions, architecture, flags, paths, etc). As such, it should be possible to share the cache across different concurrent build jobs (both for developers and CI jobs).

There are a lot of different ways to share thousands of small files across machines, but for most people this will involve NFS (AWS EFS, GCP Filestore, etc). NFS is suboptimal for this use case (provisioning IO throughput across many machines is problematic, builds should not be punished for NFS outages, file interactions need to be atomic, metadata (stat) lookups are rarely cached, etc, etc).

Instead, this is a proposal to support URLs in GOCACHE. This would allow CI servers to have a simple, central HTTP server which would accept GETs for cache lookups and PUT/POST for cache writes. For distributed systems, local proxies off the central HTTP server would provide an additional speedup.

Why would it be valuable?

As anyone who has had to interact with NFS in their career would agree, this is a much simpler configuration than trying to mount a shared filesystem.

This would also lead to the possibility of using memcache/etc URLs in the future.

What are you looking for?

What is the process to officially propose this idea and have it accepted? If it was accepted, I would like some design constraints. I could take it from there and put together a prototype.

@cagedmantis cagedmantis changed the title Support URLs in GOCACHE cmd/go: support URLs in GOCACHE Dec 7, 2020
@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Dec 7, 2020
@cagedmantis cagedmantis added this to the Backlog milestone Dec 7, 2020
Copy link

/cc @bcmills @jayconrod @matloob

Copy link

Some prior art: Bazel supports remote caching over both HTTP and gRPC based protocols.

Some of the folks that worked on Bazel RBE prototyped this in the go command last year. We found that this had the potential to speed up long-running tests, since test results can be cached remotely. However, it's detrimental for builds. Most actions performed by the go command are very fast, and the overhead of sending lots of small files over the network tends to dominate any gains we might see.

Implementing remote caching and execution adds a lot of complexity to the go command. Since it would benefit a fairly small number of users, we chose not to go forward with it at that time.

@jayconrod jayconrod modified the milestones: Backlog, Unplanned Dec 7, 2020
@jayconrod jayconrod changed the title cmd/go: support URLs in GOCACHE cmd/go: remote build cache Dec 7, 2020
Copy link

I'd love to get the speed up for caching tests since were we spend alot of time. Is there a way to identify test cache results so we could slurp them up GOCACHE and then lay them back down before the next clean build? (maybe storing/retrieviing at git commit).

Having a seperate GO_TEST_CACHE might also make this pretty feasible.

Copy link

@jayconrod - is there a public reference to the design and the attempt done by the RBE people?

I wonder if we can still utilize remote cache for CI builds done on ephemeral machines. For organizations working in monorepo and cloud-based CI this could be a real game-changer (given that the org does not want to adopt Bazel).

Is there a way to plugin to go cache so we can replace its implementation with some custom code? That would allow us to offer remote caching implementation while keeping the code of go slim and lean.

Copy link

rbalint commented Mar 15, 2023

I'd also be interested in plugging into Go's caching. implements a generic process cache working with a many compilers and reproducible commands, but not Go, because Go's build insists on using its internal caching. is a local cache that can be fine-tuned to not cache quick operations or filter by command.

Copy link

See the proposal at #59719, which pulls most of the work out of cmd/go.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
None yet

No branches or pull requests

8 participants