Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: option to disable caching for specific packages #59683

Open
paralin opened this issue Apr 17, 2023 · 7 comments
Open

proposal: cmd/go: option to disable caching for specific packages #59683

paralin opened this issue Apr 17, 2023 · 7 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. GoCommand cmd/go Proposal ToolSpeed
Milestone

Comments

@paralin
Copy link
Contributor

paralin commented Apr 17, 2023

When embedding large amounts of data with embed, the data is then stored in the Go build cache.

This can cause the Go build cache to grow quite rapidly in size.

This proposal is to add a variable similar to GOPRIVATE like GONOCACHE which would disable the Go build cache for a specific list of Go packages (or package patterns).

We want to cache compiling most packages, but some, which carry large "embed" data, should not be cached.

@gopherbot gopherbot added this to the Proposal milestone Apr 17, 2023
@seankhliao seankhliao changed the title proposal: build cache: disable for specific packages proposal: cmd/go: option to disable caching for specific packages Apr 17, 2023
@ianlancetaylor ianlancetaylor added the GoCommand cmd/go label Apr 18, 2023
@ianlancetaylor
Copy link
Contributor

CC @bcmills @matloob

@ianlancetaylor
Copy link
Contributor

I think that might be a difficult change. We started requiring the build cache in 1.12; see https://go.dev/doc/go1.11#gocache. But I'm not entirely sure.

@mvdan
Copy link
Member

mvdan commented Apr 18, 2023

@paralin just how big are your embedded files? I wonder, if they're too big for your build cache, aren't they too big for your binaries too? Or is it because you end up with multiple copies in the build cache due to different builds?

Perhaps related: the test cache already uses file attributes like modified time to avoid using the entire file contents in case files are large; see #58571 (comment). I assume this case is slightly different, because an embedded file is not just part of the cache key hash, it's part of the resulting compiled package. But we could still consider storing an absolute path to the original embedded file if the file is too large, and considering the cache entry invalid if said path no longer exists.

An alternative approach would be for Go to refuse storing build cache entries beyond a certain size. Then those packages would always be cache misses. In anycase, I would dislike having another knob like GONOCACHE.

@paralin
Copy link
Contributor Author

paralin commented Apr 18, 2023

@mvdan as part of a release process I'm embedding a large file into the binary to pack assets.

The release binary is of acceptable size (~200Mb) for downloading and running natively. But the file changes between builds, and the copies end up in the cache.

But we could still consider storing an absolute path to the original embedded file if the file is too large, and considering the cache entry invalid if said path no longer exists.

That seems like it would work to me.

I don't mind not having a knob to control this if it made the correct choice automatically.

@bcmills
Copy link
Contributor

bcmills commented Apr 18, 2023

When embedding large amounts of data with embed, the data is then stored in the Go build cache.

I am surprised that embed data is stored in the cache at all. I would expect it not to be, since the contents of any given file to be embedded should not vary based on the build configuration: we shouldn't be linking in the embed data until we link the final program, and at that point we should be able to link it directly from the source tree (bypassing the build cache entirely).

I would much rather we do that than add a cmd/go configuration knob.

I wonder if this should be considered as linker deficiency rather than a cmd/go proposal.

@bcmills bcmills added ToolSpeed compiler/runtime Issues related to the Go compiler and/or runtime. labels Apr 18, 2023
@ianlancetaylor
Copy link
Contributor

Just a note that if we don't store the content of the embedded file in the build cache, the build hash gets trickier because it depends on files that aren't in the build cache. This seems like a solvable problem but it does seem to break one of our current invariants.

@bcmills
Copy link
Contributor

bcmills commented Apr 18, 2023

The build hash already depends on files that aren't in the build cache: namely, the .go source files themselves. (It probably does change an invariant of the link action in particular, but not of the build cache in general.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. GoCommand cmd/go Proposal ToolSpeed
Projects
Status: No status
Status: Incoming
Development

No branches or pull requests

5 participants