cmd/go: compress GOCACHE

[jakebailey]

No template fit, forgive my use of the empty issue template.

In the typescript-go project, I and others on the team keep accidentally running ourselves out of disk space (or, running CI runners out of disk space) due to how large GOCACHE seems to be for our codebase.

If you just do:

$ export GOCACHE="$(pwd)/gocache"
$ git clone --depth=1 https://github.com/microsoft/typescript-go.git
$ cd typescript-go
$ go test ./... > /dev/null

Then the cache is 3.2GB:

$ du -sh $GOCACHE
3.2G    /home/jabaile/work/gocache-example/gocache

The top 10 largest files in the cache are:

185M    27/27480acf832d161dc0faad35f12db3d7bac9e21dfcc22a8b891c393fc3c39684-d
61M     14/143367227694be40742d296d489b5de454f68b9ddeccd99c1351e93d22da7877-d
58M     dc/dcbc45767bc9ae462401888cae7f79f4e5e62aceee572a3da131fa4574cde732-d
50M     e6/e6d125f1833dca10b62878d07b24e8ad1e9a57c86299ac892642f4cad3d2414d-d
50M     96/962cdf6c83e6a527324d1bb74a8a9c0bb814fc61527be153dd47277221e3ed53-d
47M     ec/ecbd61c1400299ccfaa9dc901bfd4b491727b04352ce38dc74ab78b2c6f802a3-d
47M     eb/eb7eeb23cb39432385ead152ad5a74d46f8ba42e2b39fd7c37318d8e6a44943d-d
47M     e8/e8eac35ad09dc74b9e5ef05ac9c5fe1984059ff4fabe19188a0d6078ea211e09-d
46M     60/608d4671f8cdaec8703838eb21369a2a191ded0611a4c080f06ec62b1f6dc4d6-d
45M     3b/3b628f4a9c11f4884f33bd416585ab50ba7e213427e5d609e3d584bac3629d00-d

Out of curiosity, I ran these through zstd to see how compressible they are, and indeed:

185M    15M   27/27480acf832d161dc0faad35f12db3d7bac9e21dfcc22a8b891c393fc3c39684-d
61M     7.8M  14/143367227694be40742d296d489b5de454f68b9ddeccd99c1351e93d22da7877-d
58M     5.4M  dc/dcbc45767bc9ae462401888cae7f79f4e5e62aceee572a3da131fa4574cde732-d
50M     5.2M  e6/e6d125f1833dca10b62878d07b24e8ad1e9a57c86299ac892642f4cad3d2414d-d
50M     5.2M  96/962cdf6c83e6a527324d1bb74a8a9c0bb814fc61527be153dd47277221e3ed53-d
47M     4.3M  ec/ecbd61c1400299ccfaa9dc901bfd4b491727b04352ce38dc74ab78b2c6f802a3-d
47M     4.3M  eb/eb7eeb23cb39432385ead152ad5a74d46f8ba42e2b39fd7c37318d8e6a44943d-d
47M     4.3M  e8/e8eac35ad09dc74b9e5ef05ac9c5fe1984059ff4fabe19188a0d6078ea211e09-d
46M     4.2M  60/608d4671f8cdaec8703838eb21369a2a191ded0611a4c080f06ec62b1f6dc4d6-d
45M     3.8M  3b/3b628f4a9c11f4884f33bd416585ab50ba7e213427e5d609e3d584bac3629d00-d

These files become often 10x smaller. And in fact, if I .tar.zst the entire GOCACHE, the 3.2GB is just 276MB when compressed! (Looking at my GHA logs, we have an 11GB GOCACHE that is compressed to about 1GB, so that savings seems to stick.)

Although this doesn't itself fix the problem of "GOCACHE big and not cleaned often", it does seem like making it 10x smaller could help make it happen in 10x more time.

I tried looking into making a GOCACHEPROG implementation to do this, but I could not figure a way that this could work given the GOCACHEPROG protocol expects Get to have placed the original file on disk. So, my expectation is that the Go toolchain has to be the one to compress the data itself.

There's no support for compressing with zstd quite yet (#62513); I tried using gz and it was roughly 20x slower to compress than zstd, which doesn't bode well for fast compilation (but, I did not change the compress options).

Related:

• cmd/go: clean GOCACHE based on disk usage #29561 (cmd/go: clean GOCACHE based on disk usage #29561 (comment))
• cmd/compile: long symbol names for instantiated generics => large object files (though not executables) #50438
• cmd/compile: very long names for simple generic usage #71535
• cmd/compile: objects emit generic instantiations for imported packages #56718

[gabyhelp]

Related Issues

• cmd/go: clean GOCACHE based on disk usage #29561
• cmd/go: auto cache clean fail #31931 (closed)
• cmd/go: size of go-build directory increasing rapidly #68872 (closed)
• cmd/distpack: offer Zstandard-compressed archives in addition to gzip #62446
• cmd/go: test reported as cached, but not actually cached #41593
• Go builder caching policies might need another look #32578 (closed)
• cmd/go: add GOEXPERIMENT=cacheprog to let a child process implement the internal action/output cache #59719 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[adonovan]

What are these huge files? I guess they are executables, mostly tests, in which case the value of caching them is pretty low since a single bit change in any dependency causes them never to be needed again. So perhaps the solution is to tweak the cache trimming algorithm to run more often and prune more aggressively for large files. (BTW @ianlancetaylor has been thinking about the cache this week in the context of #76314.)

Given how vast the drives are on modern machines--at least from the perspective of anyone but a video producer/consumer, and certainly that of yours truly who remembers spending his summer wages on a 20MB hard disk--it's hard for me to believe that the solution to this problem is to spend CPU cycles compressing huge files we are probably never going to look at again. (Also, if these are test executables, how can go test execute them when compressed?)

[andrewbranch]

I’m one of Jake's teammates who is frequently plagued by this—totally agree the best solution would be to delete irrelevant cache entries, but as an anecdotal data point to the large hard drives argument, my primary dev machine is a 2023 Mac Mini that came with a 500 GB drive. About a third of that is system data, and a few years of random usage means I'm running with max 70 GB free on a daily basis. That entire 70 GB gets eaten by the Go build cache about once a week on average, and sometimes within a single day, I guess depending on the workflow I’m in (though I haven’t figured out exactly what I’m doing that makes it spike so fast at certain times but not others). Following this with great interest 😄

[mvdan]

I actually think that using Go on smaller dev machines like ultrabooks is fairly common, given how the compiler isn't very CPU demanding. And those machines tend to not have that much memory or disk space. I've never even developed with Go on a desktop machine. So I think your setup is entirely normal :)

[jakebailey]

I'll also note that the standard GHA Linux runners only have 20GB of space free; their default images are something like a 90GB image with 70GB of random software / caches preinstalled. In our workflows, I've had to start including a step that deletes some of this just to make them pass.

What are these huge files? I guess they are executables, mostly tests

They are archive files:

$ file ...
27/27480acf832d161dc0faad35f12db3d7bac9e21dfcc22a8b891c393fc3c39684-d: current ar archive
14/143367227694be40742d296d489b5de454f68b9ddeccd99c1351e93d22da7877-d: current ar archive
dc/dcbc45767bc9ae462401888cae7f79f4e5e62aceee572a3da131fa4574cde732-d: current ar archive
e6/e6d125f1833dca10b62878d07b24e8ad1e9a57c86299ac892642f4cad3d2414d-d: current ar archive
96/962cdf6c83e6a527324d1bb74a8a9c0bb814fc61527be153dd47277221e3ed53-d: current ar archive
ec/ecbd61c1400299ccfaa9dc901bfd4b491727b04352ce38dc74ab78b2c6f802a3-d: current ar archive
eb/eb7eeb23cb39432385ead152ad5a74d46f8ba42e2b39fd7c37318d8e6a44943d-d: current ar archive
e8/e8eac35ad09dc74b9e5ef05ac9c5fe1984059ff4fabe19188a0d6078ea211e09-d: current ar archive
60/608d4671f8cdaec8703838eb21369a2a191ded0611a4c080f06ec62b1f6dc4d6-d: current ar archive
3b/3b628f4a9c11f4884f33bd416585ab50ba7e213427e5d609e3d584bac3629d00-d: current ar archive

it's hard for me to believe that the solution to this problem is to spend CPU cycles compressing huge files we are probably never going to look at again. (Also, if these are test executables, how can go test execute them when compressed?)

On my personal machines, I use btrfs, which auto-compresses everything with zstd; the docs I read about this (https://wiki.archlinux.org/title/Btrfs#Compression for example) say that this is beneficial because on the whole, you save more time by using CPU time to compress and thereby spending less time doing IO. I am not lucky enough to be able to do this for work, where I'm in WSL 😓

So perhaps the solution is to tweak the cache trimming algorithm to run more often and prune more aggressively for large files.

I of course agree with that! I linked the other threads about this as I think they're very relevant. Some mechanism to help would be great from any angle; the compression is just one angle I thought of and tested after reading the linked threads.

[jakebailey]

Just to show what a "real" GOCACHE looks like for me, here are the stats for my machine after doing a bunch of stuff yesterday:

$ du -sh $(go env GOCACHE)
182G    /home/jabaile/.cache/go-build
$ find $GOCACHE -type f -exec du -h {} + | sort -rh | head -n 10 | sed "s|$GOCACHE/||"
185M    f9/f998bb90045aecf19996c7413a3c5a03e136e7da0cc008e758f2fe1b463a21b6-d
185M    ef/efa368191aedccf2712a15806c6fe0325a7b8db99c4a910b46b12d4997e507f9-d
185M    de/de4e29d63f8d2db12444dcb01da9053f646e3f97d00950b3b7176124387a6f3b-d
185M    d0/d039d260c04771374ed8e359ed68c9ff8180f1b52b9a9a10b5fc84783f5ae971-d
185M    c7/c71043689a4408e69cd3ce979ce0af5c29ad013550952b1af44681332340b999-d
185M    c3/c36938da57754ffd91db67d8c7f15f277a4ac3c610895b656a35795058733af5-d
185M    94/9404c069c7dda429d2b7f11b1fbd567aed838eef04fd302a8240033b215dab85-d
185M    8e/8ecc196d9db4e34efbd81c94dd6e8fed41382c7314134c7dd87d3e4bbbe4450e-d
185M    72/7288596298796dc55808f90c015c89c4a15fe25f115d65c9865c7f1bd8dd4793-d
185M    63/63336e82c9ec581648e19e78be5fde45b4020039f0959690d238f8d38c8621df-d
$ time tar -cf gocache.tar.zst --use-compress-program=zstd $GOCACHE
tar -cf gocache.tar.zst --use-compress-program=zstd $GOCACHE  291.59s user 276.85s system 159% cpu 5:56.19 total
$ l gocache.tar.zst                          
-rw-r--r-- 1 jabaile jabaile 16G Nov 18 09:31 gocache.tar.zst

$ strings $GOCACHE/f9/f998bb90045aecf19996c7413a3c5a03e136e7da0cc008e758f2fe1b463a21b6-d | head -n5
!<arch>
__.PKGDEF       0           0     0     644     625566    `
go object linux amd64 go1.25.4 GOAMD64=v1 X:regabiwrappers,regabiargs,aliastypeparams,swissmap,synchashtriemap,dwarf5
build id "n0wg2KLNeYHRrcRA-xOy/zlD-wI1JVRCqlJ5LPF26"
astfmterrorsunsafeinternal/reflectlitereflectliteinternal/goarchgoarchinternal/unsafeheaderunsafeheaderinternal/abiabiruntimeinternal/byteorderbyteorderinternal/cpucpuinternal/runtime/syssysinternal/goosgoosinternal/runtime/atomicatomicinternal/runtime/mathmathinternal/goexperimentgoexperimentinternal/runtime/cgroupcgroupinternal/bytealgbytealginternal/runtime/strconvstrconvinternal/runtime/syscallsyscallinternal/coverage/rtcovrtcovinternal/runtime/gcgcinternal/runtime/mapsmapsinternal/raceraceinternal/asanasaninternal/msanmsaninternal/godebugsgodebugsinternal/profilerecordprofilerecordinternal/stringslitestringsliteinternal/runtime/exithookexithookinternal/chacha8randchacha8randinternal/trace/tracev2tracev2slicescmpitermath/bitsbitsunicode/utf8utf8internal/fmtsortfmtsortreflectsyncsync/atomicinternal/syncinternal/synctestsynctestunicodeinternal/itoaitoaioosinternal/filepathlitefilepathliteio/fsfstimeinternal/godebuggodebuginternal/bisectbisectinternal/oserroroserrorpathinternal/testlogtestloginternal/pollpollinternal/syscall/unixunixinternal/syscall/execenvexecenvstringsgithub.com/microsoft/typescript-go/internal/collectionscollectionsencodinggithub.com/go-json-experiment/jsonjsonbytesgithub.com/go-json-experiment/json/internalinternalgithub.com/go-json-experiment/json/internal/jsonflagsjsonflagsgithub.com/go-json-experiment/json/internal/jsonoptsjsonoptsgithub.com/go-json-experiment/json/jsontextjsontextgithub.com/go-json-experiment/json/internal/jsonwirejsonwireunicode/utf16utf16encoding/base32base32encoding/base64base64encoding/hexhexencoding/binarybinarygithub.com/microsoft/typescript-go/internal/corecoregithub.com/microsoft/typescript-go/internal/tspathtspathpath/filepathfilepathgithub.com/microsoft/typescript-go/internal/stringutilstringutilregexpregexp/syntaxsyntaxsortnet/urlurlnet/netipnetipuniqueweakcontextgolang.org/x/text/languagelanguagegolang.org/x/text/internal/languagegolang.org/x/text/internal/tagtaggolang.org/x/text/internal/language/compactcompactgithub.com/microsoft/typescript-go/internal/debugdebuggithub.com/microsoft/typescript-go/internal/jsonutiljsonutilgolang.org/x/sync/errgrouperrgroupgithub.com/microsoft/typescript-go/internal/diagnosticsdiagnosticsgithub.com/microsoft/typescript-go/internal/ast/home/jabaile/work/TypeScript-go/internal/ast/ast.goKind/home/jabaile/work/TypeScript-go/internal/ast/kind.go/home/jabaile/work/TypeScript-go/internal/ast/kind_stringer_generated.goStringiFlags/home/jabaile/work/TypeScript-go/internal/ast/nodeflags.goNodeFlagsLoc/home/jabaile/work/TypeScript-go/internal/core/text.goposTextPosendPostEndLenIsValidContainsContainsInclusiveContainsExclusiveWithPosWithEndContainedByt2OverlapsIntersectsTextRangestartbuiltinmaxminid$GOROOT/src/sync/atomic/type.go_LockUnlocknoCopyesc:align64vLoadxStorevalSwapnewoldCompareAndSwapswappedAdddeltaAndmaskOrUint64esc:

Honestly, I have no idea how to tell which package this archive file is for 😄

I am willing to bet it's the checker!

[adonovan]

On my personal machines, I use btrfs, which auto-compresses everything with zstd; the docs I read about this (https://wiki.archlinux.org/title/Btrfs#Compression for example) say that this is beneficial because on the whole, you save more time by using CPU time to compress and thereby spending less time doing IO.

That's only going to help in single-threaded scenarios, not builds, where parallelism will turn your writes from an IO-bound task into a CPU-bound one. (Indeed your link says as much.)

Point taken that disks are filling pretty rapidly. We should definitely tweak the cache trimming algorithm. One approach Ian was exploring was fork+execing an asynchronous child process that would be allowed to outlive the parent. I worry that that may create more problems than it solves (e.g. accidental build-up of child processes; or container termination at end of go build causing termination of child processes before they have done much work). An alternative might be some kind of 'pacing' strategy in which go build does do cache trimming synchronously, as today, but does it perhaps more often and in bounded size chunks (e.g. at most one of the 256 cache shards, round robin).

[prattmic]

Honestly, I have no idea how to tell which package this archive file is for 😄

This probably isn't the canonical approach, but you can likely get a good sense from go tool nm, where most of the symbols should be from the target package.

[jakebailey]

Ah, it's probably our AST package (which is what I had assumed originally).

...
github.com/microsoft/typescript-go/internal/ast.(*ArrowFunction).AsJSDocReturnTag
github.com/microsoft/typescript-go/internal/ast.(*ArrowFunction).AsJSDocReturnTag.arginfo1<1>
github.com/microsoft/typescript-go/internal/ast.(*ArrowFunction).AsJSDocReturnTag.argliveinfo<1>
github.com/microsoft/typescript-go/internal/ast.(*ArrowFunction).AsJSDocSatisfiesTag
github.com/microsoft/typescript-go/internal/ast.(*ArrowFunction).AsJSDocSatisfiesTag.arginfo1<1>
github.com/microsoft/typescript-go/internal/ast.(*ArrowFunction).AsJSDocSatisfiesTag.argliveinfo<1>
github.com/microsoft/typescript-go/internal/ast.(*ArrowFunction).AsJSDocSeeTag
github.com/microsoft/typescript-go/internal/ast.(*ArrowFunction).AsJSDocSeeTag.arginfo1<1>
github.com/microsoft/typescript-go/internal/ast.(*ArrowFunction).AsJSDocSeeTag.argliveinfo<1>
...
go.shape.(*f7fff7e5bd2848b7d26d73eae8678c9e84c3132993bd86465d918f60bd1dcea7).Statement
go.shape.(*f7fff7e5bd2848b7d26d73eae8678c9e84c3132993bd86465d918f60bd1dcea7).Statement.arginfo1<1>
go.shape.(*f7fff7e5bd2848b7d26d73eae8678c9e84c3132993bd86465d918f60bd1dcea7).Statement.argliveinfo<1>
go.shape.(*f7fff7e5bd2848b7d26d73eae8678c9e84c3132993bd86465d918f60bd1dcea7).StatementList
go.shape.(*f7fff7e5bd2848b7d26d73eae8678c9e84c3132993bd86465d918f60bd1dcea7).StatementList.arginfo1<1>
go.shape.(*f7fff7e5bd2848b7d26d73eae8678c9e84c3132993bd86465d918f60bd1dcea7).StatementList.argliveinfo<1>
go.shape.(*f7fff7e5bd2848b7d26d73eae8678c9e84c3132993bd86465d918f60bd1dcea7).Statements
go.shape.(*f7fff7e5bd2848b7d26d73eae8678c9e84c3132993bd86465d918f60bd1dcea7).Statements.arginfo1<1>
go.shape.(*f7fff7e5bd2848b7d26d73eae8678c9e84c3132993bd86465d918f60bd1dcea7).Statements.argliveinfo<1>
...

I know this package is a world of hurt for reasons that do include "big giant type names" 😅

(Yay JS for having so many syntactic constructs.)

[ianlancetaylor]

@adonovan

I guess they are executables, mostly tests

Note that we do not currently cache test executables. The only executables we cache are those from go run and go tool. We do not cache executables from go build, go install, or go test.

[ianlancetaylor]

Point taken that disks are filling pretty rapidly. We should definitely tweak the cache trimming algorithm. One approach Ian was exploring was fork+execing an asynchronous child process that would be allowed to outlive the parent. I worry that that may create more problems than it solves (e.g. accidental build-up of child processes; or container termination at end of go build causing termination of child processes before they have done much work). An alternative might be some kind of 'pacing' strategy in which go build does do cache trimming synchronously, as today, but does it perhaps more often and in bounded size chunks (e.g. at most one of the 256 cache shards, round robin).

As you know, the background process (which I agree may not be the best approach) addresses the problem of the go tool hanging upon command completion. It does not address the problem described in this issue, of the cache taking up too much disk space. The same applies to the incremental approach.

My current cache is 37GB, so, ten times the size of the OP. I just don't care, because my laptop has a terabyte disk.

It seems to me that smaller systems are likely older and likely also have less CPU horsepower available for compression. So I don't think compression by default sounds right.

That said, I certainly agree that we should make GOCACHEPROG support compressing the files for those who want it. It would be helpful if somebody could sketch out what changes are required to make that possible.

I think it would be promising if there were a way for new versions of a compiled package to displace older versions of a compiled package. It wouldn't always be the right choice, but my guess is that it would help more than it would hurt. But I don't see a way to support that with the current cache design.

[jakebailey]

My current cache is 37GB, so, ten times the size of the OP. I just don't care, because my laptop has a terabyte disk.

To be clear, the 3GB in my example is just from one compile. My actual cache is sitting at about 200GB at the moment. (#76337 (comment))

That said, I certainly agree that we should make GOCACHEPROG support compressing the files for those who want it. It would be helpful if somebody could sketch out what changes are required to make that possible.

From my looking when I tried to do this, while the Go toolchain does itself understand when it needs something on disk versus just bytes, the protocol does not and just wants everything to exist on disk. It's even stranger becuase the operation is Get() -> file path, which makes it hard for a GOCACHEPROG to know when it's safe to delete the file it just returned, though I suppose multiple GOCACHEPROGs of the same thing could use Close() to conspire to clean up when safe.

[ianlancetaylor]

Apologies for misunderstanding the cache size.

Thanks for the note about cacheprog. It makes sense that the protocol expects disk files, because the go tool is going to pass file names to the compiler and linker.

Still, it seems to me that the current protocol can support compressed files without changing else. The cacheprog should create a temporary working directory used for holding files during the build.

• Put will copy the file into the working directory. It will also compress the file into the permanent cache. That is, there will be two copies of the data, one compressed, one uncompressed.
• Get will:
  • Look for the file in the working directory. If found, return it.
  • Otherwise, look for the file in the cache. If found, uncompress it into the working directory and return it.
  • Otherwise, return a cache miss.
• Close will delete the working directory.

[ianlancetaylor]

We could also change the compiler and linker (and perhaps other tools, I'm not sure) to automatically decompress files. Basically, if the file starts with the zstd magic number 0xfd2fb528 we run it through zstd. This would not be a trivial change as the linker, at least, expects a seekable file. It appears that fully decompressing on file open would be extra work that is not always required.