x/build: gotip-linux-amd64-misccompile builder fails with no space left on device

[cagedmantis]

Trybot runs are failing on gotip-linux-amd64-misccompile with no space left on device errors. A sample log from one of the trybot runs:

[Back to build](https://cr-buildbucket.appspot.com/build/8723117275214065601)
[Raw log](https://logs.chromium.org/logs/golang/buildbucket/cr-buildbucket/8723117275214065601/+/u/step/11/log/2?format=raw) | Switch to full mode | [Login](https://logs.chromium.org/auth/openid/login?r=%2Flogs%2Fgolang%2Fbuildbucket%2Fcr-buildbucket%2F8723117275214065601%2F%2B%2Fu%2Fstep%2F11%2Flog%2F2)
[I2025-02-12T18:15:42.811271Z 403391 0 sink.go:277] SinkServer: warm-up started
[I2025-02-12T18:15:42.811412Z 403391 0 sink.go:350] SinkServer: starting HTTP server...
[I2025-02-12T18:15:42.813500Z 403391 0 sink.go:282] SinkServer: warm-up ended
[I2025-02-12T18:15:42.813987Z 403391 0 cmd_stream.go:492] rdb-stream: starting the test command - ["/home/swarming/.swarming/w/ir/cache/tools/bin/result_adapter" "go" "-v=false" "--" "/home/swarming/.swarming/w/ir/x/w/goroot/bin/go" "tool" "dist" "test" "-json" "-compile-only"]
net: write /home/swarming/.swarming/w/ir/x/t/go-build2504983716/b1853/vet.cfg: no space left on device
# net/netip_test [net/netip.test]
compile: writing output: write $WORK/b1852/_pkg_.a: no space left on device
# net/http.test
/home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_amd64/link: mapping output file failed: no space left on device
net/rpc: write /home/swarming/.swarming/w/ir/x/t/go-build2504983716/b1855/vet.cfg: no space left on device
# net/http_test
# [net/http_test]
vet: failed to export analysis facts: write $WORK/b1799/vet.out: no space left on device
net/smtp_test: write /home/swarming/.swarming/w/ir/x/t/go-build2504983716/b1864/importcfg: no space left on device
net/rpc/jsonrpc: write /home/swarming/.swarming/w/ir/x/t/go-build2504983716/b1860/vet.cfg: no space left on device
net/url: mkdir /home/swarming/.swarming/w/ir/x/t/go-build2504983716/b1868/: no space left on device
os: mkdir /home/swarming/.swarming/w/ir/x/t/go-build2504983716/b1872/: no space left on device
# net/rpc
# [net/rpc]
vet: failed to export analysis facts: write $WORK/b1854/vet.out: no space left on device
...

Issue 71667 was created but found not to be related.

@golang/release @prattmic

[gabyhelp]

Related Issues

• build: build failure on gotip-linux-amd64-longtest-race #66485 (closed)
• build: unrecognized build failure on gotip-linux-amd64-newinliner that was actually due to a core file being left behind #65977 (closed)
• x/build: build and test failures on openbsd-amd64 builders #66474
• build: build failure on gotip-linux-amd64-misccompile #67486 (closed)
• build: build failure on gotip-linux-amd64-misccompile #68016 (closed)
• x/build: frequent dragonfly-amd64 failures with "No space left on device" since 2021-12-07 #50538 (closed)
• build: build failure on gotip-darwin-amd64_12 #65822
• build: build failure on gotip-linux-riscv64 #69091 (closed)
• x/build: linux-arm builder can't finish an all.bash run when test sharding isn't used #40872 (closed)
• build: build failure on x_telemetry-gotip-linux-amd64-misccompile #67480 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[dmitshur]

Thanks for filing an issue to track this.

Oddly, as Michael K. pointed out, it seems this isn't happening in post-submit so far, yet fairly often in pre-submit recently. Compare with:

https://ci.chromium.org/ui/p/golang/builders/ci/gotip-linux-amd64-misccompile?limit=200
https://ci.chromium.org/ui/p/golang/builders/ci/go1.24-linux-amd64-misccompile?limit=200

As a side note, I wonder if somehow watchflakes isn't reporting this at all, or if it's going to some issue that we're not looking at. For example, issue #57902 has a post <- log ~ `no space left on device` pattern yet this hasn't shown up there. I guess we can get most of this information via other means.

[gopherbot]

Change https://go.dev/cl/649075 mentions this issue: DO NOT SUBMIT: cmd/dist: dump disk use data

[prattmic]

I got a disk space dump in https://logs.chromium.org/logs/golang/buildbucket/cr-buildbucket/8723101639890049153/+/u/step/13/log/2

The culprit is wasmtime

275G	/home/swarming/.cache/wasmtime/modules/wasmtime-5fc1252ff139d78b6c27d23d50a54732b0ab09aa
275G	/home/swarming/.cache/wasmtime/modules
275G	/home/swarming/.cache/wasmtime

[prattmic]

wasmtime appears to use XDG_CACHE_DIR if set, via https://crates.io/crates/directories-next#:~:text=/Movies/)-,ProjectDirs,-The%20intended%20use -> https://github.com/bytecodealliance/wasmtime/blob/ec9f885a2ed877deb3ce64050583be0688106f54/crates/cache/src/config.rs#L426.

I propose we set XDG_CACHE_DIR to a temporary directory within the build directory, as we do for TMP and GOCACHE.

[dmitshur]

Oh, so the $HOME/.cache isn't being cleaned by anything? Yeah, using a cache directory in a per-build work directory that is cleaned sounds good.

Did you mean XDG_CACHE_HOME (s/_DIR/_HOME)? Setting that will also catch anything that uses os.UserCacheDir (and doesn't try to override the environment), which seems like a good thing. At some point we might also want to look at macOS and Windows - e.g., it looks like %LocalAppData% is being set, but unclear if that's a directory that gets cleaned.

[prattmic]

Yes, I did mean XDG_CACHE_HOME, apologies.

I suppose I don't know if anything is cleaning $HOME/.cache, but it doesn't look like it. There is also some bazel stuff in there (probably from perf builders).

[cherrymui]

As a side note, I wonder if somehow watchflakes isn't reporting this at all, or if it's going to some issue that we're not looking at. For example, issue #57902 has a post <- log ~ no space left on device pattern yet this hasn't shown up there. I guess we can get most of this information via other means.

I think there was some recent (perhaps a few month ago) update for LUCI report and the log field on watchflakes doesn't match well. This is on my list to look into.