You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
gotip-linux-ppc64_power10 builders are consistently failing, almost always with an infra failure that looks like the test runner is killed before it runs any tests.
The bots don't seem completely broken, as x repo builds seem to work OK.
The dashboard shows the first failing CL as https://go.dev/cl/817160, but I'm not sure if that is related or this is an infra problem that occurred at the same time.
Now it is not consistently failing, but there are still OOM failures from time to time, e.g. #82093 . I put a watchflakes pattern on it to catch the flaky failures.
=== RUN TestScript/cover_pkgall_imports
=== PAUSE TestScript/cover_pkgall_imports
=== CONT TestScript/cover_pkgall_imports
script_test.go:141: 2026-10-06T16:36:41Z
script_test.go:143: $WORK=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-891296889/tmpdir958801807/cover_pkgall_imports154385183
go proxy: no archive golang.org v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd v0.1.0: file does not exist
go proxy: no archive golang.org/x v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal/doc v0.1.0: file does not exist
...
> env GO111MODULE=on
> go test -coverpkg=all ./...
[stdout]
example.com/cov/notest FAIL example.com/cov/onlytest [build failed]
FAIL example.com/cov/withtest [build failed]
FAIL
[stderr]
internal/bisect: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64/compile: fork/exec /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64/compile: cannot allocate memory
script_test.go:165: FAIL: testdata/script/cover_pkgall_imports.txt:9: go test -coverpkg=all ./...: exit status 1
--- FAIL: TestScript/cover_pkgall_imports (29.70s)
=== RUN TestScript/cover_pkgall_multiple_mains
=== PAUSE TestScript/cover_pkgall_multiple_mains
=== CONT TestScript/cover_pkgall_multiple_mains
script_test.go:141: 2026-10-06T18:26:01Z
script_test.go:143: $WORK=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-2858938080/tmpdir936362776/cover_pkgall_multiple_mains1762880427
go proxy: no archive golang.org/x/pkgsite/cmd/internal v0.1.0: file does not exist
go proxy: no archive golang.org/x v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal/doc v0.1.0: file does not exist
go proxy: no archive golang.org v0.1.0: file does not exist
...
> env GO111MODULE=on
> go test -coverpkg=all ./...
[stdout]
example.com/cov/mainonly FAIL example.com/cov/mainwithtest [build failed]
FAIL example.com/cov/xtest [build failed]
FAIL
[stderr]
runtime: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64/compile: signal: killed
script_test.go:165: FAIL: testdata/script/cover_pkgall_multiple_mains.txt:8: go test -coverpkg=all ./...: exit status 1
--- FAIL: TestScript/cover_pkgall_multiple_mains (36.40s)
=== RUN TestScript/cover_runs
=== PAUSE TestScript/cover_runs
=== CONT TestScript/cover_runs
script_test.go:141: 2026-10-07T06:49:46Z
script_test.go:143: $WORK=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-1068548180/tmpdir394930794/cover_runs1271586757
script_test.go:165:
PATH=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-1068548180/tmpdir394930794/testbin:/home/swarming/.swarming/w/ir/x/w/goroot/bin:/home/swarming/.swarming/w/ir/x/w/goroot/bin:/home/swarming/.swarming/w/ir/x/w/goroot/bin:/home/swarming/.swarming/w/ir/cache/tools/bin:/home/swarming/.swarming/w/ir/bbagent_utility_packages:/home/swarming/.swarming/w/ir/bbagent_utility_packages/bin:/home/swarming/.swarming/w/ir/cipd_bin_packages:/home/swarming/.swarming/w/ir/cipd_bin_packages/bin:/home/swarming/.swarming/w/ir/cache/cipd_client:/home/swarming/.swarming/w/ir/cache/cipd_client/bin:/home/swarming/.swarming/cipd_cache/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOME=/no-home
CCACHE_DISABLE=1
GOARCH=ppc64
...
matched: ok regexp 0.653s coverage: 18.3% of statements in strings
> go test -short -cover strings math regexp
[stdout]
ok strings 0.189s coverage: 96.3% of statements
ok math 0.049s coverage: 85.6% of statements
signal: killed
FAIL regexp 3.643s
FAIL
script_test.go:165: FAIL: testdata/script/cover_runs.txt:8: go test -short -cover strings math regexp: exit status 1
--- FAIL: TestScript/cover_runs (20.62s)
=== RUN TestScript/cover_pkgall_imports
=== PAUSE TestScript/cover_pkgall_imports
=== CONT TestScript/cover_pkgall_imports
script_test.go:141: 2026-10-06T22:32:17Z
script_test.go:143: $WORK=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-2914670656/tmpdir1196671994/cover_pkgall_imports3035302300
go proxy: no archive golang.org/x/pkgsite/cmd/internal/doc v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd v0.1.0: file does not exist
go proxy: no archive golang.org v0.1.0: file does not exist
go proxy: no archive golang.org/x v0.1.0: file does not exist
...
> env GO111MODULE=on
> go test -coverpkg=all ./...
[stdout]
example.com/cov/notest FAIL example.com/cov/onlytest [build failed]
FAIL example.com/cov/withtest [build failed]
FAIL
[stderr]
runtime: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64/compile: signal: killed
script_test.go:165: FAIL: testdata/script/cover_pkgall_imports.txt:9: go test -coverpkg=all ./...: exit status 1
--- FAIL: TestScript/cover_pkgall_imports (37.24s)
=== RUN TestScript/cover_pkgall_imports
=== PAUSE TestScript/cover_pkgall_imports
=== CONT TestScript/cover_pkgall_imports
script_test.go:141: 2026-10-07T04:25:54Z
script_test.go:143: $WORK=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-665522584/tmpdir2527675834/cover_pkgall_imports852261711
go proxy: no archive golang.org/x/pkgsite/cmd/internal/doc v0.1.0: file does not exist
go proxy: no archive golang.org/x v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal v0.1.0: file does not exist
go proxy: no archive golang.org v0.1.0: file does not exist
...
> env GO111MODULE=on
> go test -coverpkg=all ./...
[stdout]
example.com/cov/notest FAIL example.com/cov/onlytest [build failed]
FAIL example.com/cov/withtest [build failed]
FAIL
[stderr]
runtime: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64/compile: signal: killed
script_test.go:165: FAIL: testdata/script/cover_pkgall_imports.txt:9: go test -coverpkg=all ./...: exit status 1
--- FAIL: TestScript/cover_pkgall_imports (29.09s)
=== RUN TestScript/cover_runs
=== PAUSE TestScript/cover_runs
=== CONT TestScript/cover_runs
script_test.go:141: 2026-10-07T00:04:53Z
script_test.go:143: $WORK=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-4293414848/tmpdir1376721/cover_runs699593374
script_test.go:165:
PATH=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-4293414848/tmpdir1376721/testbin:/home/swarming/.swarming/w/ir/x/w/goroot/bin:/home/swarming/.swarming/w/ir/x/w/goroot/bin:/home/swarming/.swarming/w/ir/x/w/goroot/bin:/home/swarming/.swarming/w/ir/cache/tools/bin:/home/swarming/.swarming/w/ir/bbagent_utility_packages:/home/swarming/.swarming/w/ir/bbagent_utility_packages/bin:/home/swarming/.swarming/w/ir/cipd_bin_packages:/home/swarming/.swarming/w/ir/cipd_bin_packages/bin:/home/swarming/.swarming/w/ir/cache/cipd_client:/home/swarming/.swarming/w/ir/cache/cipd_client/bin:/home/swarming/.swarming/cipd_cache/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOME=/no-home
CCACHE_DISABLE=1
GOARCH=ppc64
...
matched: ok regexp 0.669s coverage: 18.3% of statements in strings
> go test -short -cover strings math regexp
[stdout]
ok strings 0.199s coverage: 96.3% of statements
ok math 0.099s coverage: 85.6% of statements
signal: killed
FAIL regexp 4.389s
FAIL
script_test.go:165: FAIL: testdata/script/cover_runs.txt:8: go test -short -cover strings math regexp: exit status 1
--- FAIL: TestScript/cover_runs (23.40s)
=== RUN TestScript/cover_pkgall_imports
=== PAUSE TestScript/cover_pkgall_imports
=== CONT TestScript/cover_pkgall_imports
script_test.go:141: 2026-10-07T02:15:05Z
script_test.go:143: $WORK=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-2313142556/tmpdir3208887344/cover_pkgall_imports2781281911
go proxy: no archive golang.org v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal/doc v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd v0.1.0: file does not exist
go proxy: no archive golang.org/x v0.1.0: file does not exist
...
> env GO111MODULE=on
> go test -coverpkg=all ./...
[stdout]
example.com/cov/notest FAIL example.com/cov/onlytest [build failed]
FAIL example.com/cov/withtest [build failed]
FAIL
[stderr]
runtime: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64/compile: signal: killed
script_test.go:165: FAIL: testdata/script/cover_pkgall_imports.txt:9: go test -coverpkg=all ./...: exit status 1
--- FAIL: TestScript/cover_pkgall_imports (32.69s)
=== RUN TestScript/cover_pkgall_imports
=== PAUSE TestScript/cover_pkgall_imports
=== CONT TestScript/cover_pkgall_imports
script_test.go:141: 2026-10-09T03:59:54Z
script_test.go:143: $WORK=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-4145337573/tmpdir687616573/cover_pkgall_imports2692882368
go proxy: no archive golang.org/x/pkgsite/cmd/internal v0.1.0: file does not exist
go proxy: no archive golang.org/x v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal/doc v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd v0.1.0: file does not exist
go proxy: no archive golang.org v0.1.0: file does not exist
...
> env GO111MODULE=on
> go test -coverpkg=all ./...
[stdout]
example.com/cov/notest FAIL example.com/cov/onlytest [build failed]
FAIL example.com/cov/withtest [build failed]
FAIL
[stderr]
runtime: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64/compile: signal: killed
script_test.go:165: FAIL: testdata/script/cover_pkgall_imports.txt:9: go test -coverpkg=all ./...: exit status 1
--- FAIL: TestScript/cover_pkgall_imports (35.85s)
=== RUN TestScript/cover_runs
=== PAUSE TestScript/cover_runs
=== CONT TestScript/cover_runs
script_test.go:141: 2026-10-09T03:59:48Z
script_test.go:143: $WORK=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-4145337573/tmpdir687616573/cover_runs2062498340
script_test.go:165:
PATH=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-4145337573/tmpdir687616573/testbin:/home/swarming/.swarming/w/ir/x/w/goroot/bin:/home/swarming/.swarming/w/ir/x/w/goroot/bin:/home/swarming/.swarming/w/ir/x/w/goroot/bin:/home/swarming/.swarming/w/ir/cache/tools/bin:/home/swarming/.swarming/w/ir/bbagent_utility_packages:/home/swarming/.swarming/w/ir/bbagent_utility_packages/bin:/home/swarming/.swarming/w/ir/cipd_bin_packages:/home/swarming/.swarming/w/ir/cipd_bin_packages/bin:/home/swarming/.swarming/w/ir/cache/cipd_client:/home/swarming/.swarming/w/ir/cache/cipd_client/bin:/home/swarming/.swarming/cipd_cache/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOME=/no-home
CCACHE_DISABLE=1
GOARCH=ppc64
...
matched: ok regexp 0.925s coverage: 18.3% of statements in strings
> go test -short -cover strings math regexp
[stdout]
ok strings 0.342s coverage: 96.3% of statements
ok math 0.142s coverage: 85.6% of statements
signal: killed
FAIL regexp 5.051s
FAIL
script_test.go:165: FAIL: testdata/script/cover_runs.txt:8: go test -short -cover strings math regexp: exit status 1
--- FAIL: TestScript/cover_runs (29.15s)
=== RUN TestScript/cover_pkgall_imports
=== PAUSE TestScript/cover_pkgall_imports
=== CONT TestScript/cover_pkgall_imports
script_test.go:141: 2026-10-08T19:13:55Z
script_test.go:143: $WORK=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-1949574654/tmpdir3830290263/cover_pkgall_imports2135160456
go proxy: no archive golang.org v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal/doc v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal v0.1.0: file does not exist
go proxy: no archive golang.org/x v0.1.0: file does not exist
...
> env GO111MODULE=on
> go test -coverpkg=all ./...
[stdout]
example.com/cov/notest FAIL example.com/cov/onlytest [build failed]
FAIL example.com/cov/withtest [build failed]
FAIL
[stderr]
runtime: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64/compile: signal: killed
script_test.go:165: FAIL: testdata/script/cover_pkgall_imports.txt:9: go test -coverpkg=all ./...: exit status 1
--- FAIL: TestScript/cover_pkgall_imports (30.10s)
=== RUN TestScript/cover_error
=== PAUSE TestScript/cover_error
=== CONT TestScript/cover_error
script_test.go:141: 2026-10-09T01:53:50Z
script_test.go:143: $WORK=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-4064732250/tmpdir4069431132/cover_error1354748283
script_test.go:165:
PATH=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-4064732250/tmpdir4069431132/testbin:/home/swarming/.swarming/w/ir/x/w/goroot/bin:/home/swarming/.swarming/w/ir/x/w/goroot/bin:/home/swarming/.swarming/w/ir/x/w/goroot/bin:/home/swarming/.swarming/w/ir/cache/tools/bin:/home/swarming/.swarming/w/ir/bbagent_utility_packages:/home/swarming/.swarming/w/ir/bbagent_utility_packages/bin:/home/swarming/.swarming/w/ir/cipd_bin_packages:/home/swarming/.swarming/w/ir/cipd_bin_packages/bin:/home/swarming/.swarming/w/ir/cache/cipd_client:/home/swarming/.swarming/w/ir/cache/cipd_client/bin:/home/swarming/.swarming/cipd_cache/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOME=/no-home
CCACHE_DISABLE=1
GOARCH=ppc64
...
> cp p_test.go $WORK/gopath2/src/p_test.go
> cd $WORK/gopath1/src
# Get errors from a go test into stderr.txt (0.910s)
> ! go test coverbad
[stderr]
go: error obtaining buildID for go tool cgo: fork/exec /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64/cgo: cannot allocate memory
[exit status 1]
> stderr '^\.[\\/]p\.go:4:2' # look for error at coverbad/p.go:4
script_test.go:165: FAIL: testdata/script/cover_error.txt:24: stderr ^\.[\\/]p\.go:4:2: no match for `(?m)^\.[\\/]p\.go:4:2` in stderr
--- FAIL: TestScript/cover_error (0.91s)
=== RUN TestScript/cover_pkgall_imports
=== PAUSE TestScript/cover_pkgall_imports
=== CONT TestScript/cover_pkgall_imports
script_test.go:141: 2026-10-09T17:35:24Z
script_test.go:143: $WORK=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-2204536976/tmpdir2086271903/cover_pkgall_imports1131115057
go proxy: no archive golang.org/x/pkgsite/cmd/internal v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd v0.1.0: file does not exist
go proxy: no archive golang.org v0.1.0: file does not exist
go proxy: no archive golang.org/x v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal/doc v0.1.0: file does not exist
...
> env GO111MODULE=on
> go test -coverpkg=all ./...
[stdout]
example.com/cov/notest FAIL example.com/cov/onlytest [build failed]
FAIL example.com/cov/withtest [build failed]
FAIL
[stderr]
runtime: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64/compile: signal: killed
script_test.go:165: FAIL: testdata/script/cover_pkgall_imports.txt:9: go test -coverpkg=all ./...: exit status 1
--- FAIL: TestScript/cover_pkgall_imports (38.55s)
=== RUN TestScript/cover_pkgall_imports
=== PAUSE TestScript/cover_pkgall_imports
=== CONT TestScript/cover_pkgall_imports
script_test.go:141: 2026-10-10T03:37:52Z
script_test.go:143: $WORK=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-3895747806/tmpdir1096536143/cover_pkgall_imports2110705521
go proxy: no archive golang.org v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal/doc v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal v0.1.0: file does not exist
go proxy: no archive golang.org/x v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd v0.1.0: file does not exist
...
> env GO111MODULE=on
> go test -coverpkg=all ./...
[stdout]
example.com/cov/notest FAIL example.com/cov/onlytest [build failed]
FAIL example.com/cov/withtest [build failed]
FAIL
[stderr]
runtime: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64/compile: signal: killed
script_test.go:165: FAIL: testdata/script/cover_pkgall_imports.txt:9: go test -coverpkg=all ./...: exit status 1
--- FAIL: TestScript/cover_pkgall_imports (36.37s)
=== RUN TestScript/cover_pkgall_imports
=== PAUSE TestScript/cover_pkgall_imports
=== CONT TestScript/cover_pkgall_imports
script_test.go:141: 2026-10-09T20:56:57Z
script_test.go:143: $WORK=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-3824932379/tmpdir557185659/cover_pkgall_imports1696668719
go proxy: no archive golang.org v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal/doc v0.1.0: file does not exist
go proxy: no archive golang.org/x v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd v0.1.0: file does not exist
...
> env GO111MODULE=on
> go test -coverpkg=all ./...
[stdout]
example.com/cov/notest FAIL example.com/cov/onlytest [build failed]
FAIL example.com/cov/withtest [build failed]
FAIL
[stderr]
runtime: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64/compile: signal: killed
script_test.go:165: FAIL: testdata/script/cover_pkgall_imports.txt:9: go test -coverpkg=all ./...: exit status 1
--- FAIL: TestScript/cover_pkgall_imports (34.54s)
=== RUN TestScript/cover_runs
=== PAUSE TestScript/cover_runs
=== CONT TestScript/cover_runs
script_test.go:141: 2026-10-09T21:54:08Z
script_test.go:143: $WORK=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-2988628540/tmpdir3113210856/cover_runs4178276049
script_test.go:165:
PATH=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-2988628540/tmpdir3113210856/testbin:/home/swarming/.swarming/w/ir/x/w/goroot/bin:/home/swarming/.swarming/w/ir/x/w/goroot/bin:/home/swarming/.swarming/w/ir/x/w/goroot/bin:/home/swarming/.swarming/w/ir/cache/tools/bin:/home/swarming/.swarming/w/ir/bbagent_utility_packages:/home/swarming/.swarming/w/ir/bbagent_utility_packages/bin:/home/swarming/.swarming/w/ir/cipd_bin_packages:/home/swarming/.swarming/w/ir/cipd_bin_packages/bin:/home/swarming/.swarming/w/ir/cache/cipd_client:/home/swarming/.swarming/w/ir/cache/cipd_client/bin:/home/swarming/.swarming/cipd_cache/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOME=/no-home
CCACHE_DISABLE=1
GOARCH=ppc64
...
matched: ok regexp 0.819s coverage: 18.3% of statements in strings
> go test -short -cover strings math regexp
[stdout]
ok strings 0.075s coverage: 96.3% of statements
ok math 0.079s coverage: 85.6% of statements
signal: killed
FAIL regexp 1.999s
FAIL
script_test.go:165: FAIL: testdata/script/cover_runs.txt:8: go test -short -cover strings math regexp: exit status 1
--- FAIL: TestScript/cover_runs (19.28s)
=== RUN TestScript/cover_pkgall_multiple_mains
=== PAUSE TestScript/cover_pkgall_multiple_mains
=== CONT TestScript/cover_pkgall_multiple_mains
script_test.go:141: 2026-10-14T20:11:06Z
script_test.go:143: $WORK=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-521611815/tmpdir3098255540/cover_pkgall_multiple_mains2887289680
go proxy: no archive golang.org v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal/doc v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal v0.1.0: file does not exist
go proxy: no archive golang.org/x v0.1.0: file does not exist
...
> env GO111MODULE=on
> go test -coverpkg=all ./...
[stdout]
example.com/cov/mainonly FAIL example.com/cov/mainwithtest [build failed]
FAIL example.com/cov/xtest [build failed]
FAIL
[stderr]
runtime: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64/compile: signal: killed
script_test.go:165: FAIL: testdata/script/cover_pkgall_multiple_mains.txt:8: go test -coverpkg=all ./...: exit status 1
--- FAIL: TestScript/cover_pkgall_multiple_mains (29.56s)
=== RUN TestScript/cover_runs
=== PAUSE TestScript/cover_runs
=== CONT TestScript/cover_runs
script_test.go:141: 2026-10-14T20:11:01Z
script_test.go:143: $WORK=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-521611815/tmpdir3098255540/cover_runs2008135171
script_test.go:165:
PATH=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-521611815/tmpdir3098255540/testbin:/home/swarming/.swarming/w/ir/x/w/goroot/bin:/home/swarming/.swarming/w/ir/x/w/goroot/bin:/home/swarming/.swarming/w/ir/x/w/goroot/bin:/home/swarming/.swarming/w/ir/cache/tools/bin:/home/swarming/.swarming/w/ir/bbagent_utility_packages:/home/swarming/.swarming/w/ir/bbagent_utility_packages/bin:/home/swarming/.swarming/w/ir/cipd_bin_packages:/home/swarming/.swarming/w/ir/cipd_bin_packages/bin:/home/swarming/.swarming/w/ir/cache/cipd_client:/home/swarming/.swarming/w/ir/cache/cipd_client/bin:/home/swarming/.swarming/cipd_cache/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOME=/no-home
CCACHE_DISABLE=1
GOARCH=ppc64
...
matched: ok regexp 0.915s coverage: 18.3% of statements in strings
> go test -short -cover strings math regexp
[stdout]
ok strings 0.180s coverage: 96.3% of statements
ok math 0.017s coverage: 85.6% of statements
signal: killed
FAIL regexp 2.245s
FAIL
script_test.go:165: FAIL: testdata/script/cover_runs.txt:8: go test -short -cover strings math regexp: exit status 1
--- FAIL: TestScript/cover_runs (20.24s)
=== RUN TestScript/cover_pkgall_imports
=== PAUSE TestScript/cover_pkgall_imports
=== CONT TestScript/cover_pkgall_imports
script_test.go:141: 2026-10-15T16:33:36Z
script_test.go:143: $WORK=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-4198841473/tmpdir4137762747/cover_pkgall_imports2516088230
go proxy: no archive golang.org/x/pkgsite/cmd/internal v0.1.0: file does not exist
go proxy: no archive golang.org/x v0.1.0: file does not exist
go proxy: no archive golang.org v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal/doc v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd v0.1.0: file does not exist
...
> env GO111MODULE=on
> go test -coverpkg=all ./...
[stdout]
example.com/cov/notest FAIL example.com/cov/onlytest [build failed]
FAIL example.com/cov/withtest [build failed]
FAIL
[stderr]
runtime: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64/compile: signal: killed
script_test.go:165: FAIL: testdata/script/cover_pkgall_imports.txt:9: go test -coverpkg=all ./...: exit status 1
--- FAIL: TestScript/cover_pkgall_imports (28.10s)
=== RUN TestScript/cover_pkgall_imports
=== PAUSE TestScript/cover_pkgall_imports
=== CONT TestScript/cover_pkgall_imports
script_test.go:141: 2026-10-16T18:09:03Z
script_test.go:143: $WORK=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-1365037110/tmpdir3617890909/cover_pkgall_imports4090843124
go proxy: no archive golang.org/x/pkgsite/cmd/internal/doc v0.1.0: file does not exist
go proxy: no archive golang.org/x v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal v0.1.0: file does not exist
go proxy: no archive golang.org v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd v0.1.0: file does not exist
...
> env GO111MODULE=on
> go test -coverpkg=all ./...
[stdout]
example.com/cov/notest FAIL example.com/cov/onlytest [build failed]
FAIL example.com/cov/withtest [build failed]
FAIL
[stderr]
runtime: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64/compile: signal: killed
script_test.go:165: FAIL: testdata/script/cover_pkgall_imports.txt:9: go test -coverpkg=all ./...: exit status 1
--- FAIL: TestScript/cover_pkgall_imports (36.96s)
=== RUN TestScript/cover_pkgall_imports
=== PAUSE TestScript/cover_pkgall_imports
=== CONT TestScript/cover_pkgall_imports
script_test.go:141: 2026-10-16T22:41:57Z
script_test.go:143: $WORK=/home/swarming/.swarming/w/ir/x/t/cmd-go-test-3428664159/tmpdir973315666/cover_pkgall_imports3822434501
go proxy: no archive golang.org v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal v0.1.0: file does not exist
go proxy: no archive golang.org/x v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd/internal/doc v0.1.0: file does not exist
go proxy: no archive golang.org/x/pkgsite/cmd v0.1.0: file does not exist
...
> env GO111MODULE=on
> go test -coverpkg=all ./...
[stdout]
example.com/cov/notest FAIL example.com/cov/onlytest [build failed]
FAIL example.com/cov/withtest [build failed]
FAIL
[stderr]
runtime: /home/swarming/.swarming/w/ir/x/w/goroot/pkg/tool/linux_ppc64/compile: signal: killed
script_test.go:165: FAIL: testdata/script/cover_pkgall_imports.txt:9: go test -coverpkg=all ./...: exit status 1
--- FAIL: TestScript/cover_pkgall_imports (33.71s)
Activity
pmur commentedon Sep 4, 2026
Looking at the logs, I see a bunch of recent OOM kills.
The last couple of logs point at a very large amount being used by the "file" bucket.
gabyhelp commentedon Sep 4, 2026
Related Issues
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)
pmur commentedon Sep 8, 2026
I've reconfigured the ppc64-p10 builder from 5 to 4 containers, and increased the memory limits from 10 to 12g.
Maybe that's enough to mitigate for now, but something seems to have increased the tmpfs requirements recently.
[-]x/build: gotip-linux-ppc64_power10 consistently failing[/-][+]x/build: gotip-linux-ppc64_power10 OOM failing[/+]cherrymui commentedon Oct 13, 2026
Now it is not consistently failing, but there are still OOM failures from time to time, e.g. #82093 . I put a watchflakes pattern on it to catch the flaky failures.
pmur commentedon Oct 13, 2026
I suspect something isn't cleaning up the cache, tmpfs, or whatever as frequently as it should be.
Most of the OOM are occuring because the ramfs holding the cache/tmpfs keeps going over the container limits.
gopherbot commentedon Oct 13, 2026
Found new dashboard test flakes for:
2026-10-05 12:44 gotip-linux-ppc64_power10 go@0adda8ab cmd/link/internal/ld.TestLargeTextSectionSplitting (log)
2026-10-06 16:22 gotip-linux-ppc64_power10 go@ffef0611 cmd/go.TestScript/cover_pkgall_imports (log)
2026-10-06 16:22 gotip-linux-ppc64_power10 go@ffef0611 cmd/go.TestScript/cover_pkgall_multiple_mains (log)
2026-10-06 16:22 gotip-linux-ppc64_power10 go@ffef0611 cmd/go.TestScript/cover_pkgall_runtime (log)
2026-10-06 18:15 gotip-linux-ppc64_power10 go@7ccdc93a cmd/go.TestScript/cover_pkgall_multiple_mains (log)
2026-10-06 18:15 gotip-linux-ppc64_power10 go@7ccdc93a cmd/go.TestScript/cover_pkgall_runtime (log)
2026-10-06 18:46 gotip-linux-ppc64_power10 go@0042b271 cmd/go.TestScript/cover_pkgall_runtime (log)
2026-10-06 20:35 gotip-linux-ppc64_power10 go@2fcf747c cmd/go.TestScript/cover_pkgall_runtime (log)
2026-10-06 20:46 gotip-linux-ppc64_power10 go@dc0e349a cmd/go.TestScript/cover_pkgall_runtime (log)
2026-10-06 20:46 gotip-linux-ppc64_power10 go@dc0e349a cmd/go.TestScript/cover_runs (log)
2026-10-06 20:46 gotip-linux-ppc64_power10 go@6fb3b12e cmd/go.TestScript/cover_pkgall_imports (log)
2026-10-06 20:46 gotip-linux-ppc64_power10 go@611b5665 cmd/go.TestScript/cover_pkgall_imports (log)
2026-10-06 20:46 gotip-linux-ppc64_power10 go@611b5665 cmd/go.TestScript/cover_pkgall_runtime (log)
2026-10-06 22:13 gotip-linux-ppc64_power10 go@7418802d cmd/go.TestScript/cover_pkgall_imports (log)
2026-10-06 22:13 gotip-linux-ppc64_power10 go@7418802d cmd/go.TestScript/cover_pkgall_runtime (log)
2026-10-06 23:04 gotip-linux-ppc64_power10 go@4213d167 cmd/go.TestScript/cover_pkgall_imports (log)
2026-10-06 23:04 gotip-linux-ppc64_power10 go@4213d167 cmd/go.TestScript/cover_pkgall_multiple_mains (log)
2026-10-06 23:08 gotip-linux-ppc64_power10 go@490b7566 cmd/go.TestScript/cover_pkgall_runtime (log)
2026-10-06 23:08 gotip-linux-ppc64_power10 go@490b7566 cmd/go.TestScript/cover_runs (log)
2026-10-06 23:54 gotip-linux-ppc64_power10 go@5a035777 cmd/go.TestScript/cover_pkgall_imports (log)
2026-10-07 15:29 gotip-linux-ppc64_power10 go@c9729f4e cmd/go.TestScript/cover_pkgall_multiple_mains (log)
2026-10-07 18:11 gotip-linux-ppc64_power10 go@eaf9c685 cmd/go.TestScript/cover_pkgall_multiple_mains (log)
2026-10-07 18:11 gotip-linux-ppc64_power10 go@eaf9c685 cmd/go.TestScript/cover_pkgall_runtime (log)
2026-10-07 22:53 gotip-linux-ppc64_power10 go@0e2ac8f9 cmd/go.TestScript/cover_pkgall_runtime (log)
2026-10-08 18:28 gotip-linux-ppc64_power10 go@d73c92c0 cmd/go.TestScript/cover_pkgall_imports (log)
2026-10-08 18:28 gotip-linux-ppc64_power10 go@d73c92c0 cmd/go.TestScript/cover_runs (log)
2026-10-08 18:38 gotip-linux-ppc64_power10 go@7b536ca2 cmd/go.TestScript/cover_pkgall_imports (log)
2026-10-08 18:38 gotip-linux-ppc64_power10 go@7b536ca2 cmd/go.TestScript/cover_pkgall_multiple_mains (log)
2026-10-08 18:38 gotip-linux-ppc64_power10 go@7b536ca2 cmd/go.TestScript/cover_pkgall_runtime (log)
— watchflakes
gopherbot commentedon Oct 14, 2026
Found new dashboard test flakes for:
2026-10-08 18:47 gotip-linux-ppc64_power10 go@050075e5 cmd/go.TestScript/cover_pkgall_multiple_mains (log)
2026-10-08 18:55 gotip-linux-ppc64_power10 go@ba8f5c9a cmd/go.TestScript/cover_pkgall_multiple_mains (log)
2026-10-09 01:33 gotip-linux-ppc64_power10 go@a3b72a4a cmd/go.TestScript/cover_error (log)
2026-10-09 01:33 gotip-linux-ppc64_power10 go@a3b72a4a cmd/nm.TestCgoLib (log)
2026-10-09 17:24 gotip-linux-ppc64_power10 go@8c098771 cmd/go.TestScript/cover_pkgall_imports (log)
2026-10-09 17:24 gotip-linux-ppc64_power10 go@8c098771 cmd/go.TestScript/cover_pkgall_runtime (log)
2026-10-09 18:14 gotip-linux-ppc64_power10 go@2af3e7d3 cmd/go.TestScript/cover_pkgall_multiple_mains (log)
2026-10-09 20:12 gotip-linux-ppc64_power10 go@aa0034b6 cmd/go.TestScript/cover_pkgall_imports (log)
2026-10-09 20:12 gotip-linux-ppc64_power10 go@aa0034b6 cmd/go.TestScript/cover_pkgall_runtime (log)
2026-10-09 20:33 gotip-linux-ppc64_power10 go@90e1f5b2 cmd/go.TestScript/cover_pkgall_multiple_mains (log)
2026-10-09 20:33 gotip-linux-ppc64_power10 go@90e1f5b2 cmd/go.TestScript/cover_pkgall_runtime (log)
2026-10-09 20:34 gotip-linux-ppc64_power10 go@e203fe57 cmd/go.TestScript/cover_pkgall_imports (log)
2026-10-09 20:34 gotip-linux-ppc64_power10 go@e203fe57 cmd/go.TestScript/cover_pkgall_runtime (log)
2026-10-09 21:33 gotip-linux-ppc64_power10 go@75821a77 cmd/go.TestScript/cover_runs (log)
2026-10-12 17:34 gotip-linux-ppc64_power10 go@c4b9357a cmd/go.TestScript/cover_pkgall_runtime (log)
2026-10-13 00:02 gotip-linux-ppc64_power10 go@a6d6c003 cmd/go.TestScript/cover_pkgall_runtime (log)
— watchflakes
gopherbot commentedon Oct 15, 2026
Found new dashboard test flakes for:
2026-10-14 00:10 gotip-linux-ppc64_power10 go@e60b4c37 cmd/go.TestScript/cover_pkgall_runtime (log)
2026-10-14 15:33 gotip-linux-ppc64_power10 go@91b0089e cmd/go.TestScript/cover_pkgall_runtime (log)
2026-10-14 15:33 gotip-linux-ppc64_power10 go@91b0089e cmd/link/internal/ld.TestLargeTextSectionSplitting (log)
2026-10-14 15:33 gotip-linux-ppc64_power10 go@e8a42956 cmd/go.TestScript/cover_pkgall_multiple_mains (log)
2026-10-14 19:59 gotip-linux-ppc64_power10 go@8b3ed8a0 cmd/go.TestScript/cover_pkgall_multiple_mains (log)
2026-10-14 19:59 gotip-linux-ppc64_power10 go@8b3ed8a0 cmd/go.TestScript/cover_runs (log)
— watchflakes
gopherbot commentedon Oct 15, 2026
Found new dashboard test flakes for:
2026-10-14 22:04 gotip-linux-ppc64_power10 go@e629bade cmd/go.TestScript/cover_pkgall_imports (log)
2026-10-14 22:04 gotip-linux-ppc64_power10 go@e629bade cmd/go.TestScript/cover_pkgall_multiple_mains (log)
2026-10-14 22:04 gotip-linux-ppc64_power10 go@e629bade cmd/go.TestScript/cover_pkgall_runtime (log)
— watchflakes
gopherbot commentedon Oct 15, 2026
Found new dashboard test flakes for:
2026-10-15 16:20 gotip-linux-ppc64_power10 go@c7eb5b63 cmd/go.TestScript/cover_pkgall_imports (log)
— watchflakes
gopherbot commentedon Oct 16, 2026
Found new dashboard test flakes for:
2026-10-15 16:33 gotip-linux-ppc64_power10 go@02e6c32a cmd/go.TestScript/cover_pkgall_runtime (log)
— watchflakes
gopherbot commentedon Oct 16, 2026
Found new dashboard test flakes for:
2026-10-16 17:58 gotip-linux-ppc64_power10 go@2924f247 cmd/go.TestScript/cover_pkgall_imports (log)
2026-10-16 17:58 gotip-linux-ppc64_power10 go@2924f247 cmd/go.TestScript/cover_pkgall_multiple_mains (log)
2026-10-16 17:58 gotip-linux-ppc64_power10 go@2924f247 cmd/go.TestScript/cover_pkgall_runtime (log)
2026-10-16 18:56 gotip-linux-ppc64_power10 go@e2731651 cmd/go.TestScript/cover_pkgall_multiple_mains (log)
2026-10-16 18:56 gotip-linux-ppc64_power10 go@e2731651 cmd/go.TestScript/cover_pkgall_runtime (log)
2026-10-16 20:00 gotip-linux-ppc64_power10 go@0564ffd5 cmd/go.TestScript/cover_pkgall_runtime (log)
— watchflakes
gopherbot commentedon Oct 17, 2026
Found new dashboard test flakes for:
2026-10-16 20:45 gotip-linux-ppc64_power10 go@efb714d0 cmd/go.TestScript/cover_pkgall_multiple_mains (log)
2026-10-16 20:45 gotip-linux-ppc64_power10 go@a40a1665 cmd/go.TestScript/cover_pkgall_multiple_mains (log)
2026-10-16 20:45 gotip-linux-ppc64_power10 go@a40a1665 cmd/go.TestScript/cover_pkgall_runtime (log)
2026-10-16 20:54 gotip-linux-ppc64_power10 go@3b3ca4b7 cmd/go.TestScript/cover_pkgall_imports (log)
— watchflakes
gopherbot commentedon Oct 17, 2026
Found new dashboard test flakes for:
2026-10-16 21:47 gotip-linux-ppc64_power10 go@c3a068ee cmd/go.TestScript/cover_pkgall_runtime (log)
— watchflakes
gopherbot commentedon Oct 17, 2026
Found new dashboard test flakes for:
2026-10-16 19:55 gotip-linux-ppc64_power10 go@e940b0c9 cmd/go.TestScript/cover_pkgall_imports (log)
— watchflakes