runtime: plan9/amd64 builder hangs

[psilva261]

https://build.golang.org/log/ed9b4a8cec1f938b98f5cc517dcd8ca603486c91

ok  	regexp/syntax	0.552
Test {"go_test:runtime" "runtime"} ran over 20m0s limit (20m0.00085011s); saw output:

XXXBANNERXXX:Test execution environment.
# GOARCH: amd64
# CPU: Intel Core Processor (Broadwell)
# GOOS: plan9
# OS Version: 2000

XXXBANNERXXX:Testing packages.

I've seen similar on 9front/amd64 with less than 16 GB of RAM on recent versions when running all.rc.

[bcmills]

(CC @golang/plan9)

[bcmills]

@psilva261, maybe send a CL to set a GO_TEST_TIMEOUT_SCALE for this builder (in x/build/dashboard/builders.go)?

(Or do you think this is a deadlock in the process?)

[psilva261]

I think it's a deadlock because usually the runtime package can be tested in 1-2 minutes and with all.rc no timeout seems to ever happen

[mauri870]

The way it is timing out without any errors or messages being printed also leads me to believe it is a deadlock. I'll try to reproduce this (have to setup plan9 again) and try to work on a fix.

[mauri870]

Unfortunately I didn't had enough disk space to compile and run the tests to investigate this issue, build stopped halfway through. If someone else is available and willing to give it a try, please feel free to jump in and assist.

[psilva261]

Actually since 8fa9e3b 16GB RAM is needed instead of 8GB. On 9front with all.rc (or run.rc) tests cannot complete runtime and produce no logs. (No CPU load) But running tests directly shows out-of-memory messages:

=== RUN   TestGcPacer/ExceedMemoryLimit
=== PAUSE TestGcPacer/ExceedMemoryLimit
=== RUN   TestGcPacer/ExceedMemoryLimitNoGCPercent
=== PAUSE TestGcPacer/ExceedMemoryLimitNoGCPercent
=== RUN   TestGcPacer/MaintainMemoryLimit
=== PAUSE TestGcPacer/MaintainMemoryLimit
=== RUN   TestGcPacer/MaintainMemoryLimitNoGCPercent
=== PAUSE TestGcPacer/MaintainMemoryLimitNoGCPercent
=== CONT  TestMemmoveAlias
runtime.test 2413: Killed: Insufficient physical memory
runtime.test 2415: Killed: Insufficient physical memory
runtime.test 2412: Killed: Insufficient physical memory
runtime.test 2423: Killed: Insufficient physical memory
exit status: 'runtime.test 2412: Killed: Insufficient physical memory'
FAIL    runtime 50.313s
runtime.test 2609: Killed: Insufficient physical memory
runtime.test 2607: Killed: Insufficient physical memory
runtime.test 2608: Killed: Insufficient physical memory
runtime.test 2414: Killed: Insufficient physical memory
runtime.test 2417: Killed: Insufficient physical memory
runtime.test 2605: Killed: Insufficient physical memory
runtime.test 2606: Killed: Insufficient physical memory

(runtime tests with go test -v -short and recent patches applied)

run.rc-8GB-RAM.log
run.rc-16GB-RAM.log

[denzuko]

@mauri870 I did speak with Ori and the guys on 9gridchan about this today. Since I've already been building out a builder cluster my own use and for some business clients, I'll start tackling some of the plan9/9front arm64 issues doing a series of hackathon streams starting this weekend.

@psilva261 sounds like all.rc needs some optimizations for a start right?

[psilva261]

@denzuko Yeah probably. At the moment I can reproduce it only with 4 GB RAM without having all build files in /tmp though. (Maybe there were already other changes that make the problem less drastic) But still then the tests would hang.

I think it's multiple things at once:

• out of memory
• parent process runtime.test ... keeps running despite a child process being killed because of out of memory (this might be a problem with os/exec on plan9. That reminds me of os/exec: TestContextCancel failures on plan9/amd64 #60793)
• out of memory error not visible in the builder because go test waits for test to finish/exit to print output (eventually at least the timeout from the go test ... archive/tar archive/zip ... runtime ... invocation for all packages should make run.rc exit and print the error)

Here's also a screenshot of pstree with a -v included in

go/src/cmd/dist/test.go

Line 418 in e9b3ff1

func (opts *goTest) buildArgs(t *tester) (build, run, pkgs, testFlags []string, setupCmd func(*exec.Cmd)) {

[denzuko]

@psilva261 Thanks for the screenshot and details. That helps tremendously. Does sounds like a lot of this is coming down to memory constraints. I'll have to dig around but will help out as I can.