runtime: plan9/amd64 builder hangs

[psilva261]

https://build.golang.org/log/ed9b4a8cec1f938b98f5cc517dcd8ca603486c91

ok  	regexp/syntax	0.552
Test {"go_test:runtime" "runtime"} ran over 20m0s limit (20m0.00085011s); saw output:

XXXBANNERXXX:Test execution environment.
# GOARCH: amd64
# CPU: Intel Core Processor (Broadwell)
# GOOS: plan9
# OS Version: 2000

XXXBANNERXXX:Testing packages.

I've seen similar on 9front/amd64 with less than 16 GB of RAM on recent versions when running all.rc.

[bcmills]

(CC @golang/plan9)

[bcmills]

@psilva261, maybe send a CL to set a GO_TEST_TIMEOUT_SCALE for this builder (in x/build/dashboard/builders.go)?

(Or do you think this is a deadlock in the process?)

[psilva261]

I think it's a deadlock because usually the runtime package can be tested in 1-2 minutes and with all.rc no timeout seems to ever happen

[mauri870]

The way it is timing out without any errors or messages being printed also leads me to believe it is a deadlock. I'll try to reproduce this (have to setup plan9 again) and try to work on a fix.

[mauri870]

Unfortunately I didn't had enough disk space to compile and run the tests to investigate this issue, build stopped halfway through. If someone else is available and willing to give it a try, please feel free to jump in and assist.

[psilva261]

Actually since 8fa9e3b 16GB RAM is needed instead of 8GB. On 9front with all.rc (or run.rc) tests cannot complete runtime and produce no logs. (No CPU load) But running tests directly shows out-of-memory messages:

=== RUN   TestGcPacer/ExceedMemoryLimit
=== PAUSE TestGcPacer/ExceedMemoryLimit
=== RUN   TestGcPacer/ExceedMemoryLimitNoGCPercent
=== PAUSE TestGcPacer/ExceedMemoryLimitNoGCPercent
=== RUN   TestGcPacer/MaintainMemoryLimit
=== PAUSE TestGcPacer/MaintainMemoryLimit
=== RUN   TestGcPacer/MaintainMemoryLimitNoGCPercent
=== PAUSE TestGcPacer/MaintainMemoryLimitNoGCPercent
=== CONT  TestMemmoveAlias
runtime.test 2413: Killed: Insufficient physical memory
runtime.test 2415: Killed: Insufficient physical memory
runtime.test 2412: Killed: Insufficient physical memory
runtime.test 2423: Killed: Insufficient physical memory
exit status: 'runtime.test 2412: Killed: Insufficient physical memory'
FAIL    runtime 50.313s
runtime.test 2609: Killed: Insufficient physical memory
runtime.test 2607: Killed: Insufficient physical memory
runtime.test 2608: Killed: Insufficient physical memory
runtime.test 2414: Killed: Insufficient physical memory
runtime.test 2417: Killed: Insufficient physical memory
runtime.test 2605: Killed: Insufficient physical memory
runtime.test 2606: Killed: Insufficient physical memory

(runtime tests with go test -v -short and recent patches applied)

run.rc-8GB-RAM.log
run.rc-16GB-RAM.log