x/build: trybot build orphaned on SIGQUIT

[findleyr]

One of my recent x/tools presubmit is hung, due to the android-amd64-emu builder apparently being in an unrecoverable 'running' state.
Ongoing log here, but relevant snippet attached below as I believe those links are transient.

In particular, this is the summary after ~30 minutes:

 builder: android-amd64-emu
      rev: 9679b307334bce77cc6e50751956a4c717e9458c
 buildlet: http://10.240.0.50 GCE VM: buildlet-android-amd64-emu-rnf09a960
  started: 2020-08-21 19:03:01.75352996 +0000 UTC m=+3856.957265072
   status: still running

But the test terminated with *** Test killed with quit: ran too long (11m0s).

 builder: android-amd64-emu
      rev: 9679b307334bce77cc6e50751956a4c717e9458c
 buildlet: http://10.240.0.50 GCE VM: buildlet-android-amd64-emu-rnf09a960
  started: 2020-08-21 19:03:01.75352996 +0000 UTC m=+3856.957265072
   status: still running

Events:
  2020-08-21T19:03:01Z checking_for_snapshot 
  2020-08-21T19:03:01Z finish_checking_for_snapshot after 13.1ms
  2020-08-21T19:03:01Z get_buildlet 
  2020-08-21T19:03:27Z finish_get_buildlet after 25.3s
  2020-08-21T19:03:27Z using_buildlet 10.240.0.50:80
  2020-08-21T19:03:27Z write_snapshot_tar 
  2020-08-21T19:03:36Z finish_write_snapshot_tar after 9.2s
  2020-08-21T19:03:36Z make_and_test 
  2020-08-21T19:03:36Z fetching_subrepo tools
  2020-08-21T19:03:36Z get_source tools@f34753f77acda89dbc6358733b5a115685253ee4
  2020-08-21T19:03:36Z finish_get_source after 0s; tools@f34753f77acda89dbc6358733b5a115685253ee4
  2020-08-21T19:03:36Z listing_subrepo_modules tools
  2020-08-21T19:03:37Z finish_listing_subrepo_modules after 30.7ms; tools
  2020-08-21T19:03:37Z running_subrepo_tests tools
 +2116.4s (now)

Build log:
android-amd64-emu at 9679b307334bce77cc6e50751956a4c717e9458c building tools at f34753f77acda89dbc6358733b5a115685253ee4

testing in module mode; GOMOD=/workdir/gopath/src/golang.org/x/tools/go.mod

:: Running /workdir/go/bin/go with args ["/workdir/go/bin/go" "test" "-short" "golang.org/x/tools/..."] and env ["PATH=/workdir/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/android/sdk/platform-tools:/android/sdk/build-tools/27.0.3:/android/gradle/bin" "HOSTNAME=buildlet-android-amd64-emu-rnf09a960" "DEBIAN_FRONTEND=noninteractive" "GO_BUILDER_ENV=android-amd64-emu" "ANDROID_HOME=/android/sdk" "CGO_CFLAGS=-I/android/openal-headers" "CC_FOR_android_386=/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/i686-linux-android26-clang" "CC_FOR_android_amd64=/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/x86_64-linux-android26-clang" "HOME=/root" "USER=root" "GO_STAGE0_NET_DELAY=200ms" "GO_STAGE0_DL_DELAY=300ms" "WORKDIR=/workdir" "GOROOT_BOOTSTRAP=/workdir/go1.4" "GO_BUILDER_NAME=android-amd64-emu" "GOROOT_BOOTSTRAP=/go1.4" "GOARCH=amd64" "GOOS=android" "GOHOSTARCH=amd64" "GOHOSTOS=linux" "CGO_ENABLED=1" "GOROOT=/workdir/go" "GOPATH=/workdir/gopath" "GOPROXY=http://10.240.0.3:30157" "TMPDIR=/workdir/tmp" "GOCACHE=/workdir/gocache"] in dir /workdir/gopath/src/golang.org/x/tools

go: downloading golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543
go: downloading golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208
go: downloading golang.org/x/mod v0.3.0
go: downloading github.com/yuin/goldmark v1.1.32
go: downloading golang.org/x/net v0.0.0-20200625001655-4c5254603344
SIGQUIT: quit
PC=0x4980e8 m=0 sigcode=0

goroutine 1 [syscall]:
syscall.Syscall6(0xf7, 0x1, 0x2df, 0xc0000cb870, 0x1000004, 0x0, 0x0, 0x5067a0, 0x4a6f01, 0xc0000a62a0)
	/workdir/go/src/syscall/asm_linux_amd64.s:41 +0x5 fp=0xc0000cb820 sp=0xc0000cb818 pc=0x4980c5
os.(*Process).blockUntilWaitable(0xc00009e120, 0x4, 0x10, 0x7)
	/workdir/go/src/os/wait_waitid.go:32 +0xa6 fp=0xc0000cb910 sp=0xc0000cb820 pc=0x4a9bc6
os.(*Process).wait(0xc00009e120, 0x5174d8, 0x5174e0, 0x5174d0)
	/workdir/go/src/os/exec_unix.go:22 +0x39 fp=0xc0000cb988 sp=0xc0000cb910 pc=0x4a5e59
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:125
os/exec.(*Cmd).Wait(0xc0000c2000, 0x0, 0x0)
	/workdir/go/src/os/exec/exec.go:507 +0x65 fp=0xc0000cba00 sp=0xc0000cb988 pc=0x4d54a5
os/exec.(*Cmd).Run(0xc0000c2000, 0xc000084630, 0x43da8a)
	/workdir/go/src/os/exec/exec.go:341 +0x5c fp=0xc0000cba28 sp=0xc0000cba00 pc=0x4d48fc
os/exec.(*Cmd).CombinedOutput(0xc0000c2000, 0x3, 0x3, 0xc0000c2000, 0x462425, 0xc000000180)
	/workdir/go/src/os/exec/exec.go:567 +0x91 fp=0xc0000cba58 sp=0xc0000cba28 pc=0x4d5ab1
main.adb(0xc0000cbdb8, 0x3, 0x3, 0x0, 0xc0000a4018)
	/workdir/go/misc/android/go_android_exec.go:51 +0x68 fp=0xc0000cbb00 sp=0xc0000cba58 pc=0x4da948
main.runMain(0x0, 0x0, 0x0)
	/workdir/go/misc/android/go_android_exec.go:97 +0x247 fp=0xc0000cbf48 sp=0xc0000cbb00 pc=0x4dafe7
main.main()
	/workdir/go/misc/android/go_android_exec.go:73 +0x6a fp=0xc0000cbf88 sp=0xc0000cbf48 pc=0x4dad2a
runtime.main()
	/workdir/go/src/runtime/proc.go:205 +0x209 fp=0xc0000cbfe0 sp=0xc0000cbf88 pc=0x435bc9
runtime.goexit()
	/workdir/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0000cbfe8 sp=0xc0000cbfe0 pc=0x4664e1

goroutine 18 [IO wait]:
internal/poll.runtime_pollWait(0x7f1fe4bfcee8, 0x72, 0x533868)
	/workdir/go/src/runtime/netpoll.go:220 +0x55
internal/poll.(*pollDesc).wait(0xc0000a61f8, 0x72, 0xc000100001, 0x5b3, 0x5b3)
	/workdir/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/workdir/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0000a61e0, 0xc00010004d, 0x5b3, 0x5b3, 0x0, 0x0, 0x0)
	/workdir/go/src/internal/poll/fd_unix.go:159 +0x1b1
os.(*File).read(...)
	/workdir/go/src/os/file_posix.go:31
os.(*File).Read(0xc0000a4028, 0xc00010004d, 0x5b3, 0x5b3, 0x1e, 0x0, 0x0)
	/workdir/go/src/os/file.go:116 +0x71
bytes.(*Buffer).ReadFrom(0xc000084630, 0x5336c8, 0xc0000a4028, 0x7f1fe4bb4028, 0xc000084630, 0x1)
	/workdir/go/src/bytes/buffer.go:204 +0xb1
io.copyBuffer(0x5335e8, 0xc000084630, 0x5336c8, 0xc0000a4028, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/workdir/go/src/io/io.go:395 +0x2ff
io.Copy(...)
	/workdir/go/src/io/io.go:368
os/exec.(*Cmd).writerDescriptor.func1(0x0, 0x0)
	/workdir/go/src/os/exec/exec.go:311 +0x65
os/exec.(*Cmd).Start.func1(0xc0000c2000, 0xc00009a0a0)
	/workdir/go/src/os/exec/exec.go:441 +0x27
created by os/exec.(*Cmd).Start
	/workdir/go/src/os/exec/exec.go:440 +0x629

rax    0xf7
rbx    0xc000024800
rcx    0x4980ea
rdx    0xc0000cb870
rdi    0x1
rsi    0x2df
rbp    0xc0000cb900
rsp    0xc0000cb818
r8     0x0
r9     0x0
r10    0x1000004
r11    0x212
r12    0xf1
r13    0x0
r14    0x52f80a
r15    0x0
rip    0x4980e8
rflags 0x212
cs     0x33
fs     0x0
gs     0x0
*** Test killed with quit: ran too long (11m0s).

[dmitshur]

/cc @toothrot @cagedmantis @andybons

[dmitshur]

What was the CL where this happened? Did it eventually fail, or did you have to abort trybots?

[findleyr]

Oops, sorry. It was https://golang.org/cl/249702, patchset 3.

I eventually just uploaded another patchset to trigger a re-run of the builders.

This is definitely a minor bug, considering its infrequency and the availability of a workaround (uploading a new patchset), but I wanted to file an issue for the record. We can probably just ignore it...

[dmitshur]

No worries, thank you for the report.