x/tools: multiple tests timing out on `darwin-amd64-longtest`

[bcmills]

https://build.golang.org/log/56b688f3bdc981ffdeb03abab6d7519abbb782a6 shows numerous failures & timeouts for the x/tools repo on the new darwin-amd64-longtest builder (#35678).

They all appear to have goroutines in the [runnable] state, so perhaps we just need to bump the GO_TEST_TIMEOUT_SCALE even higher for this builder..? (attn @prattmic)

golang.org/x/tools/cmd/godoc

--- FAIL: TestWebIndex (2708.59s)
    exec.go:109: test timed out while running command: /tmp/buildlet/tmp/go-build3607350883/b272/godoc.test -http=127.0.0.1:49195 -index -index_interval=-1s
    godoc_test.go:247: /tmp/buildlet/tmp/go-build3607350883/b272/godoc.test -http=127.0.0.1:49195 -index -index_interval=-1s: exit status 2
    godoc_test.go:83: server failed to respond in 45m7.510107723s
    godoc_test.go:258: server output:
        using GOPATH mode
        SIGQUIT: quit
        PC=0x7ff80d4ff1fe m=0 sigcode=0
…
        goroutine 163 [runnable]:
        runtime.asyncPreempt2()
        	/tmp/buildlet/go/src/runtime/preempt.go:307 +0x39 fp=0xc0001368d8 sp=0xc0001368b8 pc=0x1036db9
        runtime.asyncPreempt()
        	/tmp/buildlet/go/src/runtime/preempt_amd64.s:53 +0xe7 fp=0xc000136a60 sp=0xc0001368d8 pc=0x106ca87
        index/suffixarray.assignID_8_32({0xc01ddcc000, 0x45c4f5f, 0x5000000}, {0xc02e75c000?, 0x45c4f5f?, 0xa3e7669642f3c0a?}, 0xc02e75c000?)
        	/tmp/buildlet/go/src/index/suffixarray/sais.go:613 +0x1a5 fp=0xc000136ab0 sp=0xc000136a60 pc=0x142d685
        index/suffixarray.sais_8_32({0xc01ddcc000, 0x45c4f5f, 0x5000000}, 0x5?, {0xc02e75c000?, 0x45c4f5f, 0x45c4f5f}, {0xc000136bb8, 0x200, 0x200})
        	/tmp/buildlet/go/src/index/suffixarray/sais.go:181 +0x2df fp=0xc000136b68 sp=0xc000136ab0 pc=0x142c8df
        index/suffixarray.text_32({0xc01ddcc000?, 0x0?, 0x5?}, {0xc02e75c000?, 0x0?, 0xc00040c298?})
        	/tmp/buildlet/go/src/index/suffixarray/sais.go:132 +0x86 fp=0xc0001373c8 sp=0xc000136b68 pc=0x142c586
        index/suffixarray.New({0xc01ddcc000, 0x45c4f5f, 0x5000000})
        	/tmp/buildlet/go/src/index/suffixarray/suffixarray.go:79 +0xfb fp=0xc000138430 sp=0xc0001373c8 pc=0x1432d7b
        golang.org/x/tools/godoc.(*Corpus).NewIndex(0xc000304000)
        	/tmp/buildlet/gopath/src/golang.org/x/tools/godoc/index.go:1080 +0x95c fp=0xc000138710 sp=0xc000138430 pc=0x15a58bc
        golang.org/x/tools/godoc.(*Corpus).UpdateIndex(0xc000304000)
        	/tmp/buildlet/gopath/src/golang.org/x/tools/godoc/index.go:1500 +0xb8 fp=0xc000139f38 sp=0xc000138710 pc=0x15a9498
        golang.org/x/tools/godoc.(*Corpus).RunIndexer(0xc000304000)
        	/tmp/buildlet/gopath/src/golang.org/x/tools/godoc/index.go:1535 +0x146 fp=0xc000139fc8 sp=0xc000139f38 pc=0x15a9c46
        golang.org/x/tools/cmd/godoc.main.func16()
        	/tmp/buildlet/gopath/src/golang.org/x/tools/cmd/godoc/main.go:354 +0x25 fp=0xc000139fe0 sp=0xc000139fc8 pc=0x1635e85
        runtime.goexit()
        	/tmp/buildlet/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000139fe8 sp=0xc000139fe0 pc=0x106b4c1
        created by golang.org/x/tools/cmd/godoc.main in goroutine 1
        	/tmp/buildlet/gopath/src/golang.org/x/tools/cmd/godoc/main.go:354 +0x16f0
…
FAIL	golang.org/x/tools/cmd/godoc	2839.480s

golang.org/x/tools/go/loader

package nosuchpkg is not in GOROOT (/tmp/buildlet/go/src/nosuchpkg)
open /tmp/buildlet/gopath/src/golang.org/x/tools/go/loader/missing.go: no such file or directory
open /tmp/buildlet/gopath/src/golang.org/x/tools/go/loader/missing.go: no such file or directory
/tmp/buildlet/gopath/src/golang.org/x/tools/go/loader/testdata/badpkgdecl.go:1:34: expected 'package', found 'EOF'
/tmp/buildlet/gopath/src/golang.org/x/tools/go/loader/testdata/badpkgdecl.go:1:34: expected 'package', found 'EOF'
/go/src/b/x.go:1:21: could not import c (cannot find package "c" in any of:
	/go/src/c (from $GOROOT)
	($GOPATH not set. For more details see: 'go help gopath'))
/go/src/b/x.go:1:21: could not import c (cannot find package "c" in any of:
	/go/src/c (from $GOROOT)
	($GOPATH not set. For more details see: 'go help gopath'))
/go/src/b/x.go:1:21: could not import c (/go/src/c/x.go:1:8: expected 'IDENT', found 'EOF')
/go/src/c/x.go:1:20: expected operand, found 'EOF'
cannot find package "two/three" in any of:
	/go/src/two/three (from $GOROOT)
	($GOPATH not set. For more details see: 'go help gopath')
cannot find package "http" in any of:
	/go/src/vendor/http (vendor tree)
	/go/src/http (from $GOROOT)
	($GOPATH not set. For more details see: 'go help gopath')
/go/src/c/x.go:1:31: cannot convert false (untyped bool constant) to type int
panic: test timed out after 50m0s
running tests:
	TestStdlib (49m50s)

goroutine 6102 [running]:
testing.(*M).startAlarm.func1()
	/tmp/buildlet/go/src/testing/testing.go:2260 +0x3b9
created by time.goFunc
	/tmp/buildlet/go/src/time/sleep.go:176 +0x2d

goroutine 1 [chan receive, 49 minutes]:
testing.(*T).Run(0xc0000829c0, {0x1254eae?, 0xa0c6ddbe04?}, 0x1267c58)
	/tmp/buildlet/go/src/testing/testing.go:1649 +0x3c8
testing.runTests.func1(0x13ef7c0?)
	/tmp/buildlet/go/src/testing/testing.go:2055 +0x3e
testing.tRunner(0xc0000829c0, 0xc0000dbc30)
	/tmp/buildlet/go/src/testing/testing.go:1595 +0xff
testing.runTests(0xc0000bc8c0?, {0x13ea920, 0x19, 0x19}, {0x100cc2a?, 0xd0?, 0x13eeae0?})
	/tmp/buildlet/go/src/testing/testing.go:2053 +0x445
testing.(*M).Run(0xc0000bc8c0)
	/tmp/buildlet/go/src/testing/testing.go:1925 +0x636
golang.org/x/tools/go/loader_test.TestMain(0xc00003a730?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/loader/loader_test.go:33 +0x25
main.main()
	_testmain.go:99 +0x1c6

goroutine 1454 [chan receive, 49 minutes]:
testing.(*T).Parallel(0xc005461040)
	/tmp/buildlet/go/src/testing/testing.go:1403 +0x205
golang.org/x/tools/go/loader_test.loadIO(0xc005461040)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/loader/loader_test.go:831 +0x1c
golang.org/x/tools/go/loader_test.TestLoad1(0x0?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/loader/loader_test.go:827 +0x13
testing.tRunner(0xc005461040, 0x1267be8)
	/tmp/buildlet/go/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/go/src/testing/testing.go:1648 +0x3ad

goroutine 6115 [runnable]:
os.(*File).Read(0xc0672818f0, {0xc02831d000?, 0x180?, 0x180?})
	/tmp/buildlet/go/src/os/file.go:114 +0x9d
io.ReadAll({0x12a4c80, 0xc0672818f0})
	/tmp/buildlet/go/src/io/io.go:701 +0xf9
go/parser.readSource({0xc0673e71c0?, 0x0?}, {0x124b7c0?, 0xc0672818f0?})
	/tmp/buildlet/go/src/go/parser/interface.go:37 +0x10f
go/parser.ParseFile(0xc0003a0140, {0xc0673e71c0, 0x34}, {0x124b7c0?, 0xc0672818f0?}, 0xc0002c00d9?)
	/tmp/buildlet/go/src/go/parser/interface.go:88 +0x8c
golang.org/x/tools/go/loader.parseFiles.func2(0x0, {0xc0673e71c0, 0x34})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/loader/util.go:62 +0x1e5
created by golang.org/x/tools/go/loader.parseFiles in goroutine 1687
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/loader/util.go:43 +0x18f

goroutine 1687 [semacquire]:
sync.runtime_Semacquire(0xc06aac5ad0?)
	/tmp/buildlet/go/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0xc000031400?)
	/tmp/buildlet/go/src/sync/waitgroup.go:116 +0x48
golang.org/x/tools/go/loader.parseFiles(0xc0003a0140, 0xc000000000, 0x0, {0xc049f00ae0, 0x24}, {0xc049ff7840, 0x1, 0xc000581b00?}, 0x0)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/loader/util.go:66 +0x3b2
golang.org/x/tools/go/loader.(*Config).parsePackageFiles(0xc000002000, 0xc049793800, 0x78)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/loader/loader.go:744 +0x177
golang.org/x/tools/go/loader.(*Config).Load(0xc000002000)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/loader/loader.go:618 +0xc6f
golang.org/x/tools/go/loader_test.TestStdlib(0xc005461520)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/loader/stdlib_test.go:53 +0x2fe
testing.tRunner(0xc005461520, 0x1267c58)
	/tmp/buildlet/go/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/go/src/testing/testing.go:1648 +0x3ad

goroutine 1455 [chan receive, 49 minutes]:
testing.(*T).Parallel(0xc0054611e0)
	/tmp/buildlet/go/src/testing/testing.go:1403 +0x205
golang.org/x/tools/go/loader_test.loadIO(0xc0054611e0)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/loader/loader_test.go:831 +0x1c
golang.org/x/tools/go/loader_test.TestLoad2(0xc00598ba80?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/loader/loader_test.go:828 +0x13
testing.tRunner(0xc0054611e0, 0x1267bf0)
	/tmp/buildlet/go/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/go/src/testing/testing.go:1648 +0x3ad
FAIL	golang.org/x/tools/go/loader	3128.076s

golang.org/x/tools/go/pointer

Entering directory `/tmp/buildlet/gopath/src/golang.org/x/tools/go/pointer'
Entering directory `/tmp/buildlet/gopath/src/golang.org/x/tools/go/pointer'
panic: test timed out after 50m0s
running tests:
	TestInput (49m58s)
	TestInput/testdata/a_test.go (49m58s)

goroutine 1494 [running]:
testing.(*M).startAlarm.func1()
	/tmp/buildlet/go/src/testing/testing.go:2260 +0x3b9
created by time.goFunc
	/tmp/buildlet/go/src/time/sleep.go:176 +0x2d

goroutine 1 [chan receive, 49 minutes]:
testing.(*T).Run(0xc000007d40, {0x1341d9b?, 0xa29ab7cd68?}, 0x135aed0)
	/tmp/buildlet/go/src/testing/testing.go:1649 +0x3c8
testing.runTests.func1(0x1577b80?)
	/tmp/buildlet/go/src/testing/testing.go:2055 +0x3e
testing.tRunner(0xc000007d40, 0xc0000a9c88)
	/tmp/buildlet/go/src/testing/testing.go:1595 +0xff
testing.runTests(0xc000090aa0?, {0x15716e0, 0x5, 0x5}, {0xc00008a940?, 0x100c0000a9d10?, 0x1576ea0?})
	/tmp/buildlet/go/src/testing/testing.go:2053 +0x445
testing.(*M).Run(0xc000090aa0)
	/tmp/buildlet/go/src/testing/testing.go:1925 +0x636
main.main()
	_testmain.go:59 +0x19c

goroutine 377 [chan receive, 49 minutes]:
testing.(*T).Run(0xc002454ea0, {0x1345e26?, 0x13491b2?}, 0xc004352d20)
	/tmp/buildlet/go/src/testing/testing.go:1649 +0x3c8
golang.org/x/tools/go/pointer_test.TestInput(0xc002454ea0)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/pointer/pointer_test.go:634 +0x145
testing.tRunner(0xc002454ea0, 0x135aed0)
	/tmp/buildlet/go/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 1
	/tmp/buildlet/go/src/testing/testing.go:1648 +0x3ad

goroutine 378 [runnable]:
golang.org/x/tools/go/pointer.(*nodeset).String.func1(0x32e07?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/container/intsets/sparse.go:1071 +0x8b
golang.org/x/tools/container/intsets.(*block).forEach(0xc00bb15a40, 0xc008142c18)
	/tmp/buildlet/gopath/src/golang.org/x/tools/container/intsets/sparse.go:226 +0x8e
golang.org/x/tools/container/intsets.(*Sparse).forEach(0xc006930208, 0xc00bde9fc0?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/container/intsets/sparse.go:464 +0xaa
golang.org/x/tools/container/intsets.(*Sparse).AppendTo(...)
	/tmp/buildlet/gopath/src/golang.org/x/tools/container/intsets/sparse.go:1071
golang.org/x/tools/go/pointer.(*nodeset).String(0xc006930208)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/pointer/util.go:267 +0xd7
fmt.(*pp).handleMethods(0xc002ffe0d0, 0x12e5c80?)
	/tmp/buildlet/go/src/fmt/print.go:673 +0x2b2
fmt.(*pp).printArg(0xc002ffe0d0, {0x1337660?, 0xc006930208}, 0x73)
	/tmp/buildlet/go/src/fmt/print.go:756 +0x653
fmt.(*pp).doPrintf(0xc002ffe0d0, {0x134a2f7, 0x1a}, {0xc008143198?, 0x4, 0x4})
	/tmp/buildlet/go/src/fmt/print.go:1077 +0x39e
fmt.Fprintf({0x13b0580, 0xc0128b9500}, {0x134a2f7, 0x1a}, {0xc008143198, 0x4, 0x4})
	/tmp/buildlet/go/src/fmt/print.go:224 +0x71
golang.org/x/tools/go/pointer.(*analysis).solve(0xc01274bc00)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/pointer/solve.go:53 +0x27f
golang.org/x/tools/go/pointer.Analyze(0xc0129cb180)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/pointer/analysis.go:329 +0xc46
golang.org/x/tools/go/pointer_test.doOneInput(0x1345e26?, {0xc004468500, 0x4c2}, {0xc0013a2b40, 0x49})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/pointer/pointer_test.go:385 +0x1405
golang.org/x/tools/go/pointer_test.TestInput.func1(0xc002455040)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/pointer/pointer_test.go:657 +0x253
testing.tRunner(0xc002455040, 0xc004352d20)
	/tmp/buildlet/go/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 377
	/tmp/buildlet/go/src/testing/testing.go:1648 +0x3ad
FAIL	golang.org/x/tools/go/pointer	3187.300s

golang.org/x/tools/go/ssa

panic: test timed out after 50m0s
running tests:
	TestStdlib (49m4s)
…
goroutine 10265 [runnable]:
golang.org/x/tools/go/ssa.replaceAll({0x13ddfe0, 0xc06e54e900}, {0x13dd848, 0xc06e527260})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/lift.go:500 +0x24c
golang.org/x/tools/go/ssa.rename(0xc06e551130, {0xc06e71a120?, 0x9, 0x9}, 0x125b7dd?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/lift.go:583 +0x336
golang.org/x/tools/go/ssa.rename(0xc06e5511e0, {0xc06e71a120?, 0x9, 0x9}, 0x125b59d?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/lift.go:645 +0x705
golang.org/x/tools/go/ssa.rename(0xc06e539c30, {0xc06e71a000?, 0x9, 0x9}, 0xc02dd97398?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/lift.go:645 +0x705
golang.org/x/tools/go/ssa.rename(0xc06e539970, {0xc06e71a000?, 0x9, 0x9}, 0x125b7dd?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/lift.go:645 +0x705
golang.org/x/tools/go/ssa.rename(0xc06e539760, {0xc06d5b7ef0?, 0x9, 0x9}, 0x125b59d?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/lift.go:645 +0x705
golang.org/x/tools/go/ssa.rename(0xc06e5393f0, {0xc06d5b7d40?, 0x9, 0x9}, 0x125ad49?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/lift.go:645 +0x705
golang.org/x/tools/go/ssa.rename(0xc06d6b6840, {0xc06d5b7d40?, 0x9, 0x9}, 0x9?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/lift.go:645 +0x705
golang.org/x/tools/go/ssa.rename(0xc06d6b66e0, {0xc06d5b7d40?, 0x9, 0x9}, 0xc02dd97818?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/lift.go:645 +0x705
golang.org/x/tools/go/ssa.lift(0xc00a997800)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/lift.go:214 +0x16d
golang.org/x/tools/go/ssa.(*Function).finishBody(0xc00a997800)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/func.go:320 +0x1e8
golang.org/x/tools/go/ssa.(*builder).buildFunctionBody(0x124900e?, 0xc00a997800)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2403 +0x4c9
golang.org/x/tools/go/ssa.(*builder).buildFunction(0x1249060?, 0xc00a997800)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2326 +0x28
golang.org/x/tools/go/ssa.(*builder).buildCreated(0xc02dd97e18)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2413 +0x1e
golang.org/x/tools/go/ssa.(*Package).build(0xc041d6e580)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2606 +0xc74
sync.(*Once).doSlow(0xc00ff8b310?, 0xc00ffa01c8?)
	/tmp/buildlet/go/src/sync/once.go:74 +0xbf
sync.(*Once).Do(...)
	/tmp/buildlet/go/src/sync/once.go:65
golang.org/x/tools/go/ssa.(*Package).Build(...)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2477
golang.org/x/tools/go/ssa.(*Program).Build.func1(0x0?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2462 +0x46
created by golang.org/x/tools/go/ssa.(*Program).Build in goroutine 2118
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2461 +0x187
…
goroutine 10236 [runnable]:
golang.org/x/tools/go/ssa.(*builder).expr(0x2?, 0xc02392f500, {0x13dbf20, 0xc009ac6180})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:611 +0x92
golang.org/x/tools/go/ssa.(*builder).emitCallArgs(0x13db460?, 0xc02392f500, 0xc012e7e840, 0xc0027c1640, {0x0?, 0x0, 0xc013823428?})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:1065 +0x110
golang.org/x/tools/go/ssa.(*builder).setCall(0x1306940?, 0xc011e34de0?, 0xc0027c1640, 0xc06e682c40)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:1127 +0x88
golang.org/x/tools/go/ssa.(*builder).expr0(0xc013823e18, 0xc02392f500, {0x13dbc20?, 0xc0027c1640?}, {0x7, {0x13db370, 0x159ec20}, {0x0, 0x0}})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:704 +0x2351
golang.org/x/tools/go/ssa.(*builder).expr(0xc000038500?, 0xc02392f500, {0x13dbc20, 0xc0027c1640})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:625 +0x175
golang.org/x/tools/go/ssa.(*builder).cond(0xc06e3f1d40?, 0xc02392f500, {0x13dbc20?, 0xc0027c1640?}, 0x1344580?, 0x13080e0?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:196 +0x430
golang.org/x/tools/go/ssa.(*builder).stmt(0x1?, 0xc02392f500, {0x13dbf50?, 0xc0027c1700?})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2289 +0xc08
golang.org/x/tools/go/ssa.(*builder).stmtList(...)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:946
golang.org/x/tools/go/ssa.(*builder).stmt(0xc02392f500?, 0xc02392f500, {0x13dbbc0?, 0xc00c65b1d0?})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2277 +0xe50
golang.org/x/tools/go/ssa.(*builder).buildFunctionBody(0x124900e?, 0xc02392f500)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2391 +0x41a
golang.org/x/tools/go/ssa.(*builder).buildFunction(0x1249060?, 0xc02392f500)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2326 +0x28
golang.org/x/tools/go/ssa.(*builder).buildCreated(0xc013823e18)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2413 +0x1e
golang.org/x/tools/go/ssa.(*Package).build(0xc030938a00)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2606 +0xc74
sync.(*Once).doSlow(0xc0011ab7b8?, 0xc0011ab7d0?)
	/tmp/buildlet/go/src/sync/once.go:74 +0xbf
sync.(*Once).Do(...)
	/tmp/buildlet/go/src/sync/once.go:65
golang.org/x/tools/go/ssa.(*Package).Build(...)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2477
golang.org/x/tools/go/ssa.(*Program).Build.func1(0x0?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2462 +0x46
created by golang.org/x/tools/go/ssa.(*Program).Build in goroutine 2118
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2461 +0x187
…
goroutine 10350 [runnable]:
golang.org/x/tools/go/ssa.(*builder).expr(0xc06e14d078?, 0xc00027c300, {0x13dbc20, 0xc001b47dc0})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:611 +0x92
golang.org/x/tools/go/ssa.(*builder).stmt(0x1000000000001?, 0xc00027c300, {0x13dbdd0?, 0xc000432c10?})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2160 +0x165
golang.org/x/tools/go/ssa.(*builder).stmtList(...)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:946
golang.org/x/tools/go/ssa.(*builder).stmt(0xc00027c300?, 0xc00027c300, {0x13dbbc0?, 0xc001727e60?})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2277 +0xe50
golang.org/x/tools/go/ssa.(*builder).buildFunctionBody(0x124900e?, 0xc00027c300)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2391 +0x41a
golang.org/x/tools/go/ssa.(*builder).buildFunction(0x1249060?, 0xc00027c300)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2326 +0x28
golang.org/x/tools/go/ssa.(*builder).buildCreated(0xc03ccbde18)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2413 +0x1e
golang.org/x/tools/go/ssa.(*Package).build(0xc02d22af00)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2606 +0xc74
sync.(*Once).doSlow(0xc00d813f60?, 0xc00c807830?)
	/tmp/buildlet/go/src/sync/once.go:74 +0xbf
sync.(*Once).Do(...)
	/tmp/buildlet/go/src/sync/once.go:65
golang.org/x/tools/go/ssa.(*Package).Build(...)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2477
golang.org/x/tools/go/ssa.(*Program).Build.func1(0x3331796c6f703032?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2462 +0x46
created by golang.org/x/tools/go/ssa.(*Program).Build in goroutine 2118
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/ssa/builder.go:2461 +0x187
…
FAIL	golang.org/x/tools/go/ssa	3076.631s

[bcmills]

The current GO_TEST_TIMEOUT_SCALE for this builder is 5:
https://cs.opensource.google/go/x/build/+/master:dashboard/builders.go;l=2613;drc=17fb1f8c9200b9a9675bf8c8d4cad06317848f77

That gives a timeout of ~50m per test, which does seem rather long, but barring some easy way to speed up the builder maybe it's ok not to have great throughput for x/tools as long as we occasionally get through a run.

@golang/release, if we bump the scale factor higher are we going to be at risk of running into some other cmd/coordinator timeout?

[dmitshur]

if we bump the scale factor higher are we going to be at risk of running into some other cmd/coordinator timeout?

The main timeout that comes to mind is the 2 hour global VM timeout here (that an individual host can control for itself via CustomDeleteTimeout). Edit: Actually, host-darwin-amd64-13-aws is a reverse builder so that particular timeout won't apply to it.

A GO_TEST_TIMEOUT_SCALE value of 5 is meant to be "long enough that it shouldn't get in the way of test completion". It seems a bit unlikely 50 mins being too short is the entire problem, but if the easiest thing to learn more here is to try a higher value like 10, that seems fine and that headroom is still available.

[bcmills]

Talking with @prattmic, our theory is that maybe the guest VM is running out of RAM and burning a bunch of CPU time (and disk throughput) on swapping.

hostinfo on a gomote claims only only 4GiB of RAM for 6 CPUs, and that seems like a very low ratio (especially given that most of our tuning parameters and semaphores are based on CPU and not memory).

[gopherbot]

Change https://go.dev/cl/484746 mentions this issue: Revert "env/darwin/aws: reduce guest CPU count to 4"

[gopherbot]

Change https://go.dev/cl/484747 mentions this issue: env/darwin: allocate 10GB memory to guests

[bcmills]

https://build.golang.org/?repo=golang.org%2fx%2ftools shows two passing runs in a row now.

ok  	golang.org/x/tools/cmd/godoc	343.857s
…
ok  	golang.org/x/tools/go/loader	157.235s
…
ok  	golang.org/x/tools/go/pointer	1006.297s
ok  	golang.org/x/tools/go/ssa	233.581s

ok  	golang.org/x/tools/cmd/godoc	337.798s
…
ok  	golang.org/x/tools/go/loader	163.890s
…
ok  	golang.org/x/tools/go/pointer	1495.166s
ok  	golang.org/x/tools/go/ssa	266.768s

I suspect that the go/pointer test is still hitting swap, but maybe we can live with that?

• (see x/tools/go/ssa: requires tons of memory for testing #14113)

[bcmills]

For comparison, on windows-amd64-longtest we have:

ok  	golang.org/x/tools/cmd/godoc	94.144s
…
ok  	golang.org/x/tools/go/loader	58.797s
…
ok  	golang.org/x/tools/go/pointer	156.136s
ok  	golang.org/x/tools/go/ssa	66.562s

(but that VM also has more and faster CPU, and perhaps a faster disk as well.)