x/build: linux-amd64-race TryBot straggling

[bcmills]

In the TryBot run for https://golang.org/cl/176580 at Patchset 4, most of the TryBots ran relatively quickly (less than 10m), but the linux-amd64-race builder took much longer (over 20m).

I'm not sure exactly why it was so slow, but it leads me to believe that there may be something wrong with the sharding for that builder.

(CC @bradfitz @dmitshur @andybons)

[bradfitz]

All the timing data is in BigQuery. That almost always reveals the mystery.

[bradfitz]

Looks like the @rsc's recent suffixarray changes made it slower?

https://console.cloud.google.com/bigquery?sq=872405196845:3b455f3831c54226ba07eda8cea81247

...

SELECT
    Builder, Event, APPROX_QUANTILES(Seconds, 100)[OFFSET(50)] as MedianSec, COUNT(*) as N
FROM
    builds.Spans
WHERE
    Error = ''
    AND Builder = 'linux-amd64-race'
    AND StartTime > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24*7 HOUR)
    AND Repo = "go"
    And IsTry = true
GROUP BY 1, 2
ORDER BY 3 DESC

Row	Builder	Event	MedianSec	N
1	linux-amd64-race	make_and_test	625.462161107	285
2	linux-amd64-race	run_test:go_test_bench:index/suffixarray	388.381372503	278
3	linux-amd64-race	make	130.729277047	276
4	linux-amd64-race	run_test:go_test_bench:runtime	117.176056921	278
5	linux-amd64-race	get_helper	58.835877058	1381
6	linux-amd64-race	get_buildlet	57.736557395	285
7	linux-amd64-race	run_test:go_test_bench:math/big	48.183967875	278
8	linux-amd64-race	create_gce_buildlet	47.280073747	1666
9	linux-amd64-race	run_test:go_test:runtime	43.819345904	279
10	linux-amd64-race	run_test:go_test_bench:net	38.822814725	278
11	linux-amd64-race	run_test:go_test_bench:text/tabwriter	35.119872601	278
12	linux-amd64-race	run_test:go_test:net	33.723890843	281
13	linux-amd64-race	run_test:go_test_bench:bytes	30.623900061	278
14	linux-amd64-race	run_test:go_test_bench:strings	30.278805037	278
15	linux-amd64-race	run_test:go_test_bench:compress/flate	27.559467869	278
16	linux-amd64-race	run_test:go_test:os/exec	26.914272922	279
17	linux-amd64-race	create_gce_instance	25.102813238	1669
18	linux-amd64-race	run_test:go_test_bench:crypto/tls	24.369628886	278
19	linux-amd64-race	run_test:go_test_bench:net/http	21.715966586	278
20	linux-amd64-race	wait_buildlet_start	21.17260563	1666
21	linux-amd64-race	run_test:go_test:os/signal	20.97289739	279
22	linux-amd64-race	run_test:go_test:go/internal/gcimporter	19.999651301	281
23	linux-amd64-race	run_test:go_test:runtime/race	19.223213255	277
24	linux-amd64-race	run_test:go_test_bench:sync	17.451266096	277
25	linux-amd64-race	run_test:go_test_bench:strconv	17.317289797	277
26	linux-amd64-race	run_test:go_test:net/http	17.160193173	280
27	linux-amd64-race	install_race_std	16.297853892	276
28	linux-amd64-race	run_test:go_test_bench:math	15.798795113	277
29	linux-amd64-race	run_test:go_test_bench:context	12.820779637	277
30	linux-amd64-race	run_test:go_test:os	12.79174036	278
31	linux-amd64-race	run_test:go_test:runtime/pprof	12.656720241	274
32	linux-amd64-race	run_test:go_test:go/types	12.650777936	280
33	linux-amd64-race	run_test:go_test:compress/flate	12.288297681	281
34	linux-amd64-race	run_test:go_test_bench:encoding/json	11.248818408	273
35	linux-amd64-race	run_test:go_test:internal/cpu	10.493141873999999	280
36	linux-amd64-race	run_test:go_test_bench:time	10.472495406	272
37	linux-amd64-race	run_test:go_test_bench:math/bits	10.459982868	272
38	linux-amd64-race	run_test:go_test:internal/trace	10.349812287	279
39	linux-amd64-race	run_test:go_test:crypto/tls	10.322860367	281
40	linux-amd64-race	run_test:go_test:compress/zlib	10.282296089999999	281

[bradfitz]

But suffixarray seems fast on linux with -race locally, even before Russ's recent commit.