x/build/coordinator: retrying test on same buildlet after timeout #12685
Description
Something's still not quite right with the retrying of failed sharded tests.
Here the spectralnorm shootout test times out after 20 minutes, three times in a row on the same buildlet. After a failure, it should not be reused, yet the logs show it always using "scaleway-prod-27" (maybe the logs are wrong? maybe it did use different buildlets?)
And worse, then shootout:knucleotide gets run on the same buildlet, despite that one having already been closed.
Investigate.
linux-arm-arm5 rev c0ca9f46, failed; build log; http://scaleway-prod-27 reverse peer scaleway-prod-27/212.47.248.197:38262 for modes [linux-arm linux-arm-arm5], 7m44.336474212s ago
2015-09-18T20:38:09Z get_buildlet
+0.0s 2015-09-18T20:38:09Z got_machine http://scaleway-prod-27 reverse peer scaleway-prod-27/212.47.248.197:38262 for modes [linux-arm linux-arm-arm5]
+0.4s 2015-09-18T20:38:10Z cleaned_up scaleway-prod-27
+0.0s 2015-09-18T20:38:10Z got_buildlet scaleway-prod-27
+17.1s 2015-09-18T20:38:27Z start_write_snapshot_tar
+35.0s 2015-09-18T20:39:02Z end_write_snapshot_tar
+0.0s 2015-09-18T20:39:02Z pre_exec
+0.3s 2015-09-18T20:39:02Z discovering_tests
+2.3s 2015-09-18T20:39:04Z starting_tests 190 tests
+0.1s 2015-09-18T20:39:04Z start_tests scaleway-prod-27: [go_test:archive/tar go_test:archive/zip go_test:bufio go_test:bytes go_test:compress/bzip2 go_test:compress/flate go_test:compress/gzip go_test:compress/lzw go_test:compress/zlib go_test:container/heap go_test:container/list go_test:container/ring go_test:crypto/aes go_test:crypto/cipher go_test:crypto/des go_test:crypto/dsa go_test:crypto/ecdsa go_test:crypto/elliptic go_test:crypto/hmac go_test:crypto/md5]
+30.0s 2015-09-18T20:39:34Z still_waiting_on_test go_test:archive/tar
+15.8s 2015-09-18T20:39:50Z end_tests scaleway-prod-27: [go_test:archive/tar go_test:archive/zip go_test:bufio go_test:bytes go_test:compress/bzip2 go_test:compress/flate go_test:compress/gzip go_test:compress/lzw go_test:compress/zlib go_test:container/heap go_test:container/list go_test:container/ring go_test:crypto/aes go_test:crypto/cipher go_test:crypto/des go_test:crypto/dsa go_test:crypto/ecdsa go_test:crypto/elliptic go_test:crypto/hmac go_test:crypto/md5]; ok (test exec = 45.775872368s)
+0.0s 2015-09-18T20:39:50Z start_tests scaleway-prod-27: [go_test:crypto/rand go_test:crypto/rc4 go_test:crypto/rsa go_test:crypto/sha1 go_test:crypto/sha256 go_test:crypto/sha512 go_test:crypto/subtle go_test:crypto/tls go_test:crypto/x509 go_test:database/sql go_test:database/sql/driver go_test:debug/dwarf go_test:debug/elf go_test:debug/gosym]
+30.0s 2015-09-18T20:40:20Z still_waiting_on_test go_test:crypto/rand
+13.7s 2015-09-18T20:40:34Z end_tests scaleway-prod-27: [go_test:crypto/rand go_test:crypto/rc4 go_test:crypto/rsa go_test:crypto/sha1 go_test:crypto/sha256 go_test:crypto/sha512 go_test:crypto/subtle go_test:crypto/tls go_test:crypto/x509 go_test:database/sql go_test:database/sql/driver go_test:debug/dwarf go_test:debug/elf go_test:debug/gosym]; ok (test exec = 43.731389977s)
+0.0s 2015-09-18T20:40:34Z start_tests scaleway-prod-27: [go_test:debug/macho go_test:debug/pe go_test:debug/plan9obj go_test:encoding/ascii85 go_test:encoding/asn1 go_test:encoding/base32 go_test:encoding/base64 go_test:encoding/binary go_test:encoding/csv go_test:encoding/gob go_test:encoding/hex go_test:encoding/json go_test:encoding/pem go_test:encoding/xml go_test:errors go_test:expvar go_test:flag]
+30.0s 2015-09-18T20:41:04Z still_waiting_on_test go_test:debug/macho
+14.5s 2015-09-18T20:41:18Z end_tests scaleway-prod-27: [go_test:debug/macho go_test:debug/pe go_test:debug/plan9obj go_test:encoding/ascii85 go_test:encoding/asn1 go_test:encoding/base32 go_test:encoding/base64 go_test:encoding/binary go_test:encoding/csv go_test:encoding/gob go_test:encoding/hex go_test:encoding/json go_test:encoding/pem go_test:encoding/xml go_test:errors go_test:expvar go_test:flag]; ok (test exec = 44.532480843s)
+0.0s 2015-09-18T20:41:18Z start_tests scaleway-prod-27: [go_test:fmt go_test:go/ast go_test:go/build go_test:go/constant go_test:go/doc go_test:go/format go_test:go/internal/gccgoimporter go_test:go/internal/gcimporter go_test:go/parser go_test:go/printer go_test:go/scanner go_test:go/token]
+30.0s 2015-09-18T20:41:48Z still_waiting_on_test go_test:fmt
+6.3s 2015-09-18T20:41:55Z end_tests scaleway-prod-27: [go_test:fmt go_test:go/ast go_test:go/build go_test:go/constant go_test:go/doc go_test:go/format go_test:go/internal/gccgoimporter go_test:go/internal/gcimporter go_test:go/parser go_test:go/printer go_test:go/scanner go_test:go/token]; ok (test exec = 36.32986443s)
+0.0s 2015-09-18T20:41:55Z start_tests scaleway-prod-27: [go_test:go/types go_test:hash/adler32 go_test:hash/crc32 go_test:hash/crc64 go_test:hash/fnv go_test:html go_test:html/template go_test:image go_test:image/color go_test:image/draw go_test:image/gif go_test:image/jpeg]
+30.0s 2015-09-18T20:42:25Z still_waiting_on_test go_test:go/types
+20.0s 2015-09-18T20:42:45Z end_tests scaleway-prod-27: [go_test:go/types go_test:hash/adler32 go_test:hash/crc32 go_test:hash/crc64 go_test:hash/fnv go_test:html go_test:html/template go_test:image go_test:image/color go_test:image/draw go_test:image/gif go_test:image/jpeg]; ok (test exec = 49.985392699s)
+0.0s 2015-09-18T20:42:45Z start_tests scaleway-prod-27: [go_test:image/png go_test:index/suffixarray go_test:internal/singleflight go_test:internal/trace go_test:io go_test:io/ioutil go_test:log go_test:log/syslog go_test:math go_test:math/big go_test:math/cmplx go_test:math/rand]
+30.0s 2015-09-18T20:43:15Z still_waiting_on_test go_test:image/png
+24.3s 2015-09-18T20:43:39Z end_tests scaleway-prod-27: [go_test:image/png go_test:index/suffixarray go_test:internal/singleflight go_test:internal/trace go_test:io go_test:io/ioutil go_test:log go_test:log/syslog go_test:math go_test:math/big go_test:math/cmplx go_test:math/rand]; ok (test exec = 54.280072872s)
+0.0s 2015-09-18T20:43:39Z start_tests scaleway-prod-27: [go_test:mime go_test:mime/multipart go_test:mime/quotedprintable go_test:net]
+30.0s 2015-09-18T20:44:09Z still_waiting_on_test go_test:mime
+8.8s 2015-09-18T20:44:18Z end_tests scaleway-prod-27: [go_test:mime go_test:mime/multipart go_test:mime/quotedprintable go_test:net]; ok (test exec = 38.8502554s)
+0.0s 2015-09-18T20:44:18Z start_tests scaleway-prod-27: [go_test:net/http]
+30.0s 2015-09-18T20:44:48Z still_waiting_on_test go_test:net/http
+19.0s 2015-09-18T20:45:07Z end_tests scaleway-prod-27: [go_test:net/http]; ok (test exec = 48.968881907s)
+0.0s 2015-09-18T20:45:07Z start_tests scaleway-prod-27: [go_test:net/http/cgi go_test:net/http/cookiejar go_test:net/http/fcgi go_test:net/http/httptest go_test:net/http/httputil go_test:net/http/internal go_test:net/internal/socktest go_test:net/mail go_test:net/rpc go_test:net/rpc/jsonrpc go_test:net/smtp go_test:net/textproto]
+30.0s 2015-09-18T20:45:37Z still_waiting_on_test go_test:net/http/cgi
+6.8s 2015-09-18T20:45:44Z end_tests scaleway-prod-27: [go_test:net/http/cgi go_test:net/http/cookiejar go_test:net/http/fcgi go_test:net/http/httptest go_test:net/http/httputil go_test:net/http/internal go_test:net/internal/socktest go_test:net/mail go_test:net/rpc go_test:net/rpc/jsonrpc go_test:net/smtp go_test:net/textproto]; ok (test exec = 36.772400888s)
+0.0s 2015-09-18T20:45:44Z start_tests scaleway-prod-27: [go_test:net/url go_test:os go_test:os/exec go_test:os/signal go_test:os/user go_test:path go_test:path/filepath]
+22.2s 2015-09-18T20:46:06Z end_tests scaleway-prod-27: [go_test:net/url go_test:os go_test:os/exec go_test:os/signal go_test:os/user go_test:path go_test:path/filepath]; ok (test exec = 22.233468034s)
+0.0s 2015-09-18T20:46:06Z start_tests scaleway-prod-27: [go_test:reflect go_test:regexp go_test:regexp/syntax]
+25.4s 2015-09-18T20:46:31Z end_tests scaleway-prod-27: [go_test:reflect go_test:regexp go_test:regexp/syntax]; ok (test exec = 25.393355415s)
+0.0s 2015-09-18T20:46:31Z start_tests scaleway-prod-27: [go_test:runtime]
+30.0s 2015-09-18T20:47:01Z still_waiting_on_test go_test:runtime
+30.0s 2015-09-18T20:47:31Z still_waiting_on_test go_test:runtime
+30.0s 2015-09-18T20:48:01Z still_waiting_on_test go_test:runtime
+30.0s 2015-09-18T20:48:31Z still_waiting_on_test go_test:runtime
+30.0s 2015-09-18T20:49:01Z still_waiting_on_test go_test:runtime
+7.6s 2015-09-18T20:49:09Z end_tests scaleway-prod-27: [go_test:runtime]; ok (test exec = 2m37.5949148s)
+0.0s 2015-09-18T20:49:09Z start_tests scaleway-prod-27: [go_test:runtime/debug go_test:runtime/pprof go_test:runtime/trace go_test:sort go_test:strconv]
+28.5s 2015-09-18T20:49:37Z end_tests scaleway-prod-27: [go_test:runtime/debug go_test:runtime/pprof go_test:runtime/trace go_test:sort go_test:strconv]; ok (test exec = 28.516480039s)
+0.0s 2015-09-18T20:49:37Z start_tests scaleway-prod-27: [go_test:strings go_test:sync go_test:sync/atomic go_test:syscall go_test:testing go_test:testing/quick go_test:text/scanner go_test:text/tabwriter go_test:text/template go_test:text/template/parse]
+30.0s 2015-09-18T20:50:07Z still_waiting_on_test go_test:strings
+3.9s 2015-09-18T20:50:11Z end_tests scaleway-prod-27: [go_test:strings go_test:sync go_test:sync/atomic go_test:syscall go_test:testing go_test:testing/quick go_test:text/scanner go_test:text/tabwriter go_test:text/template go_test:text/template/parse]; ok (test exec = 33.898438554s)
+0.0s 2015-09-18T20:50:11Z start_tests scaleway-prod-27: [go_test:time go_test:unicode go_test:unicode/utf16 go_test:unicode/utf8 go_test:cmd/addr2line go_test:cmd/api go_test:cmd/asm/internal/asm go_test:cmd/asm/internal/lex]
+23.9s 2015-09-18T20:50:35Z end_tests scaleway-prod-27: [go_test:time go_test:unicode go_test:unicode/utf16 go_test:unicode/utf8 go_test:cmd/addr2line go_test:cmd/api go_test:cmd/asm/internal/asm go_test:cmd/asm/internal/lex]; ok (test exec = 23.895531576s)
+0.0s 2015-09-18T20:50:35Z start_tests scaleway-prod-27: [go_test:cmd/compile/internal/big go_test:cmd/compile/internal/gc go_test:cmd/cover go_test:cmd/doc go_test:cmd/fix]
+30.0s 2015-09-18T20:51:05Z still_waiting_on_test go_test:cmd/compile/internal/big
+27.9s 2015-09-18T20:51:33Z end_tests scaleway-prod-27: [go_test:cmd/compile/internal/big go_test:cmd/compile/internal/gc go_test:cmd/cover go_test:cmd/doc go_test:cmd/fix]; ok (test exec = 57.900938415s)
+0.0s 2015-09-18T20:51:33Z start_tests scaleway-prod-27: [go_test:cmd/go]
+30.0s 2015-09-18T20:52:03Z still_waiting_on_test go_test:cmd/go
+20.0s 2015-09-18T20:52:23Z end_tests scaleway-prod-27: [go_test:cmd/go]; ok (test exec = 50.005802338s)
+0.0s 2015-09-18T20:52:23Z start_tests scaleway-prod-27: [go_test:cmd/gofmt go_test:cmd/internal/goobj go_test:cmd/internal/obj go_test:cmd/internal/obj/x86 go_test:cmd/newlink go_test:cmd/nm go_test:cmd/objdump go_test:cmd/pack go_test:cmd/pprof/internal/profile]
+30.0s 2015-09-18T20:52:53Z still_waiting_on_test go_test:cmd/gofmt
+9.7s 2015-09-18T20:53:03Z end_tests scaleway-prod-27: [go_test:cmd/gofmt go_test:cmd/internal/goobj go_test:cmd/internal/obj go_test:cmd/internal/obj/x86 go_test:cmd/newlink go_test:cmd/nm go_test:cmd/objdump go_test:cmd/pack go_test:cmd/pprof/internal/profile]; ok (test exec = 39.738189045s)
+0.0s 2015-09-18T20:53:03Z start_tests scaleway-prod-27: [go_test:cmd/vendor/golang.org/x/arch/arm/armasm go_test:cmd/vendor/golang.org/x/arch/x86/x86asm go_test:cmd/vet]
+30.0s 2015-09-18T20:53:33Z still_waiting_on_test go_test:cmd/vendor/golang.org/x/arch/arm/armasm
+18.1s 2015-09-18T20:53:51Z end_tests scaleway-prod-27: [go_test:cmd/vendor/golang.org/x/arch/arm/armasm go_test:cmd/vendor/golang.org/x/arch/x86/x86asm go_test:cmd/vet]; ok (test exec = 48.133458153s)
+0.0s 2015-09-18T20:53:51Z start_tests scaleway-prod-27: [runtime:cpu124]
+30.0s 2015-09-18T20:54:21Z still_waiting_on_test runtime:cpu124
+30.0s 2015-09-18T20:54:51Z still_waiting_on_test runtime:cpu124
+30.0s 2015-09-18T20:55:21Z still_waiting_on_test runtime:cpu124
+30.0s 2015-09-18T20:55:51Z still_waiting_on_test runtime:cpu124
+30.0s 2015-09-18T20:56:21Z still_waiting_on_test runtime:cpu124
+30.0s 2015-09-18T20:56:51Z still_waiting_on_test runtime:cpu124
+30.0s 2015-09-18T20:57:21Z still_waiting_on_test runtime:cpu124
+30.0s 2015-09-18T20:57:51Z still_waiting_on_test runtime:cpu124
+30.0s 2015-09-18T20:58:21Z still_waiting_on_test runtime:cpu124
+30.0s 2015-09-18T20:58:51Z still_waiting_on_test runtime:cpu124
+30.0s 2015-09-18T20:59:21Z still_waiting_on_test runtime:cpu124
+30.0s 2015-09-18T20:59:51Z still_waiting_on_test runtime:cpu124
+30.0s 2015-09-18T21:00:21Z still_waiting_on_test runtime:cpu124
+30.0s 2015-09-18T21:00:51Z still_waiting_on_test runtime:cpu124
+23.7s 2015-09-18T21:01:15Z end_tests scaleway-prod-27: [runtime:cpu124]; ok (test exec = 7m23.68955524s)
+0.0s 2015-09-18T21:01:15Z start_tests scaleway-prod-27: [sync_cpu]
+6.2s 2015-09-18T21:01:21Z end_tests scaleway-prod-27: [sync_cpu]; ok (test exec = 6.249336006s)
+0.0s 2015-09-18T21:01:21Z start_tests scaleway-prod-27: [cgo_stdio]
+5.4s 2015-09-18T21:01:26Z end_tests scaleway-prod-27: [cgo_stdio]; ok (test exec = 5.397171485s)
+0.0s 2015-09-18T21:01:26Z start_tests scaleway-prod-27: [cgo_life]
+5.5s 2015-09-18T21:01:32Z end_tests scaleway-prod-27: [cgo_life]; ok (test exec = 5.526510773s)
+0.0s 2015-09-18T21:01:32Z start_tests scaleway-prod-27: [cgo_test]
+30.0s 2015-09-18T21:02:02Z still_waiting_on_test cgo_test
+30.0s 2015-09-18T21:02:32Z still_waiting_on_test cgo_test
+30.0s 2015-09-18T21:03:02Z still_waiting_on_test cgo_test
+30.0s 2015-09-18T21:03:32Z still_waiting_on_test cgo_test
+30.0s 2015-09-18T21:04:02Z still_waiting_on_test cgo_test
+30.0s 2015-09-18T21:04:32Z still_waiting_on_test cgo_test
+30.0s 2015-09-18T21:05:02Z still_waiting_on_test cgo_test
+10.4s 2015-09-18T21:05:12Z end_tests scaleway-prod-27: [cgo_test]; ok (test exec = 3m40.402986764s)
+0.0s 2015-09-18T21:05:12Z start_tests scaleway-prod-27: [testgodefs]
+2.0s 2015-09-18T21:05:14Z end_tests scaleway-prod-27: [testgodefs]; ok (test exec = 2.017993963s)
+0.0s 2015-09-18T21:05:14Z start_tests scaleway-prod-27: [testso]
+4.1s 2015-09-18T21:05:18Z end_tests scaleway-prod-27: [testso]; ok (test exec = 4.115807617s)
+0.0s 2015-09-18T21:05:18Z start_tests scaleway-prod-27: [testsovar]
+5.1s 2015-09-18T21:05:23Z end_tests scaleway-prod-27: [testsovar]; ok (test exec = 5.066516003s)
+0.0s 2015-09-18T21:05:23Z start_tests scaleway-prod-27: [testcshared]
+30.0s 2015-09-18T21:05:53Z still_waiting_on_test testcshared
+13.0s 2015-09-18T21:06:06Z end_tests scaleway-prod-27: [testcshared]; ok (test exec = 43.033063492s)
+0.0s 2015-09-18T21:06:06Z start_tests scaleway-prod-27: [cgo_errors]
+5.3s 2015-09-18T21:06:12Z end_tests scaleway-prod-27: [cgo_errors]; ok (test exec = 5.292943546s)
+0.0s 2015-09-18T21:06:12Z start_tests scaleway-prod-27: [testsigfwd]
+4.5s 2015-09-18T21:06:16Z end_tests scaleway-prod-27: [testsigfwd]; ok (test exec = 4.451898707s)
+0.0s 2015-09-18T21:06:16Z start_tests scaleway-prod-27: [doc_progs]
+30.0s 2015-09-18T21:06:46Z still_waiting_on_test doc_progs
+13.2s 2015-09-18T21:06:59Z end_tests scaleway-prod-27: [doc_progs]; ok (test exec = 43.150266414s)
+0.0s 2015-09-18T21:06:59Z start_tests scaleway-prod-27: [wiki]
+0.2s 2015-09-18T21:06:59Z end_tests scaleway-prod-27: [wiki]; ok (test exec = 214.059923ms)
+0.0s 2015-09-18T21:06:59Z start_tests scaleway-prod-27: [codewalk]
+17.8s 2015-09-18T21:07:17Z end_tests scaleway-prod-27: [codewalk]; ok (test exec = 17.788979076s)
+0.0s 2015-09-18T21:07:17Z start_tests scaleway-prod-27: [shootout:fasta]
+3.3s 2015-09-18T21:07:21Z end_tests scaleway-prod-27: [shootout:fasta]; ok (test exec = 3.347611174s)
+0.0s 2015-09-18T21:07:21Z start_tests scaleway-prod-27: [shootout:revcomp]
+2.5s 2015-09-18T21:07:23Z end_tests scaleway-prod-27: [shootout:revcomp]; ok (test exec = 2.497388664s)
+0.0s 2015-09-18T21:07:23Z start_tests scaleway-prod-27: [shootout:nbody]
+3.6s 2015-09-18T21:07:27Z end_tests scaleway-prod-27: [shootout:nbody]; ok (test exec = 3.625977875s)
+0.0s 2015-09-18T21:07:27Z start_tests scaleway-prod-27: [shootout:binarytree]
+18.4s 2015-09-18T21:07:45Z end_tests scaleway-prod-27: [shootout:binarytree]; ok (test exec = 18.357459221s)
+0.0s 2015-09-18T21:07:45Z start_tests scaleway-prod-27: [shootout:fannkuch]
+6.4s 2015-09-18T21:07:51Z end_tests scaleway-prod-27: [shootout:fannkuch]; ok (test exec = 6.380445602s)
+0.0s 2015-09-18T21:07:51Z start_tests scaleway-prod-27: [shootout:regexdna]
+7.7s 2015-09-18T21:07:59Z end_tests scaleway-prod-27: [shootout:regexdna]; ok (test exec = 7.718835914s)
+0.0s 2015-09-18T21:07:59Z start_tests scaleway-prod-27: [shootout:spectralnorm]
+30.0s 2015-09-18T21:08:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:08:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:09:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:09:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:10:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:10:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:11:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:11:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:12:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:12:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:13:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:13:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:14:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:14:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:15:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:15:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:16:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:16:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:17:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:17:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:18:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:18:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:19:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:19:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:20:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:20:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:21:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:21:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:22:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:22:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:23:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:23:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:24:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:24:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:25:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:25:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:26:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:26:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:27:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:27:59Z still_waiting_on_test shootout:spectralnorm
+0.1s 2015-09-18T21:27:59Z end_tests scaleway-prod-27: [shootout:spectralnorm]; commErr=buildlet: timeout waiting for command to complete (test exec = 20m0.100315581s)
+0.0s 2015-09-18T21:27:59Z start_tests scaleway-prod-27: [shootout:spectralnorm]
+29.9s 2015-09-18T21:28:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:28:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:29:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:29:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:30:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:30:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:31:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:31:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:32:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:32:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:33:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:33:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:34:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:34:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:35:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:35:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:36:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:36:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:37:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:37:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:38:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:38:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:39:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:39:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:40:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:40:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:41:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:41:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:42:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:42:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:43:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:43:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:44:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:44:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:45:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:45:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:46:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:46:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:47:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:47:59Z still_waiting_on_test shootout:spectralnorm
+0.2s 2015-09-18T21:47:59Z end_tests scaleway-prod-27: [shootout:spectralnorm]; commErr=buildlet: timeout waiting for command to complete (test exec = 20m0.100096579s)
+0.0s 2015-09-18T21:47:59Z start_tests scaleway-prod-27: [shootout:spectralnorm]
+29.8s 2015-09-18T21:48:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:48:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:49:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:49:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:50:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:50:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:51:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:51:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:52:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:52:59Z still_waiting_on_test shootout:spectralnorm
+30.1s 2015-09-18T21:53:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:53:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:54:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:54:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:55:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:55:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:56:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:56:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:57:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:57:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:58:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:58:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:59:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T21:59:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T22:00:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T22:00:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T22:01:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T22:01:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T22:02:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T22:02:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T22:03:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T22:03:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T22:04:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T22:04:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T22:05:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T22:05:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T22:06:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T22:06:59Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T22:07:29Z still_waiting_on_test shootout:spectralnorm
+30.0s 2015-09-18T22:07:59Z end_tests scaleway-prod-27: [shootout:spectralnorm]; commErr=buildlet: timeout waiting for command to complete (test exec = 20m0.101402213s)
+0.0s 2015-09-18T22:07:59Z start_tests scaleway-prod-27: [shootout:knucleotide]
+0.0s 2015-09-18T22:07:59Z done with test failures
+0.2s 2015-09-18T22:08:00Z end_tests scaleway-prod-27: [shootout:knucleotide]; commErr=buildlet: Client closed (test exec = 218.971988ms)
+0.0s 2015-09-18T22:08:00Z start_tests scaleway-prod-27: [shootout:knucleotide]
+0.0s 2015-09-18T22:08:00Z end_tests scaleway-prod-27: [shootout:knucleotide]; commErr=buildlet: Client closed (test exec = 52.295µs)
+0.0s 2015-09-18T22:08:00Z start_tests scaleway-prod-27: [shootout:knucleotide]
+0.0s 2015-09-18T22:08:00Z end_tests scaleway-prod-27: [shootout:knucleotide]; commErr=buildlet: Client closed (test exec = 36.51µs)
/cc @adg @davecheney