x/build: iOS builder took 19h to obtain machine

[mknyszek]

There's a trybot run where presumably a test deadlocked, but the run never timed out.

EDIT: It actually passed as I was writing this. I guess it was waiting all that time to grab a builder? Yeah, it's right there in the log.

https://farmer.golang.org/try?commit=e95f1f1c

Screenshot for posterity:

Builder output at the time of writing:

 builder: ios-arm64-corellium
      rev: e95f1f1cedaa8284bb00dec23da292ef3821cd2e
 buildlet: http://iPhone reverse peer iPhone/18.117.67.207:49165 for host type host-ios-arm64-corellium-ios
  started: 2023-02-08 20:57:17.701884025 +0000 UTC m=+404952.505449702
   status: still running
 
Events:
  2023-02-08T20:57:17Z get_buildlet 
  2023-02-09T16:29:45Z finish_get_buildlet after 19h32m27.5s
  2023-02-09T16:29:45Z using_buildlet iPhone
  2023-02-09T16:29:45Z write_version_tar 
  2023-02-09T16:29:46Z get_source go@e95f1f1cedaa8284bb00dec23da292ef3821cd2e
  2023-02-09T16:29:46Z get_source_from_gerrit go-e95f1f1cedaa8284bb00dec23da292ef3821cd2e from gerrit
  2023-02-09T16:29:53Z finish_get_source_from_gerrit after 7.44s; go-e95f1f1cedaa8284bb00dec23da292ef3821cd2e from gerrit
  2023-02-09T16:29:53Z finish_get_source after 7.52s; go@e95f1f1cedaa8284bb00dec23da292ef3821cd2e
  2023-02-09T16:29:53Z write_go_src_tar 
  2023-02-09T16:31:27Z finish_write_go_src_tar after 1m34.1s
  2023-02-09T16:31:27Z make_and_test 
  2023-02-09T16:31:27Z make src/make.bash
  2023-02-09T16:38:51Z finish_make after 7m23.4s; src/make.bash
  2023-02-09T16:38:51Z clean_for_snapshot 
  2023-02-09T16:38:51Z finish_clean_for_snapshot after 33.6ms
  2023-02-09T16:38:51Z write_snapshot_to_gcs 
  2023-02-09T16:38:51Z fetch_snapshot_reader_from_buildlet 
  2023-02-09T16:38:51Z finish_fetch_snapshot_reader_from_buildlet after 91.1ms
  2023-02-09T16:39:07Z finish_write_snapshot_to_gcs after 16s
  2023-02-09T16:39:07Z discovering_tests 
  2023-02-09T16:39:09Z get_test_stats 
  2023-02-09T16:39:09Z finish_get_test_stats after 0s
  2023-02-09T16:39:09Z starting_tests 284 tests
  2023-02-09T16:39:09Z run_tests_multi iPhone: [go_test:archive/tar go_test:archive/zip go_test:bufio]
  2023-02-09T16:39:18Z finish_run_tests_multi after 9s; iPhone: [go_test:archive/tar go_test:archive/zip go_test:bufio]
  2023-02-09T16:39:18Z run_tests_multi iPhone: [go_test:bytes go_test:compress/bzip2 go_test:compress/flate]
  2023-02-09T16:39:27Z finish_run_tests_multi after 8.38s; iPhone: [go_test:bytes go_test:compress/bzip2 go_test:compress/flate]
  2023-02-09T16:39:27Z run_tests_multi iPhone: [go_test:compress/gzip go_test:compress/lzw go_test:compress/zlib]
  2023-02-09T16:39:37Z finish_run_tests_multi after 10.4s; iPhone: [go_test:compress/gzip go_test:compress/lzw go_test:compress/zlib]
  2023-02-09T16:39:37Z run_tests_multi iPhone: [go_test:container/heap go_test:container/list go_test:container/ring]
  2023-02-09T16:39:41Z finish_run_tests_multi after 3.32s; iPhone: [go_test:container/heap go_test:container/list go_test:container/ring]
  2023-02-09T16:39:41Z run_tests_multi iPhone: [go_test:context go_test:crypto go_test:crypto/aes]
  2023-02-09T16:39:47Z finish_run_tests_multi after 5.8s; iPhone: [go_test:context go_test:crypto go_test:crypto/aes]
  2023-02-09T16:39:47Z run_tests_multi iPhone: [go_test:crypto/cipher go_test:crypto/des go_test:crypto/dsa]
  2023-02-09T16:39:50Z finish_run_tests_multi after 3.86s; iPhone: [go_test:crypto/cipher go_test:crypto/des go_test:crypto/dsa]
  2023-02-09T16:39:50Z run_tests_multi iPhone: [go_test:crypto/ecdh go_test:crypto/ecdsa go_test:crypto/ed25519]
  2023-02-09T16:39:55Z finish_run_tests_multi after 4.18s; iPhone: [go_test:crypto/ecdh go_test:crypto/ecdsa go_test:crypto/ed25519]
  2023-02-09T16:39:55Z run_tests_multi iPhone: [go_test:crypto/elliptic go_test:crypto/hmac go_test:crypto/internal/alias]
  2023-02-09T16:40:11Z finish_run_tests_multi after 15.8s; iPhone: [go_test:crypto/elliptic go_test:crypto/hmac go_test:crypto/internal/alias]
  2023-02-09T16:40:11Z run_tests_multi iPhone: [go_test:crypto/internal/bigmod go_test:crypto/internal/boring go_test:crypto/internal/boring/bcache]
  2023-02-09T16:40:14Z finish_run_tests_multi after 3.58s; iPhone: [go_test:crypto/internal/bigmod go_test:crypto/internal/boring go_test:crypto/internal/boring/bcache]
  2023-02-09T16:40:14Z run_tests_multi iPhone: [go_test:crypto/internal/edwards25519 go_test:crypto/internal/edwards25519/field go_test:crypto/internal/nistec]
  2023-02-09T16:40:26Z finish_run_tests_multi after 12.1s; iPhone: [go_test:crypto/internal/edwards25519 go_test:crypto/internal/edwards25519/field go_test:crypto/internal/nistec]
  2023-02-09T16:40:26Z run_tests_multi iPhone: [go_test:crypto/internal/nistec/fiat go_test:crypto/md5 go_test:crypto/rand]
  2023-02-09T16:40:30Z finish_run_tests_multi after 3.17s; iPhone: [go_test:crypto/internal/nistec/fiat go_test:crypto/md5 go_test:crypto/rand]
  2023-02-09T16:40:30Z run_tests_multi iPhone: [go_test:crypto/rc4 go_test:crypto/rsa go_test:crypto/sha1]
  2023-02-09T16:40:34Z finish_run_tests_multi after 3.9s; iPhone: [go_test:crypto/rc4 go_test:crypto/rsa go_test:crypto/sha1]
  2023-02-09T16:40:34Z run_tests_multi iPhone: [go_test:crypto/sha256 go_test:crypto/sha512 go_test:crypto/subtle]
  2023-02-09T16:40:45Z finish_run_tests_multi after 11.3s; iPhone: [go_test:crypto/sha256 go_test:crypto/sha512 go_test:crypto/subtle]
  2023-02-09T16:40:45Z run_tests_multi iPhone: [go_test:crypto/tls go_test:crypto/x509 go_test:database/sql]
  2023-02-09T16:40:59Z finish_run_tests_multi after 13.8s; iPhone: [go_test:crypto/tls go_test:crypto/x509 go_test:database/sql]
  2023-02-09T16:40:59Z run_tests_multi iPhone: [go_test:database/sql/driver go_test:debug/buildinfo go_test:debug/dwarf]
  2023-02-09T16:41:04Z finish_run_tests_multi after 4.84s; iPhone: [go_test:database/sql/driver go_test:debug/buildinfo go_test:debug/dwarf]
  2023-02-09T16:41:04Z run_tests_multi iPhone: [go_test:debug/elf go_test:debug/gosym go_test:debug/macho]
  2023-02-09T16:41:08Z finish_run_tests_multi after 3.74s; iPhone: [go_test:debug/elf go_test:debug/gosym go_test:debug/macho]
  2023-02-09T16:41:08Z run_tests_multi iPhone: [go_test:debug/pe go_test:debug/plan9obj go_test:embed]
  2023-02-09T16:41:12Z finish_run_tests_multi after 4.06s; iPhone: [go_test:debug/pe go_test:debug/plan9obj go_test:embed]
  2023-02-09T16:41:12Z run_tests_multi iPhone: [go_test:embed/internal/embedtest go_test:encoding/ascii85 go_test:encoding/asn1]
  2023-02-09T16:41:25Z finish_run_tests_multi after 12.5s; iPhone: [go_test:embed/internal/embedtest go_test:encoding/ascii85 go_test:encoding/asn1]
  2023-02-09T16:41:25Z run_tests_multi iPhone: [go_test:encoding/base32 go_test:encoding/base64 go_test:encoding/binary]
  2023-02-09T16:41:30Z finish_run_tests_multi after 5.04s; iPhone: [go_test:encoding/base32 go_test:encoding/base64 go_test:encoding/binary]
  2023-02-09T16:41:30Z run_tests_multi iPhone: [go_test:encoding/csv go_test:encoding/gob go_test:encoding/hex]
  2023-02-09T16:41:36Z finish_run_tests_multi after 6.67s; iPhone: [go_test:encoding/csv go_test:encoding/gob go_test:encoding/hex]
  2023-02-09T16:41:36Z run_tests_multi iPhone: [go_test:encoding/json go_test:encoding/pem go_test:encoding/xml]
  2023-02-09T16:41:44Z finish_run_tests_multi after 7.24s; iPhone: [go_test:encoding/json go_test:encoding/pem go_test:encoding/xml]
  2023-02-09T16:41:44Z run_tests_multi iPhone: [go_test:errors go_test:expvar go_test:flag]
  2023-02-09T16:41:49Z finish_run_tests_multi after 4.65s; iPhone: [go_test:errors go_test:expvar go_test:flag]
  2023-02-09T16:41:49Z run_tests_multi iPhone: [go_test:fmt go_test:go/ast go_test:go/build]
  2023-02-09T16:42:06Z finish_run_tests_multi after 17.1s; iPhone: [go_test:fmt go_test:go/ast go_test:go/build]
  2023-02-09T16:42:06Z run_tests_multi iPhone: [go_test:go/build/constraint go_test:go/constant go_test:go/doc]
  2023-02-09T16:42:10Z finish_run_tests_multi after 3.9s; iPhone: [go_test:go/build/constraint go_test:go/constant go_test:go/doc]
  2023-02-09T16:42:10Z run_tests_multi iPhone: [go_test:go/doc/comment go_test:go/format go_test:go/importer]
  2023-02-09T16:42:14Z finish_run_tests_multi after 3.92s; iPhone: [go_test:go/doc/comment go_test:go/format go_test:go/importer]
  2023-02-09T16:42:14Z run_tests_multi iPhone: [go_test:go/internal/gccgoimporter go_test:go/internal/gcimporter go_test:go/internal/srcimporter]
  2023-02-09T16:42:18Z finish_run_tests_multi after 4.06s; iPhone: [go_test:go/internal/gccgoimporter go_test:go/internal/gcimporter go_test:go/internal/srcimporter]
  2023-02-09T16:42:18Z run_tests_multi iPhone: [go_test:go/parser go_test:go/printer go_test:go/scanner]
  2023-02-09T16:42:23Z finish_run_tests_multi after 5.53s; iPhone: [go_test:go/parser go_test:go/printer go_test:go/scanner]
  2023-02-09T16:42:23Z run_tests_multi iPhone: [go_test:go/token go_test:go/types go_test:hash]
  2023-02-09T16:42:45Z finish_run_tests_multi after 22s; iPhone: [go_test:go/token go_test:go/types go_test:hash]
  2023-02-09T16:42:45Z run_tests_multi iPhone: [go_test:hash/adler32 go_test:hash/crc32 go_test:hash/crc64]
  2023-02-09T16:42:49Z finish_run_tests_multi after 3.38s; iPhone: [go_test:hash/adler32 go_test:hash/crc32 go_test:hash/crc64]
  2023-02-09T16:42:49Z run_tests_multi iPhone: [go_test:hash/fnv go_test:hash/maphash go_test:html]
  2023-02-09T16:42:52Z finish_run_tests_multi after 3.12s; iPhone: [go_test:hash/fnv go_test:hash/maphash go_test:html]
  2023-02-09T16:42:52Z run_tests_multi iPhone: [go_test:html/template go_test:image go_test:image/color]
  2023-02-09T16:42:57Z finish_run_tests_multi after 4.83s; iPhone: [go_test:html/template go_test:image go_test:image/color]
  2023-02-09T16:42:57Z run_tests_multi iPhone: [go_test:image/draw go_test:image/gif go_test:image/jpeg]
  2023-02-09T16:43:01Z finish_run_tests_multi after 4.1s; iPhone: [go_test:image/draw go_test:image/gif go_test:image/jpeg]
  2023-02-09T16:43:01Z run_tests_multi iPhone: [go_test:image/png go_test:index/suffixarray go_test:internal/abi]
  2023-02-09T16:43:25Z finish_run_tests_multi after 23.3s; iPhone: [go_test:image/png go_test:index/suffixarray go_test:internal/abi]
  2023-02-09T16:43:25Z run_tests_multi iPhone: [go_test:internal/buildcfg go_test:internal/coverage/cformat go_test:internal/coverage/cmerge]
  2023-02-09T16:43:28Z finish_run_tests_multi after 3.01s; iPhone: [go_test:internal/buildcfg go_test:internal/coverage/cformat go_test:internal/coverage/cmerge]
  2023-02-09T16:43:28Z run_tests_multi iPhone: [go_test:internal/coverage/pods go_test:internal/coverage/slicereader go_test:internal/coverage/slicewriter]
  2023-02-09T16:43:31Z finish_run_tests_multi after 2.92s; iPhone: [go_test:internal/coverage/pods go_test:internal/coverage/slicereader go_test:internal/coverage/slicewriter]
  2023-02-09T16:43:31Z run_tests_multi iPhone: [go_test:internal/coverage/test go_test:internal/cpu go_test:internal/dag]
  2023-02-09T16:43:56Z finish_run_tests_multi after 25.2s; iPhone: [go_test:internal/coverage/test go_test:internal/cpu go_test:internal/dag]
  2023-02-09T16:43:56Z run_tests_multi iPhone: [go_test:internal/diff go_test:internal/fmtsort go_test:internal/fuzz]
  2023-02-09T16:44:02Z finish_run_tests_multi after 5.56s; iPhone: [go_test:internal/diff go_test:internal/fmtsort go_test:internal/fuzz]
  2023-02-09T16:44:02Z run_tests_multi iPhone: [go_test:internal/godebug go_test:internal/intern go_test:internal/itoa]
  2023-02-09T16:44:05Z finish_run_tests_multi after 2.91s; iPhone: [go_test:internal/godebug go_test:internal/intern go_test:internal/itoa]
  2023-02-09T16:44:05Z run_tests_multi iPhone: [go_test:internal/poll go_test:internal/profile go_test:internal/reflectlite]
  2023-02-09T16:44:19Z finish_run_tests_multi after 14.4s; iPhone: [go_test:internal/poll go_test:internal/profile go_test:internal/reflectlite]
  2023-02-09T16:44:19Z run_tests_multi iPhone: [go_test:internal/safefilepath go_test:internal/saferio go_test:internal/singleflight]
  2023-02-09T16:44:22Z finish_run_tests_multi after 3.09s; iPhone: [go_test:internal/safefilepath go_test:internal/saferio go_test:internal/singleflight]
  2023-02-09T16:44:22Z run_tests_multi iPhone: [go_test:internal/testenv go_test:internal/trace go_test:internal/types/errors]
  2023-02-09T16:44:35Z finish_run_tests_multi after 12.7s; iPhone: [go_test:internal/testenv go_test:internal/trace go_test:internal/types/errors]
  2023-02-09T16:44:35Z run_tests_multi iPhone: [go_test:internal/unsafeheader go_test:internal/xcoff go_test:io]
  2023-02-09T16:44:43Z finish_run_tests_multi after 7.86s; iPhone: [go_test:internal/unsafeheader go_test:internal/xcoff go_test:io]
  2023-02-09T16:44:43Z run_tests_multi iPhone: [go_test:io/fs go_test:io/ioutil go_test:log]
  2023-02-09T16:44:46Z finish_run_tests_multi after 3.24s; iPhone: [go_test:io/fs go_test:io/ioutil go_test:log]
  2023-02-09T16:44:46Z run_tests_multi iPhone: [go_test:log/syslog go_test:math go_test:math/big]
  2023-02-09T16:44:59Z finish_run_tests_multi after 12.6s; iPhone: [go_test:log/syslog go_test:math go_test:math/big]
  2023-02-09T16:44:59Z run_tests_multi iPhone: [go_test:math/bits go_test:math/cmplx go_test:math/rand]
  2023-02-09T16:45:29Z still_waiting_on_test go_test:math/bits
  2023-02-09T16:45:34Z finish_run_tests_multi after 34.8s; iPhone: [go_test:math/bits go_test:math/cmplx go_test:math/rand]
  2023-02-09T16:45:34Z run_tests_multi iPhone: [go_test:mime go_test:mime/multipart go_test:mime/quotedprintable]
  2023-02-09T16:45:38Z finish_run_tests_multi after 3.86s; iPhone: [go_test:mime go_test:mime/multipart go_test:mime/quotedprintable]
  2023-02-09T16:45:38Z run_tests_multi iPhone: [go_test:net go_test:net/http go_test:net/http/cgi]
  2023-02-09T16:46:04Z finish_run_tests_multi after 26.5s; iPhone: [go_test:net go_test:net/http go_test:net/http/cgi]
  2023-02-09T16:46:04Z run_tests_multi iPhone: [go_test:net/http/cookiejar go_test:net/http/fcgi go_test:net/http/httptest]
  2023-02-09T16:46:10Z finish_run_tests_multi after 5.29s; iPhone: [go_test:net/http/cookiejar go_test:net/http/fcgi go_test:net/http/httptest]
  2023-02-09T16:46:10Z run_tests_multi iPhone: [go_test:net/http/httptrace go_test:net/http/httputil go_test:net/http/internal]
  2023-02-09T16:46:23Z finish_run_tests_multi after 13s; iPhone: [go_test:net/http/httptrace go_test:net/http/httputil go_test:net/http/internal]
  2023-02-09T16:46:23Z run_tests_multi iPhone: [go_test:net/http/internal/ascii go_test:net/http/pprof go_test:net/internal/socktest]
  2023-02-09T16:46:31Z finish_run_tests_multi after 7.62s; iPhone: [go_test:net/http/internal/ascii go_test:net/http/pprof go_test:net/internal/socktest]
  2023-02-09T16:46:31Z run_tests_multi iPhone: [go_test:net/mail go_test:net/netip go_test:net/rpc]
  2023-02-09T16:46:40Z finish_run_tests_multi after 9.02s; iPhone: [go_test:net/mail go_test:net/netip go_test:net/rpc]
  2023-02-09T16:46:40Z run_tests_multi iPhone: [go_test:net/rpc/jsonrpc go_test:net/smtp go_test:net/textproto]
  2023-02-09T16:46:44Z finish_run_tests_multi after 4.62s; iPhone: [go_test:net/rpc/jsonrpc go_test:net/smtp go_test:net/textproto]
  2023-02-09T16:46:44Z run_tests_multi iPhone: [go_test:net/url go_test:os go_test:os/exec]
  2023-02-09T16:46:57Z finish_run_tests_multi after 12.6s; iPhone: [go_test:net/url go_test:os go_test:os/exec]
  2023-02-09T16:46:57Z run_tests_multi iPhone: [go_test:os/exec/internal/fdtest go_test:os/signal go_test:os/user]
  2023-02-09T16:47:04Z finish_run_tests_multi after 7s; iPhone: [go_test:os/exec/internal/fdtest go_test:os/signal go_test:os/user]
  2023-02-09T16:47:04Z run_tests_multi iPhone: [go_test:path go_test:path/filepath go_test:plugin]
  2023-02-09T16:47:10Z finish_run_tests_multi after 5.42s; iPhone: [go_test:path go_test:path/filepath go_test:plugin]
  2023-02-09T16:47:10Z run_tests_multi iPhone: [go_test:reflect go_test:regexp go_test:regexp/syntax]
  2023-02-09T16:47:22Z finish_run_tests_multi after 11.9s; iPhone: [go_test:reflect go_test:regexp go_test:regexp/syntax]
  2023-02-09T16:47:22Z run_tests_multi iPhone: [go_test:runtime go_test:runtime/cgo go_test:runtime/coverage]
  2023-02-09T16:47:52Z still_waiting_on_test go_test:runtime
  2023-02-09T16:48:06Z finish_run_tests_multi after 44.4s; iPhone: [go_test:runtime go_test:runtime/cgo go_test:runtime/coverage]
  2023-02-09T16:48:06Z run_tests_multi iPhone: [go_test:runtime/debug go_test:runtime/internal/atomic go_test:runtime/internal/math]
  2023-02-09T16:48:10Z finish_run_tests_multi after 3.13s; iPhone: [go_test:runtime/debug go_test:runtime/internal/atomic go_test:runtime/internal/math]
  2023-02-09T16:48:10Z run_tests_multi iPhone: [go_test:runtime/internal/sys go_test:runtime/metrics go_test:runtime/pprof]
  2023-02-09T16:48:18Z finish_run_tests_multi after 8.01s; iPhone: [go_test:runtime/internal/sys go_test:runtime/metrics go_test:runtime/pprof]
  2023-02-09T16:48:18Z run_tests_multi iPhone: [go_test:runtime/trace go_test:sort go_test:strconv]
  2023-02-09T16:48:40Z finish_run_tests_multi after 22s; iPhone: [go_test:runtime/trace go_test:sort go_test:strconv]
  2023-02-09T16:48:40Z run_tests_multi iPhone: [go_test:strings go_test:sync go_test:sync/atomic]
  2023-02-09T16:48:55Z finish_run_tests_multi after 15.4s; iPhone: [go_test:strings go_test:sync go_test:sync/atomic]
  2023-02-09T16:48:55Z run_tests_multi iPhone: [go_test:syscall go_test:testing go_test:testing/fstest]
  2023-02-09T16:49:13Z finish_run_tests_multi after 17.6s; iPhone: [go_test:syscall go_test:testing go_test:testing/fstest]
  2023-02-09T16:49:13Z run_tests_multi iPhone: [go_test:testing/iotest go_test:testing/quick go_test:text/scanner]
  2023-02-09T16:49:16Z finish_run_tests_multi after 3.52s; iPhone: [go_test:testing/iotest go_test:testing/quick go_test:text/scanner]
  2023-02-09T16:49:16Z run_tests_multi iPhone: [go_test:text/tabwriter go_test:text/template go_test:text/template/parse]
  2023-02-09T16:49:20Z finish_run_tests_multi after 3.86s; iPhone: [go_test:text/tabwriter go_test:text/template go_test:text/template/parse]
  2023-02-09T16:49:20Z run_tests_multi iPhone: [go_test:time go_test:unicode go_test:unicode/utf16]
  2023-02-09T16:49:36Z finish_run_tests_multi after 15.2s; iPhone: [go_test:time go_test:unicode go_test:unicode/utf16]
  2023-02-09T16:49:36Z run_tests_multi iPhone: [go_test:unicode/utf8 go_test:cmd/addr2line go_test:cmd/api]
  2023-02-09T16:49:50Z finish_run_tests_multi after 14s; iPhone: [go_test:unicode/utf8 go_test:cmd/addr2line go_test:cmd/api]
  2023-02-09T16:49:50Z run_tests_multi iPhone: [go_test:cmd/asm/internal/asm go_test:cmd/asm/internal/lex go_test:cmd/compile/internal/abt]
  2023-02-09T16:49:58Z finish_run_tests_multi after 8.28s; iPhone: [go_test:cmd/asm/internal/asm go_test:cmd/asm/internal/lex go_test:cmd/compile/internal/abt]
  2023-02-09T16:49:58Z run_tests_multi iPhone: [go_test:cmd/compile/internal/amd64 go_test:cmd/compile/internal/base go_test:cmd/compile/internal/compare]
  2023-02-09T16:50:28Z still_waiting_on_test go_test:cmd/compile/internal/amd64
  2023-02-09T16:50:54Z finish_run_tests_multi after 55.5s; iPhone: [go_test:cmd/compile/internal/amd64 go_test:cmd/compile/internal/base go_test:cmd/compile/internal/compare]
  2023-02-09T16:50:54Z run_tests_multi iPhone: [go_test:cmd/compile/internal/dwarfgen go_test:cmd/compile/internal/importer go_test:cmd/compile/internal/ir]
  2023-02-09T16:51:02Z finish_run_tests_multi after 8.41s; iPhone: [go_test:cmd/compile/internal/dwarfgen go_test:cmd/compile/internal/importer go_test:cmd/compile/internal/ir]
  2023-02-09T16:51:02Z run_tests_multi iPhone: [go_test:cmd/compile/internal/logopt go_test:cmd/compile/internal/noder go_test:cmd/compile/internal/reflectdata]
  2023-02-09T16:51:09Z finish_run_tests_multi after 6.65s; iPhone: [go_test:cmd/compile/internal/logopt go_test:cmd/compile/internal/noder go_test:cmd/compile/internal/reflectdata]
  2023-02-09T16:51:09Z run_tests_multi iPhone: [go_test:cmd/compile/internal/ssa go_test:cmd/compile/internal/syntax go_test:cmd/compile/internal/test]
  2023-02-09T16:51:39Z still_waiting_on_test go_test:cmd/compile/internal/ssa
  2023-02-09T16:52:09Z still_waiting_on_test go_test:cmd/compile/internal/ssa
  2023-02-09T16:52:09Z finish_run_tests_multi after 1m0.1s; iPhone: [go_test:cmd/compile/internal/ssa go_test:cmd/compile/internal/syntax go_test:cmd/compile/internal/test]
  2023-02-09T16:52:09Z run_tests_multi iPhone: [go_test:cmd/compile/internal/typecheck go_test:cmd/compile/internal/types go_test:cmd/compile/internal/types2]
  2023-02-09T16:52:18Z finish_run_tests_multi after 8.72s; iPhone: [go_test:cmd/compile/internal/typecheck go_test:cmd/compile/internal/types go_test:cmd/compile/internal/types2]
  2023-02-09T16:52:18Z run_tests_multi iPhone: [go_test:cmd/covdata go_test:cmd/cover go_test:cmd/dist]
  2023-02-09T16:52:23Z finish_run_tests_multi after 4.84s; iPhone: [go_test:cmd/covdata go_test:cmd/cover go_test:cmd/dist]
  2023-02-09T16:52:23Z run_tests_multi iPhone: [go_test:cmd/doc go_test:cmd/fix go_test:cmd/go]
  +18.0s (now)
 
Build log:
ios-arm64-corellium at e95f1f1cedaa8284bb00dec23da292ef3821cd2e
 
:: Running /bin/bash with args ["bash" "/tmp/workdir-host-ios-arm64-corellium-ios/go/src/make.bash"] and env ["SHELL=/bin/sh" "XPC_FLAGS=0x0" "DYLD_INSERT_LIBRARIES=/usr/lib/substitute-loader.dylib:/usr/lib/pspawn_payload.dylib" "TMUX=/tmp//tmux-0/default,146,0" "HOME=/var/root" "GO_BUILDER_ENV=host-ios-arm64-corellium-ios" "SANDBOX_TOKENS=d645db5214aa168937a6888d08ecabcd2a7d1509de72f786d9619f2e827b83dc;00;00000000;00000000;00000000;000000000000001c;com.apple.sandbox.executable;01;01000008;0fffffff000054f4;01;/Library:0dc9f79cedd4f7e121f987c553d10e3738d842bfb5e1f867fde574a0c577e048;00;00000000;00000000;00000000;000000000000001c;com.apple.sandbox.executable;01;01000008;0fffffff00061f8c;01;/Library/MobileSubstrate/DynamicLibraries:3e14ef635e1522621d5f9bf714431ad6994936324624e958ef84da375046abe9;00;00000000;00000000;00000000;000000000000001c;com.apple.sandbox.executable;01;01000008;0fffffff0000662c;01;/System:448e2fc6607dc4685fb6cc4acd73bc51d105681fa3ac35c069914a6c463a1e85;00;00000000;00000000;00000000;000000000000001c;com.apple.sandbox.executable;00;01000002;0000000000000002;01;/private/var/mnt:ab7aefd74c363b0271b03a4b5a068680fb45bebaace725e67b9073167e8e7c17;00;00000000;00000000;00000000;000000000000001c;com.apple.sandbox.executable;01;01000002;000000000000002e;01;/private/var/mobile/Library:089943121ca3a56eb08814eca0d66a780af0ed82cabf4212cd9764b266485abd;00;00000000;00000000;00000000;0000000000000020;com.apple.app-sandbox.read-write;01;01000002;0000000000000036;01;/private/var/mobile/Library/Preferences" "TERM=screen" "TMUX_PANE=%0" "SHLVL=1" "XPC_SERVICE_NAME=org.golang.builder" "PATH=/var/root/bin:/usr/bin:/bin:/usr/sbin:/sbin" "CC=/var/root/bin/clangwrap" "_=/var/root/go/bin/buildlet" "WORKDIR=/tmp/workdir-host-ios-arm64-corellium-ios" "GO_BUILDER_NAME=ios-arm64-corellium" "GOROOT_BOOTSTRAP=/var/root/go-ios-arm64-bootstrap" "GOBIN=" "TMPDIR=/tmp/workdir-host-ios-arm64-corellium-ios/tmp" "GOCACHE=/tmp/workdir-host-ios-arm64-corellium-ios/gocache" "PWD=/tmp/workdir-host-ios-arm64-corellium-ios/go/src"] in dir /tmp/workdir-host-ios-arm64-corellium-ios/go/src
 
Building Go cmd/dist using /var/root/go-ios-arm64-bootstrap. (go1.19 ios/arm64)
Building Go toolchain1 using /var/root/go-ios-arm64-bootstrap.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
warning: unable to find runtime/cgo.a
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for ios/arm64.
---
Installed Go for ios/arm64 in /tmp/workdir-host-ios-arm64-corellium-ios/go
Installed commands in /tmp/workdir-host-ios-arm64-corellium-ios/go/bin
 
##### Test execution environment.
# GOARCH: arm64
# CPU: 
# GOOS: ios
# OS Version: Darwin 20.6.0 Darwin Kernel Version 20.6.0: Tue Aug 24 21:10:43 PDT 2021; root:xnu-7195.140.44~1/RELEASE_ARM64_T8010 iPhone9,1
 
##### Testing packages.
ok  	archive/tar	0.212s
ok  	archive/zip	0.350s
ok  	bufio	0.057s
ok  	bytes	2.353s
ok  	compress/bzip2	0.066s
ok  	compress/flate	0.692s
ok  	compress/gzip	2.391s
ok  	compress/lzw	0.080s
ok  	compress/zlib	0.357s
ok  	container/heap	0.024s
ok  	container/list	0.022s
ok  	container/ring	0.021s
ok  	context	0.156s
ok  	crypto	0.023s
ok  	crypto/aes	0.036s
ok  	crypto/cipher	0.024s
ok  	crypto/des	0.025s
ok  	crypto/dsa	0.022s
ok  	crypto/ecdh	0.075s
ok  	crypto/ecdsa	0.088s
ok  	crypto/ed25519	0.132s
ok  	crypto/elliptic	0.034s
ok  	crypto/hmac	0.028s
ok  	crypto/internal/alias	0.020s
ok  	crypto/internal/bigmod	0.204s
ok  	crypto/internal/boring	0.020s
ok  	crypto/internal/boring/bcache	0.151s
ok  	crypto/internal/edwards25519	7.406s
ok  	crypto/internal/edwards25519/field	7.296s
ok  	crypto/internal/nistec	0.821s
ok  	crypto/internal/nistec/fiat	0.021s [no tests to run]
ok  	crypto/md5	0.026s
ok  	crypto/rand	0.056s
ok  	crypto/rc4	0.062s
ok  	crypto/rsa	0.198s
ok  	crypto/sha1	0.022s
ok  	crypto/sha256	0.025s
ok  	crypto/sha512	0.026s
ok  	crypto/subtle	0.214s
ok  	crypto/tls	0.671s
ok  	crypto/x509	0.800s
ok  	database/sql	1.003s
ok  	database/sql/driver	0.020s
ok  	debug/buildinfo	0.027s
ok  	debug/dwarf	0.043s
ok  	debug/elf	0.223s
ok  	debug/gosym	0.028s
ok  	debug/macho	0.025s
ok  	debug/pe	0.042s
ok  	debug/plan9obj	0.024s
ok  	embed	0.028s [no tests to run]
ok  	embed/internal/embedtest	0.027s
ok  	encoding/ascii85	0.024s
ok  	encoding/asn1	0.022s
ok  	encoding/base32	0.035s
ok  	encoding/base64	0.026s
ok  	encoding/binary	0.022s
ok  	encoding/csv	0.028s
ok  	encoding/gob	2.395s
ok  	encoding/hex	0.032s
ok  	encoding/json	0.246s
ok  	encoding/pem	0.459s
ok  	encoding/xml	0.057s
ok  	errors	0.025s
ok  	expvar	0.043s
ok  	flag	0.023s
ok  	fmt	0.103s
ok  	go/ast	0.034s
ok  	go/build	0.033s
ok  	go/build/constraint	0.024s
ok  	go/constant	0.034s
ok  	go/doc	0.080s
ok  	go/doc/comment	0.522s
ok  	go/format	0.029s
ok  	go/importer	0.023s
ok  	go/internal/gccgoimporter	0.026s
ok  	go/internal/gcimporter	0.028s
ok  	go/internal/srcimporter	0.023s
ok  	go/parser	0.263s
ok  	go/printer	0.195s
ok  	go/scanner	0.021s
ok  	go/token	0.041s
ok  	go/types	0.255s
ok  	hash	0.028s
ok  	hash/adler32	0.027s
ok  	hash/crc32	0.026s
ok  	hash/crc64	0.020s
ok  	hash/fnv	0.021s
ok  	hash/maphash	0.094s
ok  	html	0.021s
ok  	html/template	0.093s
ok  	image	0.067s
ok  	image/color	0.033s
ok  	image/draw	0.095s
ok  	image/gif	0.203s
ok  	image/jpeg	0.187s
ok  	image/png	0.372s
ok  	index/suffixarray	0.217s
ok  	internal/abi	0.020s
ok  	internal/buildcfg	0.024s
ok  	internal/coverage/cformat	0.023s
ok  	internal/coverage/cmerge	0.021s
ok  	internal/coverage/pods	0.027s
ok  	internal/coverage/slicereader	0.023s
ok  	internal/coverage/slicewriter	0.021s
ok  	internal/coverage/test	0.030s
ok  	internal/cpu	0.023s
ok  	internal/dag	0.024s
ok  	internal/diff	0.024s
ok  	internal/fmtsort	0.024s
ok  	internal/fuzz	0.026s
ok  	internal/godebug	0.022s
ok  	internal/intern	0.218s
ok  	internal/itoa	0.022s
ok  	internal/poll	0.075s
ok  	internal/profile	0.027s
ok  	internal/reflectlite	0.067s
ok  	internal/safefilepath	0.021s
ok  	internal/saferio	0.053s
ok  	internal/singleflight	0.117s
ok  	internal/testenv	0.023s
ok  	internal/trace	0.051s
ok  	internal/types/errors	0.027s
ok  	internal/unsafeheader	0.021s
ok  	internal/xcoff	0.033s
ok  	io	0.245s
ok  	io/fs	0.391s
ok  	io/ioutil	0.024s
ok  	log	0.022s
ok  	log/syslog	1.930s
ok  	math	0.027s
ok  	math/big	1.595s
ok  	math/bits	0.025s
ok  	math/cmplx	0.025s
ok  	math/rand	0.210s
ok  	mime	0.026s
ok  	mime/multipart	0.159s
ok  	mime/quotedprintable	0.025s
ok  	net	6.983s
ok  	net/http	9.793s
ok  	net/http/cgi	0.107s
ok  	net/http/cookiejar	0.039s
ok  	net/http/fcgi	0.218s
ok  	net/http/httptest	0.089s
ok  	net/http/httptrace	0.032s
ok  	net/http/httputil	0.687s
ok  	net/http/internal	0.021s
ok  	net/http/internal/ascii	0.026s
ok  	net/http/pprof	4.184s
ok  	net/internal/socktest	0.020s
ok  	net/mail	0.025s
ok  	net/netip	0.046s
ok  	net/rpc	0.136s
ok  	net/rpc/jsonrpc	0.037s
ok  	net/smtp	0.052s
ok  	net/textproto	0.024s
ok  	net/url	0.029s
ok  	os	0.595s
ok  	os/exec	0.043s
ok  	os/exec/internal/fdtest	0.030s
ok  	os/signal	4.102s
ok  	os/user	0.021s
ok  	path	0.025s
ok  	path/filepath	0.061s
ok  	plugin	0.022s
ok  	reflect	0.405s
ok  	regexp	0.480s
ok  	regexp/syntax	0.747s
ok  	runtime	8.922s
ok  	runtime/cgo	0.027s
ok  	runtime/coverage	0.021s
ok  	runtime/debug	0.045s
ok  	runtime/internal/atomic	0.046s
ok  	runtime/internal/math	0.020s
ok  	runtime/internal/sys	0.022s
ok  	runtime/metrics	0.023s
ok  	runtime/pprof	4.324s
ok  	runtime/trace	1.836s
ok  	sort	0.071s
ok  	strconv	0.368s
ok  	strings	1.804s
ok  	sync	0.359s
ok  	sync/atomic	0.486s
ok  	syscall	0.419s
ok  	testing	0.689s
ok  	testing/fstest	0.023s
ok  	testing/iotest	0.023s
ok  	testing/quick	0.047s
ok  	text/scanner	0.022s
ok  	text/tabwriter	0.023s
ok  	text/template	0.046s
ok  	text/template/parse	0.025s
ok  	time	6.874s
ok  	unicode	0.026s
ok  	unicode/utf16	0.021s
ok  	unicode/utf8	0.025s
ok  	cmd/addr2line	0.023s
 
ok  	cmd/api	0.022s
ok  	cmd/asm/internal/asm	0.495s
ok  	cmd/asm/internal/lex	0.025s
ok  	cmd/compile/internal/abt	0.026s
ok  	cmd/compile/internal/amd64	0.034s
ok  	cmd/compile/internal/base	0.028s
ok  	cmd/compile/internal/compare	0.024s
ok  	cmd/compile/internal/dwarfgen	0.031s
ok  	cmd/compile/internal/importer	0.028s
ok  	cmd/compile/internal/ir	0.024s
ok  	cmd/compile/internal/logopt	0.022s
ok  	cmd/compile/internal/noder	0.037s
ok  	cmd/compile/internal/reflectdata	0.031s [no tests to run]
ok  	cmd/compile/internal/ssa	0.149s
ok  	cmd/compile/internal/syntax	0.047s
ok  	cmd/compile/internal/test	0.082s
ok  	cmd/compile/internal/typecheck	0.027s
ok  	cmd/compile/internal/types	0.025s
ok  	cmd/compile/internal/types2	0.176s
ok  	cmd/covdata	0.026s
ok  	cmd/cover	0.027s
ok  	cmd/dist	0.021s
ok  	cmd/doc	0.026s
ok  	cmd/fix	14.964s
ok  	cmd/go	0.062s
ok  	cmd/go/internal/auth	0.033s
ok  	cmd/go/internal/cache	0.356s
ok  	cmd/go/internal/fsys	0.082s
ok  	cmd/go/internal/generate	0.041s
ok  	cmd/go/internal/get	0.040s
ok  	cmd/go/internal/imports	0.022s
ok  	cmd/go/internal/load	0.038s
ok  	cmd/go/internal/lockedfile	0.208s
ok  	cmd/go/internal/lockedfile/internal/filelock	0.208s
ok  	cmd/go/internal/modconv	0.030s
ok  	cmd/go/internal/modfetch	0.043s
ok  	cmd/go/internal/modfetch/codehost	0.038s
ok  	cmd/go/internal/modfetch/zip_sum_test	0.038s
ok  	cmd/go/internal/modindex	0.322s
ok  	cmd/go/internal/modload	0.040s
ok  	cmd/go/internal/mvs	0.031s
ok  	cmd/go/internal/par	0.051s
ok  	cmd/go/internal/str	0.020s
ok  	cmd/go/internal/test	0.041s
ok  	cmd/go/internal/vcs	0.043s
ok  	cmd/go/internal/vcweb	0.040s
ok  	cmd/go/internal/vcweb/vcstest	5.680s
ok  	cmd/go/internal/web	0.035s
ok  	cmd/go/internal/work	0.042s
ok  	cmd/gofmt	0.065s
ok  	cmd/internal/archive	0.023s
ok  	cmd/internal/buildid	0.209s
ok  	cmd/internal/dwarf	0.024s
ok  	cmd/internal/edit	0.023s
ok  	cmd/internal/goobj	0.021s
ok  	cmd/internal/moddeps	0.024s
ok  	cmd/internal/notsha256	0.024s
ok  	cmd/internal/obj	0.021s
ok  	cmd/internal/obj/arm64	0.024s
ok  	cmd/internal/obj/ppc64	0.022s
ok  	cmd/internal/obj/riscv	0.023s
ok  	cmd/internal/obj/s390x	0.021s
ok  	cmd/internal/obj/x86	0.026s
ok  	cmd/internal/objabi	0.024s
ok  	cmd/internal/pkgpath	0.023s
ok  	cmd/internal/pkgpattern	0.025s
ok  	cmd/internal/quoted	0.021s
ok  	cmd/internal/src	0.024s
ok  	cmd/internal/test2json	0.135s
ok  	cmd/link	0.027s
ok  	cmd/link/internal/benchmark	0.021s
ok  	cmd/link/internal/ld	0.037s
ok  	cmd/link/internal/loader	0.034s
ok  	cmd/nm	0.035s

[mknyszek]

OK, I stand corrected. It didn't deadlock, I guess, just ran really slowly, because the bot just passed. I guess most of that 19h was waiting?

[changkun]

Unfortunately, afak, iOS builders are either limited by the Corellium infra (killed due to hitting resources) or stuck on some build tasks without releasing the machine. (I am now even reserving a regular calendar event to remind me to reboot those builders manually)

[thanm]

hi again, looks like the ios-arm64-corellium is again wedged or having issues of some sort. The currently running job is on hour 11 of make.bash (https://farmer.golang.org/temporarylogs?name=ios-arm64-corellium&rev=541ef422e24c8010f95151f241224fa16cdce210&st=0xc0557268c0). Mentioning this since I am trying to create a gomote so as to resolve a problem with my most recent linker CL. Thanks.

[bcmills]

@changkun, it looks like the builders are wedged again. Could you restart them?

And is there some way we could set up an ios builder with more reliable hardware? Maybe a reverse builder connected to a physical iOS device?

@mknyszek, @heschi: is there any possibility that we could get the new LUCI infrastructure working for ios? That would be an immediate improvement over the status quo, since we currently don't have any ios builders with shorter-than-overnight turnaround times.

[thanm]

A while back I checked in CL http://go.dev/cl/480675 with a change to the clang wrapper used on the arm corellium builders. For this fix to take effect, someone needs to rebuild the wrapper from source and install it on the builders. Instructions for this appear to be here:

https://go.googlesource.com/build/+/208db0ba5f4e4fdfdc462e9c0af3f0e0999f6d45/env/corellium/ios/install.sh#47

@changkun if you are going to restart the builders, could you please also update the wrappers? I don't think I have the right tools/machines to do the update. Thanks.

[heschi]

@bcmills LUCI supports iOS, but figuring out where the resources are coming from is a bit of a different question. I hope to start answering that soon, but I suspect it'll be a slowish process :(

[changkun]

@bcmills I reconfigured three builders, and now they are back again.

@thanm The clangwrap is now rebuilt based on the tip of the build repository.

And is there some way we could set up an ios builder with more reliable hardware?

I would love to configure a physical device but don't have the budget to buy more.

[thanm]

The clangwrap is now rebuilt based on the tip of the build repository

Thank you!

[thanm]

I think the ios builders are wedged again. Last successful post-submit build was yesterday at 2023-04-25 13:15:38 +0000 UTC link.

Slowbot run on my CL looks hung (running now for 23 hours): link.

Is there anything that can be done here? Reboot/kick builders? Thanks.

[changkun]

I just rebooted the builders again.

[thanm]

Thanks