x/build: difference between post-submit builder and gomote makes repros less likely #42379
What version of Go are you using (
The text was updated successfully, but these errors were encountered:
One other thing I've noticed which may be related is that it seems like the openbsd-amd64-62 gomote is significantly slower than the openbsd-amd64-62 builder.
e.g., compare this post-submit run to my gomote run:
##### Testing packages. ok archive/tar 0.132s ok archive/zip 0.849s ok bufio 0.342s ok bytes 2.026s ok compress/bzip2 0.112s ok compress/flate 4.316s ok compress/gzip 0.657s ok compress/lzw 0.244s ok compress/zlib 1.988s ok container/heap 0.023s ok container/list 0.017s ok container/ring 0.016s ok context 0.153s ok crypto 0.023s ok crypto/aes 0.078s ok crypto/cipher 0.038s ok crypto/des 0.044s ok crypto/dsa 0.031s ok crypto/ecdsa 0.346s ok crypto/ed25519 0.111s ok crypto/elliptic 0.079s ok crypto/hmac 0.024s ok crypto/internal/subtle 0.020s ok crypto/md5 0.020s ok crypto/rand 0.127s ok crypto/rc4 0.055s ok crypto/rsa 0.233s ok crypto/sha1 0.015s ok crypto/sha256 0.016s ok crypto/sha512 0.024s ok crypto/subtle 0.022s ok crypto/tls 2.721s ok crypto/x509 1.747s ok database/sql 1.046s ok database/sql/driver 0.027s ok debug/dwarf 0.045s ok debug/elf 0.066s ok debug/gosym 0.057s ok debug/macho 0.027s ok debug/pe 0.037s ok debug/plan9obj 0.028s ok embed/internal/embedtest 0.033s ok encoding/ascii85 0.058s ok encoding/asn1 0.033s ok encoding/base32 0.130s ok encoding/base64 0.029s ok encoding/binary 0.038s ok encoding/csv 0.029s ok encoding/gob 0.067s ok encoding/hex 0.084s ok encoding/json 0.687s ok encoding/pem 0.048s ok encoding/xml 0.111s ok errors 0.055s ok expvar 0.043s ok flag 0.124s ok fmt 0.171s ok go/ast 0.045s ok go/build 2.028s ok go/constant 0.020s ok go/doc 0.233s ok go/format 0.039s ok go/importer 0.516s ok go/internal/gccgoimporter 0.052s ok go/internal/gcimporter 10.296s ok go/internal/srcimporter 9.709s ok go/parser 0.071s ok go/printer 0.698s ok go/scanner 0.035s ok go/token 0.059s ok go/types 4.011s ok hash 0.019s ok hash/adler32 0.040s ok hash/crc32 0.021s ok hash/crc64 0.022s ok hash/fnv 0.121s ok hash/maphash 0.254s ok html 0.025s ok html/template 0.481s ok image 0.192s ok image/color 0.042s ok image/draw 0.193s ok image/gif 3.448s ok image/jpeg 0.388s ok image/png 0.177s ok index/suffixarray 0.325s ok internal/cpu 0.040s ok internal/fmtsort 0.030s ok internal/poll 0.132s ok internal/profile 0.015s ok internal/reflectlite 0.249s ok internal/singleflight 0.028s ok internal/trace 0.207s ok internal/unsafeheader 0.018s ok internal/xcoff 0.038s ok io 0.359s ok io/fs 0.025s ok io/ioutil 0.030s ok log 0.040s ok log/syslog 1.411s ok math 0.052s ok math/big 2.772s ok math/bits 0.034s ok math/cmplx 0.020s ok math/rand 0.566s ok mime 0.055s ok mime/multipart 0.411s ok mime/quotedprintable 0.057s ok net 29.665s ok net/http 23.805s ok net/http/cgi 1.256s ok net/http/cookiejar 0.112s ok net/http/fcgi 0.040s ok net/http/httptest 0.132s ok net/http/httptrace 0.028s ok net/http/httputil 0.195s ok net/http/internal 0.020s ok net/http/pprof 4.234s ok net/internal/socktest 0.015s ok net/mail 0.029s ok net/rpc 0.068s ok net/rpc/jsonrpc 0.031s ok net/smtp 0.074s ok net/textproto 0.044s ok net/url 0.046s ok os 1.398s ok os/exec 1.937s ok os/signal 2.588s ok os/user 0.026s ok path 0.016s ok path/filepath 0.133s ok plugin 0.019s ok reflect 2.786s ok regexp 0.226s ok regexp/syntax 0.577s
Almost every big test takes 50-100%+ longer than the post-submit version.
This is a very empirical observation, and I haven't seen the post-submit results from a passing run (since the logs are immediately deleted), so perhaps there is nothing there, but this makes me wonder if the gomote is on a smaller instance or something.
Edit: this could also be cause the builder is running fewer tests? -short?
Thanks for reporting. Also CC @golang/release for awareness.
One of the most important differences that comes to mind is that when you have a gomote session running all.bash, all execution happens sequentially. The post-submit builder differs in that it's configured to shard test execution across multiple builder instances of that type:
$ goexec 'dashboard.Builders["openbsd-amd64-62"].SplitMakeRun()'
Which should explain the timing difference you've observed above.
I've only had a chance to look briefly so far, I'll see if I can look more later on.
I've looked up the logs for some passing post-submit builder runs, for reference:
They all look similar, like so:
I've investigated this and I believe I'm able to reproduce the problem of #42237 on the
I limited my scope to the
In other words, the
Here's a relevant bit from an event log of a recent failed build (full log here):
Notably, it tells me the
This gave me the idea to try reproducing the failure by doing just
Intermittent Slow Tests
Running just those tests individually didn't reproduce the problem:
So it seems there's some sort of interaction between all the runtime tests when run one after another.
As far as I can tell, the root cause here turned out to be that the problem was reproducible only when running