runtime/trace: trace tests are flaky

[mdempsky]

openbsd/amd64 failures:

--- FAIL: TestTraceFutileWakeup (0.08s)
    trace_test.go:428: failed to parse trace: g 78 is not waiting during syscall exit (offset 323, time 0)
--- FAIL: TestTraceFutileWakeup (0.13s)
    trace_test.go:428: failed to parse trace: g 82 is not waiting during syscall exit (offset 4636, time 0)
--- FAIL: TestTraceFutileWakeup (0.13s)
    trace_test.go:428: failed to parse trace: g 82 is not waiting during syscall exit (offset 1271, time 0)

linux/ppc64le failures:

--- FAIL: TestTraceFutileWakeup (0.07s)
    trace_test.go:441: goroutine 65 blocked on GoBlockSend at 7278802 right after start
--- FAIL: TestTraceFutileWakeup (0.06s)
    trace_test.go:427: failed to parse trace: g 118 is not running while GoSched (offset 5452, time 1881495)
--- FAIL: TestTraceFutileWakeup (0.07s)
    trace_test.go:441: goroutine 99 blocked on GoBlockSend at 3400181 right after start
--- FAIL: TestTraceFutileWakeup (0.10s)
    trace_test.go:440: goroutine 141 blocked on GoBlockSelect at 1140998 right after start
--- FAIL: TestTraceFutileWakeup (0.08s)
    trace_test.go:427: failed to parse trace: g 133 is not running while unpark (offset 16052, time 10507482)
--- FAIL: TestTraceFutileWakeup (0.09s)
    trace_test.go:428: failed to parse trace: g 99 is not running while GoSched (offset 36466, time 12834938)
--- FAIL: TestTraceFutileWakeup (0.08s)
    trace_test.go:441: goroutine 116 blocked on GoBlockSelect at 1975223 right after start

I haven't noticed any failures on other bots.

[mdempsky]

It affects openbsd/386 too: http://build.golang.org/log/b4604499df4e06974962fb73cd2e6afc071af0bb

[mdempsky]

For completeness, here are links to the openbsd/amd64 build logs:

http://build.golang.org/log/1ae14f4af093f30a7af8f20f31b5de7beb010c6e
http://build.golang.org/log/47fc8d7454e58f8d7f7125bd16792baf4eecb0fb
http://build.golang.org/log/889f78e5ec08712968c9b4b7cf1a7ac7698c038d

And for linux/ppc64le:

http://build.golang.org/log/08c8884c404f2196fe287cd26514acbbd322aac7
http://build.golang.org/log/1dd2acd60400366fb3e2062f8fbfce03b030f0b2
http://build.golang.org/log/41e708d5945b3480822388e6ad7834efe468c200
http://build.golang.org/log/47739c988c55b8b65c393949bb7ff4bba799734c
http://build.golang.org/log/61bea79ce160a202b1db2cf4b1d4d79b29272cca
http://build.golang.org/log/632629ec7c597d73beba2af7562c2dfaf0d31981
http://build.golang.org/log/9c25c42ebc016bfbe6b2e3a83d28c3162c05422e

[dvyukov]

Do we have access to linux/ppc64le machine?

[bradfitz]

Also failed once on nacl-amd64p32 at ec87dbf:

http://build.golang.org/log/124733943107cda60b12f6742d08ca233e206e14

[alexbrainman]

windows-amd64 failure http://build.golang.org/log/5667475ee0041dcaf3b19714360caa9a9c448173

[rjammala]

Have seen this on linux/amd64 also:

ok runtime/debug 0.006s
--- FAIL: TestTraceFutileWakeup (0.04s)
trace_test.go:445: goroutine 153 blocked on GoBlockRecv at 3249129 right after start

[rjammalamadaka@localhost src]$ uname -a
Linux localhost.localdomain 2.6.32-504.12.2.el6.x86_64 #1 SMP Wed Mar 11 22:03:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[rjammalamadaka@localhost src]$ go version
go version devel +7e26a2d Fri May 15 13:56:40 2015 +0000 linux/amd64

[josharian]

Showed up again as of Jun 9. http://build.golang.org/log/92deee43c1bd04e6be337b307a0b98a0ee818ed1

[bradfitz]

Also flaky on Linux on go1.5beta2, per report on golang-nuts:

$ ./all.bash

...

ok      runtime 36.204s
ok      runtime/debug   0.068s
--- FAIL: TestTraceStress (4.73s)
        trace_test.go:226: failed to parse trace: p 4 is running before start (offset 322664, time 65700205)
--- FAIL: TestTraceStressStartStop (1.70s)
        trace_test.go:365: failed to parse trace: g 5 is not waiting during syscall exit (offset 190855, time 1924)
--- FAIL: TestTraceFutileWakeup (0.56s)
        trace_test.go:433: failed to parse trace: p 7 is running before start (offset 119705, time 13515283)
FAIL
FAIL    runtime/pprof   13.658s
ok      sort    0.102s
ok      strconv 0.436s
ok      strings 0.170s
ok      sync    0.179s
ok      sync/atomic     0.610s
ok      syscall 0.075s
ok      testing 2.399s
ok      testing/quick   0.044s
ok      text/scanner    0.010s
ok      text/tabwriter  0.016s
ok      text/template   0.055s
ok      text/template/parse     0.018s
ok      time    2.948s
ok      unicode 0.010s
ok      unicode/utf16   0.007s
ok      unicode/utf8    0.008s
ok      cmd/addr2line   2.795s
ok      cmd/api 0.042s
ok      cmd/asm/internal/asm    0.043s
ok      cmd/asm/internal/lex    0.004s
ok      cmd/compile/internal/big        2.936s
ok      cmd/cover       4.232s
ok      cmd/doc 0.028s
ok      cmd/fix 0.021s
ok      cmd/go  108.378s
ok      cmd/gofmt       0.071s
ok      cmd/internal/goobj      0.005s
ok      cmd/internal/obj        0.014s
ok      cmd/internal/obj/x86    0.021s
ok      cmd/internal/rsc.io/arm/armasm  0.010s
ok      cmd/internal/rsc.io/x86/x86asm  0.501s
ok      cmd/newlink     0.021s
ok      cmd/nm  2.267s
ok      cmd/objdump     8.034s
ok      cmd/pack        5.847s
ok      cmd/pprof/internal/profile      0.005s
ok      cmd/vet 11.152s
2015/07/17 14:40:13 Failed: exit status 1

$ go version
go version devel +cc8f544 Fri Jul 17 03:34:11 2015 +0000 linux/amd64
$ linuxinfo
Linux debianhp64 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1 (2015-05-24)
Zwei AMD Unknown 1000MHz Prozessoren, 3990.06 Bogomips gesamt, 2891 MB RAM
Systembibliothek 2.19.0

[bradfitz]

We're getting lots of dups of this. (see referenced issues above)

We should disable these tests for Go 1.5 if they're not reliable.

@dvyukov?

[dvyukov]

Can we exclude them only short mode?
Ideally we still run them on a subset of builders where it currently passes. Some parts of runtime tracing are subtle, it would be bad if they silently regress.
I can try to write a function for linux that will try to determine whether cpu ticks are sane on the machine or not. But it is not completely trivial.
Looking at Zwei AMD Unknown 1000MHz Prozessoren, I guess in this case it is some broken virtual environment.