runtime/race: RaceAsFunc3 and many others are flaky

[bradfitz]

FreeBSD race flake:

http://build.golang.org/log/e9536db3f2c81412f3cc2225ba19c60d34ad06e2

freebsd-amd64-race at 9d6427d8992b05445029f95c9555820675dd2e3e
...
RaceAsFunc1                              .
RaceAsFunc2                              .
RaceAsFunc3                              FAILED
NoRaceAsFunc4                            .
RaceHeapParam                            .
...

[bradfitz]

Also Windows:

windows-amd64-race at 970ce1c
http://build.golang.org/log/fd04d277aea33c3502e48ec6a06176d0a7d147c9

[bradfitz]

Related: #14079

/cc @aclements for statistics. All these race flakes seem very recent.

[aclements]

$ findflakes -grep 'RaceAsFunc3 *FAILED'
First observed 445c685 22 Jan 01:13 2016 (38 commits ago)
Last observed  b4c9d01 26 Jan 16:27 2016 (2 commits ago)
56% chance failure is still happening
25% failure probability (10 of 37 commits)
Likely culprits:
   25% 445c685 cmd/compile: remove unused (dead) fields
   19% 0760023 runtime: skip TestSignalExitStatus on NetBSD
   14% c7f5831 lib/time: update to IANA release 2015g.
   11% 123510b runtime: save context value in NetBSD sigtramp
    8% 4c4476c runtime: on NetBSD and DragonFly drop signal stack in new thread
    6% 489f65b net: improve netsh usage in Windows unit tests
    4% b203f88 crypto/tls: note in comment that Certificate.Leaf is nil after parsing.
    3% 754216d text/template: fix documentation for pipelines
    3% f8f4cfa net/http: make Client propagate Request.Cancel over redirected requests
Past failures:
  ea7be51 02 Mar 17:52 2015 to 9626561 07 May 21:08 2015
    0.73% failure probability (9 of 1097 commits)

$ greplogs -md -l -dashboard -e 'RaceAsFunc3 *FAILED'
2015-03-02T17:52:11-ea7be51/windows-amd64-race
2015-03-02T18:36:04-145b6fb/windows-amd64-race
2015-03-02T18:39:01-190357d/windows-amd64-race
2015-03-02T18:46:11-bfb8ad5/windows-amd64-race
2015-04-22T20:32:16-68f5570/freebsd-amd64-race
2015-04-29T04:44:38-7e49c81/freebsd-amd64-race
2015-05-04T20:17:39-91938fd/freebsd-amd64-race
2015-05-06T11:38:08-2230e9d/windows-amd64-race
2015-05-07T21:08:29-9626561/darwin-amd64-race-cheney
2016-01-22T01:13:43-445c685/freebsd-amd64-race
2016-01-22T01:14:16-a4599ef/windows-amd64-race
2016-01-23T01:28:06-4f40182/windows-amd64-race
2016-01-23T05:47:13-7d8c8c0/windows-amd64-race
2016-01-24T05:50:53-659c632/windows-amd64-race
2016-01-24T16:07:30-970ce1c/windows-amd64-race
2016-01-24T20:21:45-9d6427d/freebsd-amd64-race
2016-01-25T02:45:09-3415d0c/freebsd-amd64-race
2016-01-25T02:50:40-801bebe/windows-amd64-race
2016-01-26T16:27:16-b4c9d01/windows-amd64-race

[bradfitz]

There are many race-related flakes on the dashboard. I feel like shipping a Go 1.6 at this point would mean shipping a potentially broken race detector.

@dvyukov, any clue why?

[mwhudson]

I ran stress -p1 ./race.test on linux/amd64 from tip and got 8 / 32 failures. Then I reverted the most obvious candidate I could think of -- c5f0f88 ("runtime/race: update race runtime") -- and I've had 0 / 16 failures. And as I was about to submit this comment, it ticked over to 1 / 22 :-) but it still seems like the update didn't help.

[gopherbot]

CL https://golang.org/cl/18968 mentions this issue.

[mwhudson]

Ran some longer tests, seems less clear. I guess it should be possible to bisect over the last few months of development given that it was a cm/dist problem preventing the tests from being run rather than anything preventing go test -race runtime/race from working. Would take a while though.