Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime/race: deflake tests #14119

Open
dvyukov opened this issue Jan 27, 2016 · 6 comments

Comments

@dvyukov
Copy link
Member

commented Jan 27, 2016

See https://go-review.googlesource.com/#/c/18968 for context.
Race tests run with GOMAXPROCS=1, this makes them more or less reliable. But the ultimate solution is to explicitly annotate tests with required execution order by means of a special "invisible" synchronization primitive (that's what is done for C++ ThreadSanitizer tests). But that would require going over 350 tests.

@dvyukov dvyukov self-assigned this Jan 27, 2016

@gopherbot

This comment has been minimized.

Copy link

commented Jan 27, 2016

CL https://golang.org/cl/18968 mentions this issue.

@aclements aclements added this to the Go1.7 milestone Jan 27, 2016

gopherbot pushed a commit that referenced this issue Jan 27, 2016
runtime/race: run tests with GOMAXPROCS=1
We set GOMAXPROCS=1 to prevent test flakiness.
There are two sources of flakiness:
1. Some tests rely on particular execution order.
   If the order is different, race does not happen at all.
2. Ironically, ThreadSanitizer runtime contains a logical race condition
   that can lead to false negatives if racy accesses happen literally at the same time.
Tests used to work reliably in the good old days of GOMAXPROCS=1.
So let's set it for now. A more reliable solution is to explicitly annotate tests
with required execution order by means of a special "invisible" synchronization primitive
(that's what is done for C++ ThreadSanitizer tests). This is issue #14119.

This reduces flakes on RaceAsFunc3 test from 60/3000 to 1/3000.

Fixes #14086
Fixes #14079
Fixes #14035

Change-Id: Ibaec6b2b21e27b62563bffbb28473a854722cf41
Reviewed-on: https://go-review.googlesource.com/18968
Reviewed-by: Austin Clements <austin@google.com>
Run-TryBot: Austin Clements <austin@google.com>
Reviewed-by: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>

@rsc rsc modified the milestones: Go1.8, Go1.7 May 18, 2016

@quentinmit quentinmit added the NeedsFix label Sep 29, 2016

@rsc

This comment has been minimized.

Copy link
Contributor

commented Sep 29, 2016

Ideally this would happen, but I think Dmitriy is the only one who can really do it.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Mar 18, 2017

#19604 was marked as a dup of this bug.

If these tests can't be made reliable, I think it's time to disable them. Or give them, say, 10 chances to succeed before reporting a failure.

Marking this as Go 1.9 to either fix them, disable them or make them try harder to pass.

@josharian

This comment has been minimized.

Copy link
Contributor

commented Apr 7, 2017

@odeke-em

This comment has been minimized.

Copy link
Member

commented Jul 8, 2017

How's it going @dvyukov? Might you have any bandwidth for this bug? Thanks.

@dvyukov

This comment has been minimized.

Copy link
Member Author

commented Jul 8, 2017

@odeke-em No, but you don't need to wait for me. We need something simple along the following lines:

--- a/src/runtime/race/testdata/mop_test.go
+++ b/src/runtime/race/testdata/mop_test.go
@@ -9,6 +9,7 @@ import (
        "crypto/sha1"
        "errors"
        "fmt"
+       "internal/race"
        "io"
        "os"
        "runtime"
@@ -62,11 +63,18 @@ func TestRaceIntRWClosures(t *testing.T) {
        var x, y int
        ch := make(chan int, 2)
 
+       nosync := make(chan bool)
        go func() {
                y = x
+               race.Disable()
+               nosync <- true
+               race.Enable()
                ch <- 1
        }()
        go func() {
+               race.Disable()
+               <-nosync
+               race.Enable()
                x = 1
                ch <- 1
        }()

However, I wasn't able to reproduce any flakes on tip. I've tried:

go test -c -race runtime/race/testdata/mop_test.go
stress -failure="PASS" ./race.test -test.run TestRaceIntRWClosures

With different values of -p and also some additional goroutines/computations in the test to defeat 'runnext`.

@dvyukov dvyukov removed their assignment Jul 8, 2017

@bradfitz bradfitz modified the milestones: Go1.10, Go1.9 Jul 14, 2017

@rsc rsc modified the milestones: Go1.10, Unplanned Nov 29, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.