testing: are T.Run and B.Run safe for concurrent use?

[tombergan]

I do not see any documentation specifying if T.Run is safe for concurrent use. I see the following doc for testing.T, which does not mention T.Run:

A test ends when its Test function returns or calls any of the methods FailNow, Fatal, Fatalf, SkipNow, Skip, or Skipf. Those methods, as well as the Parallel method, must be called only from the goroutine running the Test function.

The other reporting methods, such as the variations of Log and Error, may be called simultaneously from multiple goroutines.

There is a similar doc for testing.B.

I ask because I believe T.Run was safe for concurrent use in go 1.7. Then, this line was added in go 1.8, which made T.Run no longer safe for concurrent use. I don't particularly care whether T.Run is safe for concurrent use, but I think the semantics should be clarified.

Optimistically labeling this go1.8.

/cc @dsnet

[dsnet]

If T.Run was safe concurrently in Go1.7, even if it was ambiguous whether it should be, it's worth considering whether it should continue to be concurrent safe in Go1.8.

\cc @mpvl

(moving to Go1.8Maybe; I don't think this is a release blocker at this point)

[rsc]

Not sure whether @mpvl is around. Sent https://go-review.googlesource.com/35354.
/cc @bradfitz

[gopherbot]

CL https://golang.org/cl/35354 mentions this issue.

[flowblok]

CL https://golang.org/cl/35354 added this comment to B.Run():

// Run may be called simultaneously from multiple goroutines, but all such
// calls must happen before the outer benchmark function for b returns.

Which I don't think is true at all: calling B.Run() multiple times from multiple goroutines results in benchmarkLock.Unlock() being called multiple times before benchmarkLock.Lock(), which causes a crash.

(example code: https://play.golang.org/p/0u15AvArTK)

[dsnet]

Is this a documentation issue or a implementation issue? cl/35354 does have a test for multiple goroutines using when using testing.T.Run, but lacks a test for testing.B.Run, which clearly seems broken.

Are you proposing we properly making testing.B.Run work also? or just remove the comment.

[flowblok]

I don't mind which way it gets resolved, but it's significantly less effort to just remove the comment.

[gopherbot]

Change https://golang.org/cl/80841 mentions this issue: testing: remove claim that b.Run is safe for concurrent use