Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime/race: misses race in gRPC server because of timing #32682

Closed
pohly opened this issue Jun 19, 2019 · 5 comments
Closed

runtime/race: misses race in gRPC server because of timing #32682

pohly opened this issue Jun 19, 2019 · 5 comments

Comments

@pohly
Copy link

pohly commented Jun 19, 2019

What version of Go are you using (go version)?

$ go version
go version go1.12.6 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/pohly/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/nvme/gopath"
GOPROXY=""
GORACE=""
GOROOT="/nvme/gopath/go"
GOTMPDIR=""
GOTOOLDIR="/nvme/gopath/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/work/grpc-locking/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build250390168=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I want to enable the Go race detector in the testing of a gRPC server. gRPC calls methods concurrently in different goroutines, so those methods have to protect access to shared data via locks. I got the race detector working (it reported an artificially introduced race) but some other race (found via code review) wasn't reported although the faulty code gets invoked by different goroutines (verified via debug output).

I reduced it to a simpler example that works with just a single go test command:
https://github.com/pohly/grpc-locking

To reproduce (more details in the repo's README.md):

  • check out that repo
  • go test -v -race -run=TestGRPC . -args -delay=10ms

The delay is important. It configures the duration of a sleep between starting two goroutines which call the faulty code through gRPC. This simulates the random timing that occurs when different clients call the server from remote.

It's also relevant that gRPC is involved. The race detector always works when calling the methods directly.

What did you expect to see?

I expect to get the race reported regardless of the delay.

What did you see instead?

The race is only reported when the delay is small, like 1ms.

@pohly
Copy link
Author

pohly commented Jun 19, 2019

My current hypothesis is that the delay causes gRPC on the server side to behave sufficiently different (perhaps doing additional IO?) that the race detector sees an ordering of the goroutines that isn't actually guaranteed in this case. This might be similar to #27756.

But I don't know enough about the race detector and/or gRPC implementation to determine whether that is the case. My hope is that someone can look at this and determine whether this is a bug and/or something that can be made to work (which would be great!).

pohly added a commit to pohly/pmem-CSI that referenced this issue Jun 19, 2019
This should be enough to trigger a race warning during startup when
the nodes register themselves, but somehow that doesn't happen.
Reported against Go, see golang/go#32682

Only the manually added data race is reported:

$ kubectl logs pmem-csi-controller-0 pmem-driver
I0510 09:27:19.322456       1 glog.go:64] Version: v0.0.0-115-g5b65c286-dirty
I0510 09:27:19.332016       1 glog.go:79] Using Servername: pmem-node-controller
I0510 09:27:19.332243       1 glog.go:79] Registering node: foo, endpoint: bar at goroutine 23 [running]:
runtime/debug.Stack(0x1227a88, 0x1, 0x1)
	/usr/lib/golang/src/runtime/debug/stack.go:24 +0xab
github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*registryServer).RegisterController(0xc000190220, 0xd8d540, 0xc00009a010, 0xc0000a1580, 0x3471b8, 0xf, 0x125e8e0)
	/go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/registryserver.go:70 +0x4b0
github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*pmemDriver).Run.func2()
	/go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/pmem-csi-driver.go:173 +0xd3
created by github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*pmemDriver).Run
	/go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/pmem-csi-driver.go:175 +0x4eb
I0510 09:27:19.332327       1 glog.go:79] Registering node: foo, endpoint: bar at goroutine 24 [running]:
runtime/debug.Stack(0x1227a88, 0x1, 0x2)
	/usr/lib/golang/src/runtime/debug/stack.go:24 +0xab
github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*registryServer).RegisterController(0xc000190220, 0xd8d540, 0xc00009a010, 0xc0000a1580, 0x0, 0x0, 0x0)
	/go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/registryserver.go:70 +0x4b0
github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*pmemDriver).Run.func2()
	/go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/pmem-csi-driver.go:173 +0xd3
created by github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*pmemDriver).Run
	/go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/pmem-csi-driver.go:176 +0x504
==================
WARNING: DATA RACE
Write at 0x00c00011ccf0 by goroutine 11:
  runtime.mapassign_faststr()
      /usr/lib/golang/src/runtime/map_faststr.go:202 +0x0
  github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*registryServer).RegisterController()
      /go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/registryserver.go:73 +0x716
  github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*pmemDriver).Run.func2()
      /go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/pmem-csi-driver.go:173 +0xd2

Previous write at 0x00c00011ccf0 by goroutine 10:
  runtime.mapassign_faststr()
      /usr/lib/golang/src/runtime/map_faststr.go:202 +0x0
  github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*registryServer).RegisterController()
      /go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/registryserver.go:73 +0x716
  github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*pmemDriver).Run.func2()
      /go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/pmem-csi-driver.go:173 +0xd2

Goroutine 11 (running) created at:
  github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*pmemDriver).Run()
      /go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/pmem-csi-driver.go:176 +0x503
  github.com/intel/pmem-csi/pkg/pmem-csi-driver.Main()
      /go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/main.go:73 +0x593
  github.com/intel/pmem-csi/pkg/coverage.Run()
      /go/src/github.com/intel/pmem-csi/pkg/coverage/coverage.go:49 +0xb8
  github.com/intel/pmem-csi/cmd/pmem-csi-driver_test.TestMain()
      /go/src/github.com/intel/pmem-csi/cmd/pmem-csi-driver/main_test.go:11 +0x3a
  testing.tRunner()
      /usr/lib/golang/src/testing/testing.go:865 +0x163

Goroutine 10 (finished) created at:
  github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*pmemDriver).Run()
      /go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/pmem-csi-driver.go:175 +0x4ea
  github.com/intel/pmem-csi/pkg/pmem-csi-driver.Main()
      /go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/main.go:73 +0x593
  github.com/intel/pmem-csi/pkg/coverage.Run()
      /go/src/github.com/intel/pmem-csi/pkg/coverage/coverage.go:49 +0xb8
  github.com/intel/pmem-csi/cmd/pmem-csi-driver_test.TestMain()
      /go/src/github.com/intel/pmem-csi/cmd/pmem-csi-driver/main_test.go:11 +0x3a
  testing.tRunner()
      /usr/lib/golang/src/testing/testing.go:865 +0x163
==================
==================
WARNING: DATA RACE
Write at 0x00c0000ab0c8 by goroutine 11:
  github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*registryServer).RegisterController()
      /go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/registryserver.go:73 +0x737
  github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*pmemDriver).Run.func2()
      /go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/pmem-csi-driver.go:173 +0xd2

Previous write at 0x00c0000ab0c8 by goroutine 10:
  github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*registryServer).RegisterController()
      /go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/registryserver.go:73 +0x737
  github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*pmemDriver).Run.func2()
      /go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/pmem-csi-driver.go:173 +0xd2

Goroutine 11 (running) created at:
  github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*pmemDriver).Run()
      /go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/pmem-csi-driver.go:176 +0x503
  github.com/intel/pmem-csi/pkg/pmem-csi-driver.Main()
      /go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/main.go:73 +0x593
  github.com/intel/pmem-csi/pkg/coverage.Run()
      /go/src/github.com/intel/pmem-csi/pkg/coverage/coverage.go:49 +0xb8
  github.com/intel/pmem-csi/cmd/pmem-csi-driver_test.TestMain()
      /go/src/github.com/intel/pmem-csi/cmd/pmem-csi-driver/main_test.go:11 +0x3a
  testing.tRunner()
      /usr/lib/golang/src/testing/testing.go:865 +0x163

Goroutine 10 (finished) created at:
  github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*pmemDriver).Run()
      /go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/pmem-csi-driver.go:175 +0x4ea
  github.com/intel/pmem-csi/pkg/pmem-csi-driver.Main()
      /go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/main.go:73 +0x593
  github.com/intel/pmem-csi/pkg/coverage.Run()
      /go/src/github.com/intel/pmem-csi/pkg/coverage/coverage.go:49 +0xb8
  github.com/intel/pmem-csi/cmd/pmem-csi-driver_test.TestMain()
      /go/src/github.com/intel/pmem-csi/cmd/pmem-csi-driver/main_test.go:11 +0x3a
  testing.tRunner()
      /usr/lib/golang/src/testing/testing.go:865 +0x163
==================
I0510 09:27:19.332842       1 glog.go:79] Listening for connections on address: /csi/csi-controller.sock
I0510 09:27:19.333553       1 glog.go:79] Listening for connections on address: [::]:10000
I0510 09:27:20.977288       1 glog.go:58] GRPC call: /csi.v1.Identity/Probe
I0510 09:27:20.977399       1 glog.go:58] GRPC request: {}
I0510 09:27:20.985448       1 glog.go:58] GRPC response: {}
I0510 09:27:20.993604       1 glog.go:58] GRPC call: /csi.v1.Identity/GetPluginInfo
I0510 09:27:20.993716       1 glog.go:58] GRPC request: {}
I0510 09:27:21.001918       1 glog.go:58] GRPC response: {"name":"pmem-csi.intel.com","vendor_version":"v0.0.0-115-g5b65c286-dirty"}
I0510 09:27:21.011209       1 glog.go:58] GRPC call: /csi.v1.Identity/GetPluginCapabilities
I0510 09:27:21.011287       1 glog.go:58] GRPC request: {}
I0510 09:27:21.018718       1 glog.go:58] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
I0510 09:27:21.039403       1 glog.go:58] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0510 09:27:21.039442       1 glog.go:58] GRPC request: {}
I0510 09:27:21.047203       1 glog.go:58] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":4}}}]}
I0510 09:27:22.394398       1 glog.go:58] GRPC call: /csi.v1.Identity/Probe
I0510 09:27:22.394436       1 glog.go:58] GRPC request: {}
I0510 09:27:22.396974       1 glog.go:58] GRPC response: {}
I0510 09:27:22.401942       1 glog.go:58] GRPC call: /csi.v1.Identity/GetPluginInfo
I0510 09:27:22.401968       1 glog.go:58] GRPC request: {}
I0510 09:27:22.404297       1 glog.go:58] GRPC response: {"name":"pmem-csi.intel.com","vendor_version":"v0.0.0-115-g5b65c286-dirty"}
I0510 09:27:22.407754       1 glog.go:58] GRPC call: /csi.v1.Identity/GetPluginCapabilities
I0510 09:27:22.407784       1 glog.go:58] GRPC request: {}
I0510 09:27:22.409988       1 glog.go:58] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
I0510 09:27:22.421620       1 glog.go:58] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0510 09:27:22.421680       1 glog.go:58] GRPC request: {}
I0510 09:27:22.439456       1 glog.go:58] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":4}}}]}
I0510 09:27:55.838729       1 glog.go:58] GRPC call: /registry.v0.Registry/RegisterController
I0510 09:27:55.839521       1 glog.go:58] GRPC call: /registry.v0.Registry/RegisterController
I0510 09:27:55.838827       1 glog.go:58] GRPC request: {"endpoint":"tcp://192.168.8.8:10001","node_id":"host-3"}
I0510 09:27:55.840065       1 glog.go:79] Registering node: host-3, endpoint: tcp://192.168.8.8:10001 at goroutine 75 [running]:
runtime/debug.Stack(0x1227a88, 0xc000000001, 0x3)
	/usr/lib/golang/src/runtime/debug/stack.go:24 +0xab
github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*registryServer).RegisterController(0xc000190220, 0xd8d5c0, 0xc00021a210, 0xc0002202c0, 0xc000190220, 0xbe, 0x0)
	/go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/registryserver.go:70 +0x4b0
github.com/intel/pmem-csi/pkg/pmem-registry._Registry_RegisterController_Handler.func1(0xd8d5c0, 0xc00021a210, 0xc60c20, 0xc0002202c0, 0xc00029b928, 0x1, 0x1, 0x0)
	/go/src/github.com/intel/pmem-csi/pkg/pmem-registry/pmem-registry.pb.go:269 +0xd4
github.com/intel/pmem-csi/pkg/pmem-common.LogGRPCServer(0xd8d5c0, 0xc00021a210, 0xc60c20, 0xc0002202c0, 0xc000228100, 0xc000228120, 0xc000044a80, 0x0, 0xc00029b9d8, 0x4225d8)
	/go/src/github.com/intel/pmem-csi/pkg/pmem-common/tracing.go:31 +0x1f4
github.com/intel/pmem-csi/pkg/pmem-registry._Registry_RegisterController_Handler(0xc25720, 0xc000190220, 0xd8d5c0, 0xc00021a210, 0xc0003e6140, 0xcc4740, 0xd8d5c0, 0xc00021a210, 0xc000278000, 0x21)
	/go/src/github.com/intel/pmem-csi/pkg/pmem-registry/pmem-registry.pb.go:271 +0x241
github.com/intel/pmem-csi/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc000075680, 0xd93240, 0xc0004aed80, 0xc000146200, 0xc00011d260, 0x120d3a0, 0x0, 0x0, 0x0)
	/go/src/github.com/intel/pmem-csi/vendor/google.golang.org/grpc/server.go:971 +0x937
github.com/intel/pmem-csi/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000075680, 0xd93240, 0xc0004aed80, 0xc000146200, 0x0)
	/go/src/github.com/intel/pmem-csi/vendor/google.golang.org/grpc/server.go:1250 +0x133a
github.com/intel/pmem-csi/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0001e6fe0, 0xc000075680, 0xd93240, 0xc0004aed80, 0xc000146200)
	/go/src/github.com/intel/pmem-csi/vendor/google.golang.org/grpc/server.go:690 +0xad
created by github.com/intel/pmem-csi/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/intel/pmem-csi/vendor/google.golang.org/grpc/server.go:688 +0xb9
I0510 09:27:55.839618       1 glog.go:58] GRPC request: {"endpoint":"tcp://192.168.8.6:10001","node_id":"host-2"}
I0510 09:27:55.840157       1 glog.go:58] GRPC response: {}
I0510 09:27:55.840817       1 glog.go:79] Registering node: host-2, endpoint: tcp://192.168.8.6:10001 at goroutine 7 [running]:
runtime/debug.Stack(0x1227a88, 0xc000000001, 0x4)
	/usr/lib/golang/src/runtime/debug/stack.go:24 +0xab
github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*registryServer).RegisterController(0xc000190220, 0xd8d5c0, 0xc000092b70, 0xc00030a040, 0xc000190220, 0xb3, 0x0)
	/go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/registryserver.go:70 +0x4b0
github.com/intel/pmem-csi/pkg/pmem-registry._Registry_RegisterController_Handler.func1(0xd8d5c0, 0xc000092b70, 0xc60c20, 0xc00030a040, 0xc000297928, 0x1, 0x1, 0x0)
	/go/src/github.com/intel/pmem-csi/pkg/pmem-registry/pmem-registry.pb.go:269 +0xd4
github.com/intel/pmem-csi/pkg/pmem-common.LogGRPCServer(0xd8d5c0, 0xc000092b70, 0xc60c20, 0xc00030a040, 0xc0000a80c0, 0xc0000a80e0, 0xc000045180, 0x53170a1b12011052, 0xc0002979d8, 0x4225d8)
	/go/src/github.com/intel/pmem-csi/pkg/pmem-common/tracing.go:31 +0x1f4
github.com/intel/pmem-csi/pkg/pmem-registry._Registry_RegisterController_Handler(0xc25720, 0xc000190220, 0xd8d5c0, 0xc000092b70, 0xc000336000, 0xcc4740, 0xd8d5c0, 0xc000092b70, 0xc000028240, 0x21)
	/go/src/github.com/intel/pmem-csi/pkg/pmem-registry/pmem-registry.pb.go:271 +0x241
github.com/intel/pmem-csi/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc000075680, 0xd93240, 0xc0004ae780, 0xc00022a100, 0xc00011d260, 0x120d3a0, 0x0, 0x0, 0x0)
	/go/src/github.com/intel/pmem-csi/vendor/google.golang.org/grpc/server.go:971 +0x937
github.com/intel/pmem-csi/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000075680, 0xd93240, 0xc0004ae780, 0xc00022a100, 0x0)
	/go/src/github.com/intel/pmem-csi/vendor/google.golang.org/grpc/server.go:1250 +0x133a
github.com/intel/pmem-csi/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0001e6a50, 0xc000075680, 0xd93240, 0xc0004ae780, 0xc00022a100)
	/go/src/github.com/intel/pmem-csi/vendor/google.golang.org/grpc/server.go:690 +0xad
created by github.com/intel/pmem-csi/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/intel/pmem-csi/vendor/google.golang.org/grpc/server.go:688 +0xb9
I0510 09:27:55.840949       1 glog.go:58] GRPC response: {}
I0510 09:28:02.768374       1 glog.go:58] GRPC call: /registry.v0.Registry/RegisterController
I0510 09:28:02.768479       1 glog.go:58] GRPC request: {"endpoint":"tcp://192.168.8.4:10001","node_id":"host-1"}
I0510 09:28:02.769457       1 glog.go:79] Registering node: host-1, endpoint: tcp://192.168.8.4:10001 at goroutine 59 [running]:
runtime/debug.Stack(0x1227a88, 0xc000000001, 0x5)
	/usr/lib/golang/src/runtime/debug/stack.go:24 +0xab
github.com/intel/pmem-csi/pkg/pmem-csi-driver.(*registryServer).RegisterController(0xc000190220, 0xd8d5c0, 0xc00011c660, 0xc00030a1c0, 0xc000190220, 0xb3, 0x0)
	/go/src/github.com/intel/pmem-csi/pkg/pmem-csi-driver/registryserver.go:70 +0x4b0
github.com/intel/pmem-csi/pkg/pmem-registry._Registry_RegisterController_Handler.func1(0xd8d5c0, 0xc00011c660, 0xc60c20, 0xc00030a1c0, 0xc000297928, 0x1, 0x1, 0x0)
	/go/src/github.com/intel/pmem-csi/pkg/pmem-registry/pmem-registry.pb.go:269 +0xd4
github.com/intel/pmem-csi/pkg/pmem-common.LogGRPCServer(0xd8d5c0, 0xc00011c660, 0xc60c20, 0xc00030a1c0, 0xc0000a87a0, 0xc0000a87c0, 0xc00025c000, 0x53170a1b12011052, 0xc0002979d8, 0x4225d8)
	/go/src/github.com/intel/pmem-csi/pkg/pmem-common/tracing.go:31 +0x1f4
github.com/intel/pmem-csi/pkg/pmem-registry._Registry_RegisterController_Handler(0xc25720, 0xc000190220, 0xd8d5c0, 0xc00011c660, 0xc0003366e0, 0xcc4740, 0xd8d5c0, 0xc00011c660, 0xc000028480, 0x21)
	/go/src/github.com/intel/pmem-csi/pkg/pmem-registry/pmem-registry.pb.go:271 +0x241
github.com/intel/pmem-csi/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc000075680, 0xd93240, 0xc0004ae480, 0xc0003d8000, 0xc00011d260, 0x120d3a0, 0x0, 0x0, 0x0)
	/go/src/github.com/intel/pmem-csi/vendor/google.golang.org/grpc/server.go:971 +0x937
github.com/intel/pmem-csi/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000075680, 0xd93240, 0xc0004ae480, 0xc0003d8000, 0x0)
	/go/src/github.com/intel/pmem-csi/vendor/google.golang.org/grpc/server.go:1250 +0x133a
github.com/intel/pmem-csi/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0001e64c0, 0xc000075680, 0xd93240, 0xc0004ae480, 0xc0003d8000)
	/go/src/github.com/intel/pmem-csi/vendor/google.golang.org/grpc/server.go:690 +0xad
created by github.com/intel/pmem-csi/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/intel/pmem-csi/vendor/google.golang.org/grpc/server.go:688 +0xb9
I0510 09:28:02.769537       1 glog.go:58] GRPC response: {}
@ianlancetaylor
Copy link
Contributor

CC @dvyukov

@dvyukov
Copy link
Member

dvyukov commented Jun 19, 2019

I don't think we can make this to work. Parasitic synchronization is the problem for any happens-before-based race detectors. But this is the price for absence of false positives. The idea is if the test is run multiple times (e.g. continuous testing) then any race should pop up eventually.

@pohly
Copy link
Author

pohly commented Jun 21, 2019

Thanks for the confirmation that this is simply how race detector works. Please feel free to close this issue.

Regarding running the test multiple times: in this case the risk of it going wrong is so low, I doubt it will ever occur during normal functional testing. An API test that is specifically written to access the same object in multiple goroutines triggers it (I checked and added such a test), but that implies that the developer writing the code already knows about the need to do locking.

That's why I was hoping that the race detector would find it.

@dvyukov
Copy link
Member

dvyukov commented Jun 23, 2019

As some rehabilitation: there is a positive correlation between the probability that race detector finds a race and that it's actually harmful in real life. Which is good. For example, if there is something that gives a soft guarantee that the racing accesses happen no closer than a minute, then the race detector may not find it but it is probably not so harmful in real life.
Though, of course, it's also possible that the test runs under race detector are not representative of all real life behaviors. And then race detector will miss something that actually can happen in real life.

@golang golang locked and limited conversation to collaborators Jun 22, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants