Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data Race: Closing a stream accesses streamingState data without acquiring a lock #170

Closed
rodaine opened this issue May 31, 2024 · 0 comments · Fixed by #173
Closed

Data Race: Closing a stream accesses streamingState data without acquiring a lock #170

rodaine opened this issue May 31, 2024 · 0 comments · Fixed by #173

Comments

@rodaine
Copy link
Member

rodaine commented May 31, 2024

When a stream receive is processing concurrently with a call to close, a race can occur accessing the counters and attribute list on the streamingState as the closeSpan func does not acquire a lock before accessing the data:

WARNING: DATA RACE
Read at 0x00c010498438 by goroutine 34631:
  connectrpc.com/otelconnect.(*Interceptor).WrapStreamingClient.func1.2()
      /Users/rodaine/pkg/mod/connectrpc.com/otelconnect@v0.7.0/interceptor.go:234 +0x100

Previous write at 0x00c010498438 by goroutine 34630:
  connectrpc.com/otelconnect.(*streamingState).receive()
      /Users/rodaine/pkg/mod/connectrpc.com/otelconnect@v0.7.0/streaming.go:85 +0x230
  connectrpc.com/otelconnect.(*Interceptor).WrapStreamingClient.func1.4()
      /Users/rodaine/pkg/mod/connectrpc.com/otelconnect@v0.7.0/interceptor.go:258 +0xe0
  connectrpc.com/otelconnect.(*streamingClientInterceptor).Receive()
      /Users/rodaine/pkg/mod/connectrpc.com/otelconnect@v0.7.0/payloadinterceptor.go:30 +0xa8
  connectrpc.com/connect.receiveUnaryMessage[ ... ]()
      /Users/rodaine/pkg/mod/connectrpc.com/connect@v1.16.1/connect.go:419 +0x22c
  connectrpc.com/connect.receiveUnaryResponse[ ... ]()
      /Users/rodaine/pkg/mod/connectrpc.com/connect@v1.16.1/connect.go:376 +0xe4
  connectrpc.com/connect.(*ClientStreamForClient[ ... ]).CloseAndReceive()
      /Users/rodaine/pkg/mod/connectrpc.com/connect@v1.16.1/client_stream.go:82 +0x1d4
Similar races
WARNING: DATA RACE
Write at 0x00c010498420 by goroutine 34631:
  connectrpc.com/otelconnect.(*streamingState).addAttributes()
      /Users/rodaine/pkg/mod/connectrpc.com/otelconnect@v0.7.0/streaming.go:73 +0x238
  connectrpc.com/otelconnect.(*Interceptor).WrapStreamingClient.func1.2()
      /Users/rodaine/pkg/mod/connectrpc.com/otelconnect@v0.7.0/interceptor.go:235 +0x2b8

Previous read at 0x00c010498420 by goroutine 34630:
  connectrpc.com/otelconnect.(*streamingState).receive()
      /Users/rodaine/pkg/mod/connectrpc.com/otelconnect@v0.7.0/streaming.go:97 +0x628
  connectrpc.com/otelconnect.(*Interceptor).WrapStreamingClient.func1.4()
      /Users/rodaine/pkg/mod/connectrpc.com/otelconnect@v0.7.0/interceptor.go:258 +0xe0
  connectrpc.com/otelconnect.(*streamingClientInterceptor).Receive()
      /Users/rodaine/pkg/mod/connectrpc.com/otelconnect@v0.7.0/payloadinterceptor.go:30 +0xa8
WARNING: DATA RACE
Read at 0x00c01056c540 by goroutine 34631:
  runtime.slicecopy()
      /Users/rodaine/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.3.darwin-arm64/src/runtime/slice.go:325 +0x0
  go.opentelemetry.io/otel/metric.WithAttributes()
      /Users/rodaine/pkg/mod/go.opentelemetry.io/otel/metric@v1.27.0/instrument.go:366 +0xb8
  connectrpc.com/otelconnect.(*Interceptor).WrapStreamingClient.func1.2()
      /Users/rodaine/pkg/mod/connectrpc.com/otelconnect@v0.7.0/interceptor.go:242 +0x504

Previous write at 0x00c01056c540 by goroutine 34630:
  runtime.slicecopy()
      /Users/rodaine/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.3.darwin-arm64/src/runtime/slice.go:325 +0x0
  connectrpc.com/otelconnect.(*streamingState).addAttributes()
      /Users/rodaine/pkg/mod/connectrpc.com/otelconnect@v0.7.0/streaming.go:73 +0x20c
  connectrpc.com/otelconnect.(*streamingState).receive()
      /Users/rodaine/pkg/mod/connectrpc.com/otelconnect@v0.7.0/streaming.go:89 +0x460
  connectrpc.com/otelconnect.(*Interceptor).WrapStreamingClient.func1.4()
      /Users/rodaine/pkg/mod/connectrpc.com/otelconnect@v0.7.0/interceptor.go:258 +0xe0
  connectrpc.com/otelconnect.(*streamingClientInterceptor).Receive()

Fix should be for closeSpan to acquire the lock fwict.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant