sql/flowinfra: TestOutboxInitializesStreamBeforeReceivingAnyRows failed

[cockroach-teamcity]

sql/flowinfra.TestOutboxInitializesStreamBeforeReceivingAnyRows failed with artifacts on master @ cdace509b79339ba6c60f56e1bbe5d0b261de943:

  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:442 +0x161

Previous write at 0x00c003668600 by goroutine 745:
  google.golang.org/grpc/internal/transport.(*http2Client).operateHeaders()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1295 +0x2309
  google.golang.org/grpc/internal/transport.(*http2Client).reader()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1484 +0xdb1

Goroutine 801 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:434 +0x28d
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:334 +0x494
  github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:1067 +0x3c1
  sync.(*Once).doSlow()
      /usr/local/go/src/sync/once.go:68 +0x109
  sync.(*Once).Do()
      /usr/local/go/src/sync/once.go:59 +0x68
  github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal()
      /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:1061 +0x25c
  github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDialNode()
      /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:1026 +0x88
  github.com/cockroachdb/cockroach/pkg/rpc/nodedialer.(*Dialer).dial()
      /go/src/github.com/cockroachdb/cockroach/pkg/rpc/nodedialer/nodedialer.go:164 +0x19d
  github.com/cockroachdb/cockroach/pkg/rpc/nodedialer.(*Dialer).DialNoBreaker()
      /go/src/github.com/cockroachdb/cockroach/pkg/rpc/nodedialer/nodedialer.go:106 +0x1c6
  github.com/cockroachdb/cockroach/pkg/sql/execinfra.GetConnForOutbox()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/outboxbase.go:47 +0x242
  github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*Outbox).mainLoop()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/outbox.go:227 +0xefa
  github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*Outbox).run()
      /go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/outbox.go:426 +0x50

Goroutine 745 (running) created at:
  google.golang.org/grpc/internal/transport.newHTTP2Client()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:346 +0x1488
  google.golang.org/grpc/internal/transport.NewClientTransport()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:574 +0x6c7
  google.golang.org/grpc.(*addrConn).createTransport()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1328 +0x4f7
  google.golang.org/grpc.(*addrConn).tryAllAddrs()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1258 +0x507
  google.golang.org/grpc.(*addrConn).resetTransport()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1173 +0x33b
==================
    outbox_test.go:266: -- test log scope end --
    testing.go:1092: race detected during execution of test
--- FAIL: TestOutboxInitializesStreamBeforeReceivingAnyRows (0.02s)

Reproduce

To reproduce, try:

make stressrace TESTS=TestOutboxInitializesStreamBeforeReceivingAnyRows PKG=./pkg/sql/flowinfra TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Parameters in this failure:

• GOFLAGS=-race -parallel=4

/cc @cockroachdb/sql-queries asubiotto

This test on roachdash | Improve this report!

[knz]

See #68154