DATA RACE in the shopify/sarama

[hawkingrei]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

[2022-04-01T02:48:49.891Z] ==================

[2022-04-01T02:48:49.891Z] WARNING: DATA RACE

[2022-04-01T02:48:49.891Z] Read at 0x00000a6908d0 by goroutine 185:

[2022-04-01T02:48:49.891Z]   github.com/Shopify/sarama.version()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/!shopify/sarama@v1.32.0/version.go:8 +0x30

[2022-04-01T02:48:49.891Z]   github.com/Shopify/sarama.(*Broker).Open.func1.1()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/!shopify/sarama@v1.32.0/broker.go:183 +0x76

[2022-04-01T02:48:49.891Z]   runtime.deferreturn()

[2022-04-01T02:48:49.891Z]       /usr/local/go/src/runtime/panic.go:436 +0x32

[2022-04-01T02:48:49.891Z]   github.com/Shopify/sarama.withRecover()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/!shopify/sarama@v1.32.0/utils.go:43 +0x48

[2022-04-01T02:48:49.891Z]   github.com/Shopify/sarama.(*Broker).Open.func2()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/!shopify/sarama@v1.32.0/broker.go:172 +0x39

[2022-04-01T02:48:49.891Z] 

[2022-04-01T02:48:49.891Z] Previous write at 0x00000a6908d0 by goroutine 181:

[2022-04-01T02:48:49.891Z]   github.com/Shopify/sarama.version()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/!shopify/sarama@v1.32.0/version.go:11 +0x7e

[2022-04-01T02:48:49.891Z]   github.com/Shopify/sarama.(*Broker).Open.func1.1()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/!shopify/sarama@v1.32.0/broker.go:183 +0x76

[2022-04-01T02:48:49.891Z]   runtime.deferreturn()

[2022-04-01T02:48:49.891Z]       /usr/local/go/src/runtime/panic.go:436 +0x32

[2022-04-01T02:48:49.891Z]   github.com/Shopify/sarama.withRecover()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/!shopify/sarama@v1.32.0/utils.go:43 +0x48

[2022-04-01T02:48:49.891Z]   github.com/Shopify/sarama.(*Broker).Open.func2()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/!shopify/sarama@v1.32.0/broker.go:172 +0x39

[2022-04-01T02:48:49.891Z] 

[2022-04-01T02:48:49.891Z] Goroutine 185 (running) created at:

[2022-04-01T02:48:49.891Z]   github.com/Shopify/sarama.(*Broker).Open()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/!shopify/sarama@v1.32.0/broker.go:172 +0x3f0

[2022-04-01T02:48:49.891Z]   github.com/Shopify/sarama.(*client).Controller()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/!shopify/sarama@v1.32.0/client.go:529 +0x1b0

[2022-04-01T02:48:49.891Z]   github.com/Shopify/sarama.NewClusterAdminFromClient()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/!shopify/sarama@v1.32.0/admin.go:159 +0x3b

[2022-04-01T02:48:49.891Z]   github.com/Shopify/sarama.NewClusterAdmin()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/!shopify/sarama@v1.32.0/admin.go:152 +0x84

[2022-04-01T02:48:49.891Z]   github.com/pingcap/tiflow/pkg/kafka.NewSaramaAdminClient()

[2022-04-01T02:48:49.891Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/pkg/kafka/cluster_admin_client.go:40 +0x64

[2022-04-01T02:48:49.891Z]   github.com/pingcap/tiflow/cdc/sink.newKafkaSaramaSink()

[2022-04-01T02:48:49.891Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/sink/mq.go:366 +0x21b

[2022-04-01T02:48:49.891Z]   github.com/pingcap/tiflow/cdc/sink.init.2.func3()

[2022-04-01T02:48:49.891Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/sink/sink.go:116 +0x8c

[2022-04-01T02:48:49.891Z]   github.com/pingcap/tiflow/cdc/sink.New()

[2022-04-01T02:48:49.891Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/sink/sink.go:143 +0x2b1

[2022-04-01T02:48:49.891Z]   github.com/pingcap/tiflow/cdc/sink.Validate()

[2022-04-01T02:48:49.891Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/sink/sink.go:157 +0x144

[2022-04-01T02:48:49.891Z]   github.com/pingcap/tiflow/pkg/cmd/cli.(*createChangefeedOptions).validateSink()

[2022-04-01T02:48:49.891Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/pkg/cmd/cli/cli_changefeed_create.go:381 +0x98

[2022-04-01T02:48:49.891Z]   github.com/pingcap/tiflow/pkg/cmd/cli.(*createChangefeedOptions).run()

[2022-04-01T02:48:49.891Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/pkg/cmd/cli/cli_changefeed_create.go:436 +0x6b8

[2022-04-01T02:48:49.891Z]   github.com/pingcap/tiflow/pkg/cmd/cli.newCmdCreateChangefeed.func1()

[2022-04-01T02:48:49.891Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/pkg/cmd/cli/cli_changefeed_create.go:479 +0x151

[2022-04-01T02:48:49.891Z]   github.com/spf13/cobra.(*Command).execute()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856 +0xad7

[2022-04-01T02:48:49.891Z]   github.com/spf13/cobra.(*Command).ExecuteC()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974 +0x5c9

[2022-04-01T02:48:49.891Z]   github.com/spf13/cobra.(*Command).Execute()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902 +0x2fd

[2022-04-01T02:48:49.891Z]   github.com/pingcap/tiflow/pkg/cmd.Run()

[2022-04-01T02:48:49.891Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/pkg/cmd/cmd.go:50 +0x300

[2022-04-01T02:48:49.891Z]   github.com/pingcap/tiflow/cmd/cdc.main()

[2022-04-01T02:48:49.891Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cmd/cdc/main.go:27 +0x33

[2022-04-01T02:48:49.891Z]   github.com/pingcap/tiflow/cmd/cdc.TestRunMain()

[2022-04-01T02:48:49.891Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cmd/cdc/main_test.go:35 +0xca

[2022-04-01T02:48:49.891Z]   testing.tRunner()

[2022-04-01T02:48:49.891Z]       /usr/local/go/src/testing/testing.go:1439 +0x213

[2022-04-01T02:48:49.891Z]   testing.(*T).Run.func1()

[2022-04-01T02:48:49.891Z]       /usr/local/go/src/testing/testing.go:1486 +0x47

[2022-04-01T02:48:49.891Z] 

[2022-04-01T02:48:49.891Z] Goroutine 181 (finished) created at:

[2022-04-01T02:48:49.891Z]   github.com/Shopify/sarama.(*Broker).Open()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/!shopify/sarama@v1.32.0/broker.go:172 +0x3f0

[2022-04-01T02:48:49.891Z]   github.com/Shopify/sarama.(*client).any()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/!shopify/sarama@v1.32.0/client.go:686 +0x144

[2022-04-01T02:48:49.891Z]   github.com/Shopify/sarama.(*client).tryRefreshMetadata()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/!shopify/sarama@v1.32.0/client.go:879 +0x17e

[2022-04-01T02:48:49.891Z]   github.com/Shopify/sarama.(*client).RefreshMetadata()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/!shopify/sarama@v1.32.0/client.go:489 +0x20f

[2022-04-01T02:48:49.891Z]   github.com/Shopify/sarama.NewClient()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/!shopify/sarama@v1.32.0/client.go:170 +0x476

[2022-04-01T02:48:49.891Z]   github.com/Shopify/sarama.NewClusterAdmin()

[2022-04-01T02:48:49.891Z]       /go/pkg/mod/github.com/!shopify/sarama@v1.32.0/admin.go:148 +0x4e

[2022-04-01T02:48:49.891Z]   github.com/pingcap/tiflow/pkg/kafka.NewSaramaAdminClient()

[2022-04-01T02:48:49.891Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/pkg/kafka/cluster_admin_client.go:40 +0x64

[2022-04-01T02:48:49.891Z]   github.com/pingcap/tiflow/cdc/sink.newKafkaSaramaSink()

[2022-04-01T02:48:49.891Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/sink/mq.go:366 +0x21b

[2022-04-01T02:48:49.891Z]   github.com/pingcap/tiflow/cdc/sink.init.2.func3()

[2022-04-01T02:48:49.891Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/sink/sink.go:116 +0x8c

[2022-04-01T02:48:49.891Z]   github.com/pingcap/tiflow/cdc/sink.New()

[2022-04-01T02:48:49.891Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/sink/sink.go:143 +0x2b1

[2022-04-01T02:48:49.891Z]   github.com/pingcap/tiflow/cdc/sink.Validate()

[2022-04-01T02:48:49.891Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cdc/sink/sink.go:157 +0x144

[2022-04-01T02:48:49.892Z]   github.com/pingcap/tiflow/pkg/cmd/cli.(*createChangefeedOptions).validateSink()

[2022-04-01T02:48:49.892Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/pkg/cmd/cli/cli_changefeed_create.go:381 +0x98

[2022-04-01T02:48:49.892Z]   github.com/pingcap/tiflow/pkg/cmd/cli.(*createChangefeedOptions).run()

[2022-04-01T02:48:49.892Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/pkg/cmd/cli/cli_changefeed_create.go:436 +0x6b8

[2022-04-01T02:48:49.892Z]   github.com/pingcap/tiflow/pkg/cmd/cli.newCmdCreateChangefeed.func1()

[2022-04-01T02:48:49.892Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/pkg/cmd/cli/cli_changefeed_create.go:479 +0x151

[2022-04-01T02:48:49.892Z]   github.com/spf13/cobra.(*Command).execute()

[2022-04-01T02:48:49.892Z]       /go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856 +0xad7

[2022-04-01T02:48:49.892Z]   github.com/spf13/cobra.(*Command).ExecuteC()

[2022-04-01T02:48:49.892Z]       /go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974 +0x5c9

[2022-04-01T02:48:49.892Z]   github.com/spf13/cobra.(*Command).Execute()

[2022-04-01T02:48:49.892Z]       /go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902 +0x2fd

[2022-04-01T02:48:49.892Z]   github.com/pingcap/tiflow/pkg/cmd.Run()

[2022-04-01T02:48:49.892Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/pkg/cmd/cmd.go:50 +0x300

[2022-04-01T02:48:49.892Z]   github.com/pingcap/tiflow/cmd/cdc.main()

[2022-04-01T02:48:49.892Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cmd/cdc/main.go:27 +0x33

[2022-04-01T02:48:49.892Z]   github.com/pingcap/tiflow/cmd/cdc.TestRunMain()

[2022-04-01T02:48:49.892Z]       /home/jenkins/agent/workspace/cdc_ghpr_kafka_integration_test/go/src/github.com/pingcap/tiflow/cmd/cdc/main_test.go:35 +0xca

[2022-04-01T02:48:49.892Z]   testing.tRunner()

[2022-04-01T02:48:49.892Z]       /usr/local/go/src/testing/testing.go:1439 +0x213

[2022-04-01T02:48:49.892Z]   testing.(*T).Run.func1()

[2022-04-01T02:48:49.892Z]       /usr/local/go/src/testing/testing.go:1486 +0x47

[2022-04-01T02:48:49.892Z] ==================

[2022-04-01T02:48:49.892Z] Create changefeed successfully!

[2022-04-01T02:48:49.892Z] ID: custom-changefeed-name

[2022-04-01T02:48:49.892Z] Info: {"sink-uri":"kafka://127.0.0.1:9092/ticdc-cli-test-16814?protocol=open-protocol\u0026partition-num=4\u0026kafka-version=2.4.1\u0026max-message-bytes=10485760","opts":{},"create-time":"2022-04-01T10:48:49.704160167+08:00","start-ts":432218131581632513,"target-ts":0,"admin-job-type":0,"sort-engine":"memory","sort-dir":"","config":{"case-sensitive":true,"enable-old-value":true,"force-replicate":false,"check-gc-safe-point":true,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null},"mounter":{"worker-num":16},"sink":{"dispatchers":null,"protocol":"open-protocol","column-selectors":null},"cyclic-replication":{"enable":false,"replica-id":0,"filter-replica-ids":null,"id-buckets":0,"sync-ddl":false},"scheduler":{"type":"table-number","polling-time":-1},"consistent":{"level":"none","max-log-size":64,"flush-interval":1000,"storage":""}},"state":"normal","error":null,"sync-point-enabled":false,"sync-point-interval":600000000000,"creator-version":"v6.1.0-alpha-51-g448da119"}

[2022-04-01T02:48:49.892Z] --- FAIL: TestRunMain (0.48s)

[2022-04-01T02:48:49.892Z]     testing.go:1312: race detected during execution of test

[2022-04-01T02:48:49.892Z] FAIL

[2022-04-01T02:48:50.454Z] coverage: 3.3% of statements in github.com/pingcap/tiflow/...

[2022-04-01T02:49:12.367Z] make: *** [integration_test_kafka] Error 1

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)