Skip to content

data race in maintainer #4332

@wk989898

Description

@wk989898

What did you do?

run unit test
https://prow.tidb.net/view/gs/prow-tidb-logs/pr-logs/pull/pingcap_ticdc/4284/pull-unit-test/2028739671328034816

What did you expect to see?

No response

What did you see instead?

=== Failed
=== FAIL: maintainer TestMaintainerSchedule (1.30s)
[2026/03/03 15:58:41.514 +08:00] [INFO] [message_center.go:127] ["create message center success."] [id=57063391-03fe-45c1-a970-573941e43860] [addr=]
[2026/03/03 15:58:41.514 +08:00] [INFO] [message_center.go:160] ["Start running message center"] [id=57063391-03fe-45c1-a970-573941e43860] [addr=]
[2026/03/03 15:58:41.514 +08:00] [INFO] [replication_span.go:156] ["init groupID"] [changefeedID=default/test] [groupID=0] [span="tableID: 0, startKey: 7480000000000000ff005f720000000000fa, endKey: 7480000000000000ff005f730000000000fa, keyspaceID: 0"] [totalSpan="tableID: 0, startKey: 7480000000000000ff005f720000000000fa, endKey: 7480000000000000ff005f730000000000fa, keyspaceID: 0"]
[2026/03/03 15:58:41.514 +08:00] [INFO] [replication_span.go:91] ["new working span replication created"] [changefeedID=default/test] [dispatcherID=28315986359672099426989616743703048576] [nodeID=55abbd86-69b8-4a4f-822a-eb41cc2e6389] [checkpointTs=1] [componentStatus=Working] [schemaID=0] [tableID=0] [groupID=0] [start=7480000000000000ff005f720000000000fa] [end=7480000000000000ff005f730000000000fa]
[2026/03/03 15:58:41.515 +08:00] [INFO] [maintainer.go:268] ["changefeed maintainer started"] [changefeedID=default/test] [state=] [checkpointTs=1] [ddlDispatcherID=28315986359672099426989616743703048576] [redoTs="changefeedID:<high:7708876240597217464 low:11764975395684738449 name:\"test\" keyspace:\"default\" > resolvedTs:1 checkpointTs:1 "] [newChangefeed=false]
[2026/03/03 15:58:41.515 +08:00] [INFO] [maintainer_manager.go:270] ["received remove maintainer request"] [changefeedID=default/test]
==================
WARNING: DATA RACE
Read at 0x00c00212e180 by goroutine 688:
  runtime.mapIterStart()
      /usr/local/go/src/runtime/map_swiss.go:160 +0x0
  github.com/pingcap/ticdc/maintainer.(*Maintainer).trySendMaintainerCloseRequestToAllNode()
      /home/prow/go/src/github.com/pingcap/ticdc/maintainer/maintainer.go:995 +0x171
  github.com/pingcap/ticdc/maintainer.(*Maintainer).tryCloseChangefeed()
      /home/prow/go/src/github.com/pingcap/ticdc/maintainer/maintainer.go:988 +0x4d1
  github.com/pingcap/ticdc/maintainer.(*Maintainer).onRemoveMaintainer()
      /home/prow/go/src/github.com/pingcap/ticdc/maintainer/maintainer.go:505 +0x171
  github.com/pingcap/ticdc/maintainer.(*Maintainer).onMessage()
      /home/prow/go/src/github.com/pingcap/ticdc/maintainer/maintainer.go:478 +0x25b
  github.com/pingcap/ticdc/maintainer.(*Maintainer).HandleEvent()
      /home/prow/go/src/github.com/pingcap/ticdc/maintainer/maintainer.go:340 +0x54f
  github.com/pingcap/ticdc/maintainer.(*Maintainer).runHandleEvents()
      /home/prow/go/src/github.com/pingcap/ticdc/maintainer/maintainer.go:1152 +0x2e4
  github.com/pingcap/ticdc/maintainer.NewMaintainer.gowrap1()
      /home/prow/go/src/github.com/pingcap/ticdc/maintainer/maintainer.go:257 +0x4f
Previous write at 0x00c00212e180 by goroutine 647:
  runtime.mapassign_faststr()
      /usr/local/go/src/internal/runtime/maps/runtime_faststr_swiss.go:263 +0x0
  github.com/pingcap/ticdc/maintainer.TestMaintainerSchedule()
      /home/prow/go/src/github.com/pingcap/ticdc/maintainer/maintainer_test.go:296 +0x776
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1934 +0x21c
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1997 +0x44
Goroutine 688 (running) created at:
  github.com/pingcap/ticdc/maintainer.NewMaintainer()
      /home/prow/go/src/github.com/pingcap/ticdc/maintainer/maintainer.go:257 +0x259c
  github.com/pingcap/ticdc/maintainer.NewMaintainerForRemove()
      /home/prow/go/src/github.com/pingcap/ticdc/maintainer/maintainer.go:291 +0x2c4
  github.com/pingcap/ticdc/maintainer.(*Manager).onRemoveMaintainerRequest()
      /home/prow/go/src/github.com/pingcap/ticdc/maintainer/maintainer_manager.go:262 +0x3ab
  github.com/pingcap/ticdc/maintainer.(*Manager).onDispatchMaintainerRequest()
      /home/prow/go/src/github.com/pingcap/ticdc/maintainer/maintainer_manager.go:290 +0x666
  github.com/pingcap/ticdc/maintainer.(*Manager).handleMessage()
      /home/prow/go/src/github.com/pingcap/ticdc/maintainer/maintainer_manager.go:324 +0x14a
  github.com/pingcap/ticdc/maintainer.(*Manager).Run()
      /home/prow/go/src/github.com/pingcap/ticdc/maintainer/maintainer_manager.go:134 +0x2c4
  github.com/pingcap/ticdc/maintainer.TestStopNotExistsMaintainer.func2()
      /home/prow/go/src/github.com/pingcap/ticdc/maintainer/maintainer_manager_test.go:487 +0x46
Goroutine 647 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:1997 +0x9d2
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:2477 +0x85
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1934 +0x21c
  testing.runTests()
      /usr/local/go/src/testing/testing.go:2475 +0x96c
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:2337 +0xed4
  main.main()
      _testmain.go:125 +0x164
  runtime.main()
      /usr/local/go/src/runtime/proc.go:285 +0x29c
  github.com/cilium/ebpf/link.init()
      /home/prow/go/pkg/mod/github.com/cilium/ebpf@v0.4.0/link/syscalls.go:50 +0x10f
  github.com/cilium/ebpf/link.init()
      /home/prow/go/pkg/mod/github.com/cilium/ebpf@v0.4.0/link/syscalls.go:29 +0xab
  runtime.doInit1()
      /usr/local/go/src/runtime/proc.go:7670 +0xd4
  github.com/cilium/ebpf.init()
      /home/prow/go/pkg/mod/github.com/cilium/ebpf@v0.4.0/syscalls.go:437 +0x248
  github.com/cilium/ebpf.init()
      /home/prow/go/pkg/mod/github.com/cilium/ebpf@v0.4.0/prog.go:421 +0x1e4
  github.com/cilium/ebpf.init()
      /home/prow/go/pkg/mod/github.com/cilium/ebpf@v0.4.0/syscalls.go:419 +0x17b
  github.com/cilium/ebpf.init()
      /home/prow/go/pkg/mod/github.com/cilium/ebpf@v0.4.0/syscalls.go:235 +0x10f
  github.com/cilium/ebpf.init()
      /home/prow/go/pkg/mod/github.com/cilium/ebpf@v0.4.0/syscalls.go:217 +0xab
  runtime.doInit1()
      /usr/local/go/src/runtime/proc.go:7670 +0xd4
  github.com/cilium/ebpf/internal/btf.init()
      /home/prow/go/pkg/mod/github.com/cilium/ebpf@v0.4.0/internal/btf/btf.go:720 +0xa9
==================

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

(paste TiCDC version here)

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions