Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DATA RACE at the resource_group/controller.(*groupCostController) #49311

Closed
Tracked by #41316
hawkingrei opened this issue Dec 9, 2023 · 5 comments · Fixed by #49715 or tikv/pd#7520
Closed
Tracked by #41316

DATA RACE at the resource_group/controller.(*groupCostController) #49311

hawkingrei opened this issue Dec 9, 2023 · 5 comments · Fixed by #49715 or tikv/pd#7520
Labels
affects-7.5 component/pd severity/minor type/bug This issue is a bug.

Comments

@hawkingrei
Copy link
Member

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

==================
WARNING: DATA RACE
Write at 0x00c005754c80 by goroutine 31038:
  github.com/tikv/pd/client/resource_group/controller.(*groupCostController).modifyMeta()
      external/com_github_tikv_pd_client/resource_group/controller/controller.go:1162 +0x9d
  github.com/tikv/pd/client/resource_group/controller.(*ResourceGroupsController).Start.func1()
      external/com_github_tikv_pd_client/resource_group/controller/controller.go:332 +0x2b44
Previous read at 0x00c005754c80 by goroutine 32356:
  github.com/tikv/pd/client/resource_group/controller.(*groupCostController).onRequestWait()
      external/com_github_tikv_pd_client/resource_group/controller/controller.go:1248 +0xe58
  github.com/tikv/pd/client/resource_group/controller.(*ResourceGroupsController).OnRequestWait()
      external/com_github_tikv_pd_client/resource_group/controller/controller.go:534 +0xcf
  github.com/tikv/client-go/v2/internal/client.buildResourceControlInterceptor.func1.1()
      external/com_github_tikv_client_go_v2/internal/client/client_interceptor.go:119 +0x1cb
  github.com/tikv/client-go/v2/internal/client.interceptedClient.SendRequest()
      external/com_github_tikv_client_go_v2/internal/client/client_interceptor.go:60 +0x2e5
[2023/12/08 23:08:04.652 +08:00] [INFO] [callback.go:111] ["on job run after"] [job="ID:102, Type:alter resource group, State:done, SchemaState:public, SchemaID:1, TableID:0, RowCount:0, ArgLen:1, start time: 2023-12-08 23:08:04.62 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:0"]
  github.com/tikv/client-go/v2/internal/client.(*interceptedClient).SendRequest()
      <autogenerated>:1 +0xb3
  github.com/tikv/client-go/v2/internal/client.reqCollapse.SendRequest()
      external/com_github_tikv_client_go_v2/internal/client/client_collapse.go:74 +0x113
  github.com/tikv/client-go/v2/internal/client.(*reqCollapse).SendRequest()
      <autogenerated>:1 +0xa4
  github.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).sendReqToRegion()
      external/com_github_tikv_client_go_v2/internal/locate/region_request.go:1681 +0xf42
  github.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).SendReqCtx()
      external/com_github_tikv_client_go_v2/internal/locate/region_request.go:1461 +0x2495
  github.com/tikv/client-go/v2/txnkv/txnsnapshot.(*ClientHelper).SendReqCtx()
      external/com_github_tikv_client_go_v2/txnkv/txnsnapshot/client_helper.go:146 +0x48d
  github.com/pingcap/tidb/pkg/store/copr.(*copIteratorWorker).handleTaskOnce()
      pkg/store/copr/coprocessor.go:1240 +0x1969
  github.com/pingcap/tidb/pkg/store/copr.(*copIteratorWorker).handleTask()
      pkg/store/copr/coprocessor.go:1130 +0x225
  github.com/pingcap/tidb/pkg/store/copr.(*copIteratorWorker).run()
      pkg/store/copr/coprocessor.go:817 +0x176
  github.com/pingcap/tidb/pkg/store/copr.(*copIterator).open.func1()
      pkg/store/copr/coprocessor.go:861 +0x4f
Goroutine 31038 (running) created at:
  github.com/tikv/pd/client/resource_group/controller.(*ResourceGroupsController).Start()
      external/com_github_tikv_pd_client/resource_group/controller/controller.go:201 +0x189
  github.com/pingcap/tidb/pkg/domain.(*Domain).initResourceGroupsController()
      pkg/domain/runaway.go:434 +0x185
  github.com/pingcap/tidb/pkg/domain.(*Domain).Init()
      pkg/domain/domain.go:1238 +0x1a14
  github.com/pingcap/tidb/pkg/session.(*domainMap).Get.func1()
      pkg/session/tidb.go:94 +0x6f0
  github.com/pingcap/tidb/pkg/util.RunWithRetry()
      pkg/util/misc.go:69 +0x84
  github.com/pingcap/tidb/pkg/session.(*domainMap).Get()
      pkg/session/tidb.go:80 +0x348
  github.com/pingcap/tidb/pkg/session.createSessionWithOpt()
      pkg/session/session.go:3578 +0x66
  github.com/pingcap/tidb/pkg/session.createSession()
      pkg/session/session.go:3570 +0x35
  github.com/pingcap/tidb/pkg/session.createSessionsImpl()
      pkg/session/session.go:3555 +0x81
  github.com/pingcap/tidb/pkg/session.createSessions()
      pkg/session/session.go:3542 +0x44
  github.com/pingcap/tidb/pkg/session.bootstrapSessionImpl()
      pkg/session/session.go:3317 +0x3a6
  github.com/pingcap/tidb/pkg/session.BootstrapSession()
      pkg/session/session.go:3271 +0xad
  github.com/pingcap/tidb/pkg/testkit.bootstrap()
      pkg/testkit/mockstore.go:227 +0x94
  github.com/pingcap/tidb/pkg/testkit.CreateMockStoreAndDomain()
      pkg/testkit/mockstore.go:200 +0xd5
  pkg/ddl/tests/resourcegroup/resourcegroup_test_test.TestResourceGroupBasic()
      pkg/ddl/tests/resourcegroup/resource_group_test.go:39 +0x4c
  testing.tRunner()
      GOROOT/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1648 +0x44
Goroutine 32356 (running) created at:
  github.com/pingcap/tidb/pkg/store/copr.(*copIterator).open()
      pkg/store/copr/coprocessor.go:861 +0x106
  github.com/pingcap/tidb/pkg/store/copr.(*CopClient).Send()
      pkg/store/copr/coprocessor.go:105 +0x416
  github.com/pingcap/tidb/pkg/distsql.Select()
      pkg/distsql/distsql.go:91 +0x9f5
  github.com/pingcap/tidb/pkg/distsql.SelectWithRuntimeStats()
      pkg/distsql/distsql.go:146 +0xa6
  github.com/pingcap/tidb/pkg/executor.selectResultHook.SelectResult()
      pkg/executor/table_reader.go:62 +0x19d
  github.com/pingcap/tidb/pkg/executor.(*TableReaderExecutor).buildResp()
      pkg/executor/table_reader.go:337 +0x96d
  github.com/pingcap/tidb/pkg/executor.(*TableReaderExecutor).Open()
      pkg/executor/table_reader.go:218 +0xcf4
  github.com/pingcap/tidb/pkg/executor/internal/exec.Open()
      pkg/executor/internal/exec/executor.go:268 +0xb8
  github.com/pingcap/tidb/pkg/executor/internal/exec.(*BaseExecutor).Open()
      pkg/executor/internal/exec/executor.go:167 +0x23b
  github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).Open()
      pkg/executor/aggregate/agg_hash_executor.go:208 +0x195
  github.com/pingcap/tidb/pkg/executor/internal/exec.Open()
      pkg/executor/internal/exec/executor.go:268 +0xb8
  github.com/pingcap/tidb/pkg/executor/internal/exec.(*BaseExecutor).Open()
      pkg/executor/internal/exec/executor.go:167 +0xe8
  github.com/pingcap/tidb/pkg/executor.(*SelectionExec).Open()
      pkg/executor/executor.go:1546 +0x3f
  github.com/pingcap/tidb/pkg/executor/internal/exec.Open()
      pkg/executor/internal/exec/executor.go:268 +0xb8
  github.com/pingcap/tidb/pkg/executor/internal/exec.(*BaseExecutor).Open()
      pkg/executor/internal/exec/executor.go:167 +0x23b
  github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).Open()
      pkg/executor/aggregate/agg_hash_executor.go:208 +0x195
  github.com/pingcap/tidb/pkg/executor/internal/exec.Open()
      pkg/executor/internal/exec/executor.go:268 +0xb8
  github.com/pingcap/tidb/pkg/executor/internal/exec.(*BaseExecutor).Open()
      pkg/executor/internal/exec/executor.go:167 +0xe8
  github.com/pingcap/tidb/pkg/executor.(*HashJoinExec).Open()
      pkg/executor/join.go:204 +0x3f
  github.com/pingcap/tidb/pkg/executor/internal/exec.Open()
      pkg/executor/internal/exec/executor.go:268 +0xb8
  github.com/pingcap/tidb/pkg/executor/sortexec.(*SortExec).Open()
      pkg/executor/sortexec/sort.go:99 +0x34e
  github.com/pingcap/tidb/pkg/executor/internal/exec.Open()
      pkg/executor/internal/exec/executor.go:268 +0xb8
  github.com/pingcap/tidb/pkg/executor/internal/exec.(*BaseExecutor).Open()
      pkg/executor/internal/exec/executor.go:167 +0xe8
  github.com/pingcap/tidb/pkg/executor.(*ProjectionExec).Open()
      pkg/executor/projection.go:88 +0x3f
  github.com/pingcap/tidb/pkg/executor/internal/exec.Open()
      pkg/executor/internal/exec/executor.go:268 +0xb8
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).openExecutor()
      pkg/executor/adapter.go:1240 +0xf4
  github.com/pingcap/tidb/pkg/executor.(*ExecStmt).Exec()
      pkg/executor/adapter.go:567 +0xeb6
  github.com/pingcap/tidb/pkg/session.runStmt()
      pkg/session/session.go:2393 +0x71a
  github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt()
      pkg/session/session.go:2235 +0x1af8
  github.com/pingcap/tidb/pkg/session.(*session).ExecuteInternal()
      pkg/session/session.go:1599 +0x3b9
  github.com/pingcap/tidb/pkg/ddl/internal/session.(*Session).Execute()
      pkg/ddl/internal/session/session.go:85 +0x27e
  github.com/pingcap/tidb/pkg/ddl.(*ddl).getJob()
      pkg/ddl/job_table.go:117 +0x1bb
  github.com/pingcap/tidb/pkg/ddl.(*ddl).getReorgJob()
      pkg/ddl/job_table.go:247 +0x86
  github.com/pingcap/tidb/pkg/ddl.(*ddl).getReorgJob-fm()
      <autogenerated>:1 +0x3d
  github.com/pingcap/tidb/pkg/ddl.(*ddl).loadDDLJobAndRun()
      pkg/ddl/job_table.go:365 +0x408
  github.com/pingcap/tidb/pkg/ddl.(*ddl).startDispatchLoop()
      pkg/ddl/job_table.go:313 +0xe75
  github.com/pingcap/tidb/pkg/ddl.(*ddl).startDispatchLoop-fm()
      <autogenerated>:1 +0x33
  github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1()
      pkg/util/wait_group_wrapper.go:157 +0x86
==================

4. What is your TiDB version? (Required)
@hawkingrei hawkingrei added the type/bug This issue is a bug. label Dec 9, 2023
@hawkingrei hawkingrei changed the title DATA RACE resource_group/controller.(*groupCostController) DATA RACE at the resource_group/controller.(*groupCostController) Dec 9, 2023
@hawkingrei hawkingrei mentioned this issue Dec 9, 2023
Open
@glorv
Copy link
Contributor

glorv commented Dec 11, 2023

@CabinfeverB @nolouch PTAL

@nolouch nolouch mentioned this issue Dec 11, 2023
Merged
@CabinfeverB
Copy link
Contributor

/label affects-7.5

@CabinfeverB
Copy link
Contributor

/label severity/minor

@ti-chi-bot ti-chi-bot
Copy link

ti-chi-bot bot commented Dec 11, 2023

@CabinfeverB: The label(s) severity/minor cannot be applied. These labels are supported: fuzz/sqlancer, challenge-program, compatibility-breaker, first-time-contributor, contribution, good first issue, correctness, duplicate, proposal, security, ok-to-test, needs-ok-to-test, needs-more-info, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.5, needs-cherry-pick-release-7.1, needs-cherry-pick-release-7.5, affects-5.4, affects-6.1, affects-6.5, affects-7.1, affects-7.5, may-affects-5.4, may-affects-6.1, may-affects-6.5, may-affects-7.1, may-affects-7.5.

In response to this:

/label severity/minor

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@CabinfeverB
Copy link
Contributor

/severity minor

ti-chi-bot bot pushed a commit to tikv/pd that referenced this issue Dec 11, 2023
@nolouch
resource_control: fix data race in controller (#7520)
fcdeec4 
close #4399, ref pingcap/tidb#49311

the meta information is not safe to visit concurrency
- address the data race condition

Signed-off-by: Shuning Chen <nolouch@ShuningdeMacBook-Pro.local>

Co-authored-by: Shuning Chen <nolouch@ShuningdeMacBook-Pro.local>
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Dec 11, 2023
@nolouch @ti-chi-bot
This is an automated cherry-pick of tikv#7520
34a49de 
close tikv#4399, ref pingcap/tidb#49311

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this issue Dec 11, 2023
Merged
ti-chi-bot bot pushed a commit to tikv/pd that referenced this issue Dec 13, 2023
@ti-chi-bot @nolouch
resource_control: fix data race in controller (#7520) (#7526)
d2074a9 
close #4399, ref pingcap/tidb#49311

the meta information is not safe to visit concurrency
- address the data race condition

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: nolouch <nolouch@gmail.com>

Co-authored-by: ShuNing <nolouch@gmail.com>
Co-authored-by: nolouch <nolouch@gmail.com>
@hawkingrei hawkingrei mentioned this issue Dec 22, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Dec 25, 2023
@hawkingrei
*: upgrade client-go (#49715)
660b1f3 
close #49311
@ti-chi-bot ti-chi-bot mentioned this issue Dec 25, 2023
Closed
13 tasks
@nolouch nolouch mentioned this issue Jan 3, 2024
Merged
5 tasks
ti-chi-bot bot pushed a commit that referenced this issue Jan 4, 2024
@nolouch
mod: upgrade pd client to latest release-7.5 (#50034)
63e621a 
close #49311
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-7.5 component/pd severity/minor type/bug This issue is a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

*: upgrade client-go hawkingrei/tidb
resource_control: fix data race in controller nolouch/pd
4 participants
@hawkingrei @jebter @glorv @CabinfeverB