Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DATA RACE in the sortexec #52281

Closed
hawkingrei opened this issue Apr 1, 2024 · 7 comments · Fixed by #52417
Closed

DATA RACE in the sortexec #52281

hawkingrei opened this issue Apr 1, 2024 · 7 comments · Fixed by #52417
Labels
component/test severity/major sig/execution SIG execution type/bug This issue is a bug.

Comments

@hawkingrei
Copy link
Member

hawkingrei commented Apr 1, 2024 •
edited

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

https://tiprow.hawkingrei.com/view/gs/pingcapprow/logs/tidb_data_race/1774651064025354240#1:build-log.txt%3A11511

==================
WARNING: DATA RACE
Read at 0x00c0015701d0 by goroutine 111:
  github.com/pingcap/tidb/pkg/executor/sortexec.(*parallelSortSpillHelper).spill.func2()
      pkg/executor/sortexec/parallel_sort_spill_helper.go:127 +0x164
  github.com/pingcap/tidb/pkg/executor/sortexec.(*parallelSortSpillHelper).spill.func3()
      pkg/executor/sortexec/parallel_sort_spill_helper.go:134 +0x41
Previous write at 0x00c0015701d0 by goroutine 62:
  github.com/pingcap/tidb/pkg/executor/sortexec.(*SortExec).fetchChunksParallel()
      pkg/executor/sortexec/sort.go:678 +0x784
  github.com/pingcap/tidb/pkg/executor/sortexec.(*SortExec).fetchChunks()
      pkg/executor/sortexec/sort.go:535 +0x92
  github.com/pingcap/tidb/pkg/executor/sortexec.(*SortExec).Next()
      pkg/executor/sortexec/sort.go:277 +0x339
  github.com/pingcap/tidb/pkg/executor/sortexec_test.executeSortExecutor()
      pkg/executor/sortexec/sort_spill_test.go:178 +0x124
  github.com/pingcap/tidb/pkg/executor/sortexec_test.oneSpillCase()
      pkg/executor/sortexec/parallel_sort_spill_test.go:37 +0x9c
  github.com/pingcap/tidb/pkg/executor/sortexec_test.TestParallelSortSpillDisk()
      pkg/executor/sortexec/parallel_sort_spill_test.go:102 +0x839
  github.com/pingcap/failpoint.parseTerm()
      external/com_github_pingcap_failpoint/terms.go:149 +0x364
  github.com/pingcap/failpoint.parse()
      external/com_github_pingcap_failpoint/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      external/com_github_pingcap_failpoint/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).Enable()
      external/com_github_pingcap_failpoint/failpoint.go:54 +0x3e
  github.com/pingcap/failpoint.(*Failpoints).Enable()
      external/com_github_pingcap_failpoint/failpoints.go:105 +0x296
  github.com/pingcap/failpoint.Enable()
      external/com_github_pingcap_failpoint/failpoints.go:222 +0x231
  github.com/pingcap/tidb/pkg/executor/sortexec_test.TestParallelSortSpillDisk()
      pkg/executor/sortexec/parallel_sort_spill_test.go:86 +0x20d
  github.com/pingcap/failpoint.parseTerm()
      external/com_github_pingcap_failpoint/terms.go:149 +0x364
  github.com/pingcap/failpoint.parse()
      external/com_github_pingcap_failpoint/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      external/com_github_pingcap_failpoint/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).Enable()
      external/com_github_pingcap_failpoint/failpoint.go:54 +0x3e
  github.com/pingcap/failpoint.(*Failpoints).Enable()
      external/com_github_pingcap_failpoint/failpoints.go:105 +0x296
  github.com/pingcap/failpoint.Enable()
      external/com_github_pingcap_failpoint/failpoints.go:222 +0x1e4
  github.com/pingcap/tidb/pkg/executor/sortexec_test.TestParallelSortSpillDisk()
      pkg/executor/sortexec/parallel_sort_spill_test.go:84 +0x1be
  testing.tRunner()
      GOROOT/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1648 +0x44
Goroutine 111 (running) created at:
  github.com/pingcap/tidb/pkg/executor/sortexec.(*parallelSortSpillHelper).spill()
      pkg/executor/sortexec/parallel_sort_spill_helper.go:119 +0x1fc
  github.com/pingcap/tidb/pkg/executor/sortexec.(*SortExec).checkSpillAndExecute()
      pkg/executor/sortexec/sort.go:700 +0x99
  github.com/pingcap/tidb/pkg/executor/sortexec.(*SortExec).fetchChunksFromChild()
      pkg/executor/sortexec/sort.go:752 +0x4c4
  github.com/pingcap/tidb/pkg/executor/sortexec.(*SortExec).fetchChunksParallel.func1()
      pkg/executor/sortexec/sort.go:674 +0x46
  github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1()
      pkg/util/wait_group_wrapper.go:157 +0x86
Goroutine 62 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1648 +0x845
  testing.runTests.func1()
      GOROOT/src/testing/testing.go:2054 +0x84
  testing.tRunner()
      GOROOT/src/testing/testing.go:1595 +0x261
  testing.runTests()
      GOROOT/src/testing/testing.go:2052 +0x8ad
  testing.(*M).Run()
      GOROOT/src/testing/testing.go:1925 +0xcd7
  main.main()
      bazel-out/k8-fastbuild/bin/pkg/executor/sortexec/sortexec_test_/testmain.go:143 +0x5d1
==================
==================
WARNING: DATA RACE
Read at 0x00c0003b1798 by goroutine 111:
  github.com/pingcap/tidb/pkg/executor/sortexec.(*parallelSortWorker).sortLocalRows()
      pkg/executor/sortexec/parallel_sort_worker.go:129 +0x37
  github.com/pingcap/tidb/pkg/executor/sortexec.(*parallelSortSpillHelper).spill.func2()
      pkg/executor/sortexec/parallel_sort_spill_helper.go:127 +0x17d
  github.com/pingcap/tidb/pkg/executor/sortexec.(*parallelSortSpillHelper).spill.func3()
      pkg/executor/sortexec/parallel_sort_spill_helper.go:134 +0x41
Previous write at 0x00c0003b1798 by goroutine 62:
  github.com/pingcap/tidb/pkg/executor/sortexec.newParallelSortWorker()
      pkg/executor/sortexec/parallel_sort_worker.go:66 +0x453
  github.com/pingcap/tidb/pkg/executor/sortexec.(*SortExec).fetchChunksParallel()
      pkg/executor/sortexec/sort.go:678 +0x219
  github.com/pingcap/tidb/pkg/executor/sortexec.(*SortExec).fetchChunks()
      pkg/executor/sortexec/sort.go:535 +0x92
  github.com/pingcap/tidb/pkg/executor/sortexec.(*SortExec).Next()
      pkg/executor/sortexec/sort.go:277 +0x339
  github.com/pingcap/tidb/pkg/executor/sortexec_test.executeSortExecutor()
      pkg/executor/sortexec/sort_spill_test.go:178 +0x124
  github.com/pingcap/tidb/pkg/executor/sortexec_test.oneSpillCase()
      pkg/executor/sortexec/parallel_sort_spill_test.go:37 +0x9c
  github.com/pingcap/tidb/pkg/executor/sortexec_test.TestParallelSortSpillDisk()
      pkg/executor/sortexec/parallel_sort_spill_test.go:102 +0x839
  github.com/pingcap/failpoint.parseTerm()
      external/com_github_pingcap_failpoint/terms.go:149 +0x364
  github.com/pingcap/failpoint.parse()
      external/com_github_pingcap_failpoint/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      external/com_github_pingcap_failpoint/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).Enable()
      external/com_github_pingcap_failpoint/failpoint.go:54 +0x3e
  github.com/pingcap/failpoint.(*Failpoints).Enable()
      external/com_github_pingcap_failpoint/failpoints.go:105 +0x296
  github.com/pingcap/failpoint.Enable()
      external/com_github_pingcap_failpoint/failpoints.go:222 +0x231
  github.com/pingcap/tidb/pkg/executor/sortexec_test.TestParallelSortSpillDisk()
      pkg/executor/sortexec/parallel_sort_spill_test.go:86 +0x20d
  github.com/pingcap/failpoint.parseTerm()
      external/com_github_pingcap_failpoint/terms.go:149 +0x364
  github.com/pingcap/failpoint.parse()
      external/com_github_pingcap_failpoint/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      external/com_github_pingcap_failpoint/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).Enable()
      external/com_github_pingcap_failpoint/failpoint.go:54 +0x3e
  github.com/pingcap/failpoint.(*Failpoints).Enable()
      external/com_github_pingcap_failpoint/failpoints.go:105 +0x296
  github.com/pingcap/failpoint.Enable()
      external/com_github_pingcap_failpoint/failpoints.go:222 +0x1e4
  github.com/pingcap/tidb/pkg/executor/sortexec_test.TestParallelSortSpillDisk()
      pkg/executor/sortexec/parallel_sort_spill_test.go:84 +0x1be
  testing.tRunner()
      GOROOT/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1648 +0x44
Goroutine 111 (running) created at:
  github.com/pingcap/tidb/pkg/executor/sortexec.(*parallelSortSpillHelper).spill()
      pkg/executor/sortexec/parallel_sort_spill_helper.go:119 +0x1fc
  github.com/pingcap/tidb/pkg/executor/sortexec.(*SortExec).checkSpillAndExecute()
      pkg/executor/sortexec/sort.go:700 +0x99
  github.com/pingcap/tidb/pkg/executor/sortexec.(*SortExec).fetchChunksFromChild()
      pkg/executor/sortexec/sort.go:752 +0x4c4
  github.com/pingcap/tidb/pkg/executor/sortexec.(*SortExec).fetchChunksParallel.func1()
      pkg/executor/sortexec/sort.go:674 +0x46
  github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1()
      pkg/util/wait_group_wrapper.go:157 +0x86
Goroutine 62 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1648 +0x845
  testing.runTests.func1()
      GOROOT/src/testing/testing.go:2054 +0x84
  testing.tRunner()
      GOROOT/src/testing/testing.go:1595 +0x261
  testing.runTests()
      GOROOT/src/testing/testing.go:2052 +0x8ad
  testing.(*M).Run()
      GOROOT/src/testing/testing.go:1925 +0xcd7
  main.main()
      bazel-out/k8-fastbuild/bin/pkg/executor/sortexec/sortexec_test_/testmain.go:143 +0x5d1
==================
==================
WARNING: DATA RACE
Read at 0x00c0003b14f0 by goroutine 109:
  github.com/pingcap/tidb/pkg/executor/sortexec.(*parallelSortWorker).multiWayMergeLocalSortedRows()
      pkg/executor/sortexec/parallel_sort_worker.go:98 +0x4c
  github.com/pingcap/tidb/pkg/executor/sortexec.(*parallelSortWorker).sortLocalRows()
      pkg/executor/sortexec/parallel_sort_worker.go:133 +0x55
  github.com/pingcap/tidb/pkg/executor/sortexec.(*parallelSortSpillHelper).spill.func2()
      pkg/executor/sortexec/parallel_sort_spill_helper.go:127 +0x17d
  github.com/pingcap/tidb/pkg/executor/sortexec.(*parallelSortSpillHelper).spill.func3()
      pkg/executor/sortexec/parallel_sort_spill_helper.go:134 +0x41
Previous write at 0x00c0003b14f0 by goroutine 62:
  github.com/pingcap/tidb/pkg/executor/sortexec.newParallelSortWorker()
      pkg/executor/sortexec/parallel_sort_worker.go:66 +0x453
  github.com/pingcap/tidb/pkg/executor/sortexec.(*SortExec).fetchChunksParallel()
      pkg/executor/sortexec/sort.go:678 +0x219
  github.com/pingcap/tidb/pkg/executor/sortexec.(*SortExec).fetchChunks()
      pkg/executor/sortexec/sort.go:535 +0x92
  github.com/pingcap/tidb/pkg/executor/sortexec.(*SortExec).Next()
      pkg/executor/sortexec/sort.go:277 +0x339
  github.com/pingcap/tidb/pkg/executor/sortexec_test.executeSortExecutor()
      pkg/executor/sortexec/sort_spill_test.go:178 +0x124
  github.com/pingcap/tidb/pkg/executor/sortexec_test.oneSpillCase()
      pkg/executor/sortexec/parallel_sort_spill_test.go:37 +0x9c
  github.com/pingcap/tidb/pkg/executor/sortexec_test.TestParallelSortSpillDisk()
      pkg/executor/sortexec/parallel_sort_spill_test.go:102 +0x839
  github.com/pingcap/failpoint.parseTerm()
      external/com_github_pingcap_failpoint/terms.go:149 +0x364
  github.com/pingcap/failpoint.parse()
      external/com_github_pingcap_failpoint/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      external/com_github_pingcap_failpoint/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).Enable()
      external/com_github_pingcap_failpoint/failpoint.go:54 +0x3e
  github.com/pingcap/failpoint.(*Failpoints).Enable()
      external/com_github_pingcap_failpoint/failpoints.go:105 +0x296
  github.com/pingcap/failpoint.Enable()
      external/com_github_pingcap_failpoint/failpoints.go:222 +0x231
  github.com/pingcap/tidb/pkg/executor/sortexec_test.TestParallelSortSpillDisk()
      pkg/executor/sortexec/parallel_sort_spill_test.go:86 +0x20d
  github.com/pingcap/failpoint.parseTerm()
      external/com_github_pingcap_failpoint/terms.go:149 +0x364
  github.com/pingcap/failpoint.parse()
      external/com_github_pingcap_failpoint/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      external/com_github_pingcap_failpoint/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).Enable()
      external/com_github_pingcap_failpoint/failpoint.go:54 +0x3e
  github.com/pingcap/failpoint.(*Failpoints).Enable()
      external/com_github_pingcap_failpoint/failpoints.go:105 +0x296
  github.com/pingcap/failpoint.Enable()
      external/com_github_pingcap_failpoint/failpoints.go:222 +0x1e4
  github.com/pingcap/tidb/pkg/executor/sortexec_test.TestParallelSortSpillDisk()
      pkg/executor/sortexec/parallel_sort_spill_test.go:84 +0x1be
  testing.tRunner()
      GOROOT/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1648 +0x44
Goroutine 109 (running) created at:
  github.com/pingcap/tidb/pkg/executor/sortexec.(*parallelSortSpillHelper).spill()
      pkg/executor/sortexec/parallel_sort_spill_helper.go:119 +0x1fc
  github.com/pingcap/tidb/pkg/executor/sortexec.(*SortExec).checkSpillAndExecute()
      pkg/executor/sortexec/sort.go:700 +0x99
  github.com/pingcap/tidb/pkg/executor/sortexec.(*SortExec).fetchChunksFromChild()
      pkg/executor/sortexec/sort.go:752 +0x4c4
  github.com/pingcap/tidb/pkg/executor/sortexec.(*SortExec).fetchChunksParallel.func1()
      pkg/executor/sortexec/sort.go:674 +0x46
  github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1()
      pkg/util/wait_group_wrapper.go:157 +0x86
Goroutine 62 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1648 +0x845
  testing.runTests.func1()
      GOROOT/src/testing/testing.go:2054 +0x84
  testing.tRunner()
      GOROOT/src/testing/testing.go:1595 +0x261
  testing.runTests()
      GOROOT/src/testing/testing.go:2052 +0x8ad
  testing.(*M).Run()
      GOROOT/src/testing/testing.go:1925 +0xcd7
  main.main()
      bazel-out/k8-fastbuild/bin/pkg/executor/sortexec/sortexec_test_/testmain.go:143 +0x5d1
==================

4. What is your TiDB version? (Required)
@hawkingrei hawkingrei added the type/bug This issue is a bug. label Apr 1, 2024
@yibin87
Copy link
Contributor

yibin87 commented Apr 1, 2024

/cc @xzhangxian1008

This was referenced Apr 8, 2024
Merged
Closed
@xzhangxian1008
Copy link
Contributor

/severity major

@xzhangxian1008 xzhangxian1008 mentioned this issue Apr 12, 2024
Closed
@xzhangxian1008
Copy link
Contributor

/remove-label may-affects-5.4

@ti-chi-bot ti-chi-bot bot removed the may-affects-5.4 This bug maybe affects 5.4.x versions. label Apr 12, 2024
@xzhangxian1008
Copy link
Contributor

/remove-label may-affects-6.1

@xzhangxian1008
Copy link
Contributor

/remove-label may-affects-6.5

@xzhangxian1008
Copy link
Contributor

/remove-label may-affects-7.1

@xzhangxian1008
Copy link
Contributor

/remove-label may-affects-7.5

ti-chi-bot bot pushed a commit that referenced this issue Apr 12, 2024
@xzhangxian1008
executor: fix data race in the sortexec (#52417)
58267e1 
close #52281
3AceShowHand pushed a commit to 3AceShowHand/tidb that referenced this issue Apr 16, 2024
@xzhangxian1008 @3AceShowHand
executor: fix data race in the sortexec (pingcap#52417)
2c499a3 
close pingcap#52281
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component/test severity/major sig/execution SIG execution type/bug This issue is a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

executor: fix data race in the sortexec xzhangxian1008/tidb
5 participants
@hawkingrei @jebter @yibin87 @xzhangxian1008 @ti-chi-bot