Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DATA RACE TestReorgPartExtensivePart #51845

Open
Tracked by #41316
hawkingrei opened this issue Mar 17, 2024 · 3 comments · May be fixed by #53017
Open
Tracked by #41316

DATA RACE TestReorgPartExtensivePart #51845

hawkingrei opened this issue Mar 17, 2024 · 3 comments · May be fixed by #53017
Assignees
@mjonss
Labels
affects-7.5 affects-8.1 component/tablepartition This issue is related to Table Partition of TiDB. component/test severity/major sig/sql-infra SIG: SQL Infra type/bug This issue is a bug.

Comments

@hawkingrei
Copy link
Member

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

==================
WARNING: DATA RACE
Read at 0x00c01e825198 by goroutine 36414:
  github.com/pingcap/tidb/pkg/table/tables.(*TableCommon).Meta()
      pkg/table/tables/tables.go:265 +0x3b
  github.com/pingcap/tidb/pkg/table/tables.(*partitionedTable).Meta()
      <autogenerated>:1 +0x17
  github.com/pingcap/tidb/pkg/ttl/ttlworker.(*TTLTimersSyncer).SyncTimers()
      pkg/ttl/ttlworker/timer_sync.go:189 +0x1859
  github.com/pingcap/tidb/pkg/ttl/ttlworker.(*JobManager).onTimerTick()
      pkg/ttl/ttlworker/job_manager.go:305 +0x409
  github.com/pingcap/tidb/pkg/ttl/ttlworker.(*JobManager).jobLoop()
      pkg/ttl/ttlworker/job_manager.go:204 +0x126d
  github.com/pingcap/tidb/pkg/ttl/ttlworker.(*JobManager).jobLoop-fm()
      <autogenerated>:1 +0x33
  github.com/pingcap/tidb/pkg/ttl/ttlworker.(*baseWorker).loop()
      pkg/ttl/ttlworker/worker.go:134 +0xbb
  github.com/pingcap/tidb/pkg/ttl/ttlworker.(*baseWorker).loop-fm()
      <autogenerated>:1 +0x33
  github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1()
      pkg/util/wait_group_wrapper.go:157 +0x86
Previous write at 0x00c01e825198 by goroutine 146784:
  github.com/pingcap/tidb/pkg/table/tables.SwapReorgPartFields()
      pkg/table/tables/testutil.go:30 +0x25e
  pkg/table/tables/test/partition/partition_test.checkDMLInAllStates.func1()
      pkg/table/tables/test/partition/partition_test.go:2639 +0x15fb
  pkg/table/tables/test/partition/partition_test.(*testCallback).OnJobRunBefore()
      pkg/table/tables/test/partition/partition_test.go:2087 +0x65
  github.com/pingcap/tidb/pkg/ddl.(*worker).HandleDDLJobTable()
      pkg/ddl/ddl_worker.go:926 +0x33a
  github.com/pingcap/tidb/pkg/ddl.(*ddl).delivery2Worker.func1()
      pkg/ddl/job_table.go:462 +0x805
  github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1()
      pkg/util/wait_group_wrapper.go:157 +0x86
Goroutine 36414 (running) created at:
  github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run()
      pkg/util/wait_group_wrapper.go:155 +0xe4
  github.com/pingcap/tidb/pkg/ttl/ttlworker.(*baseWorker).Start()
      pkg/ttl/ttlworker/worker.go:71 +0x12e
  github.com/pingcap/tidb/pkg/domain.(*Domain).StartTTLJobManager()
      pkg/domain/domain.go:2937 +0x1c9
  github.com/pingcap/tidb/pkg/session.bootstrapSessionImpl()
      pkg/session/session.go:3493 +0x1b51
  github.com/pingcap/tidb/pkg/domain.(*Domain).rebuildSysVarCache()
      pkg/domain/sysvar_cache.go:147 +0x974
  github.com/pingcap/tidb/pkg/sessionctx/variable.glob..func202()
      pkg/sessionctx/variable/sysvar.go:1035 +0x58
  github.com/pingcap/tidb/pkg/sessionctx/variable.(*SysVar).ValidateWithRelaxedValidation()
      pkg/sessionctx/variable/variable.go:365 +0x210
  github.com/pingcap/tidb/pkg/domain.(*Domain).rebuildSysVarCache()
      pkg/domain/sysvar_cache.go:146 +0x904
  github.com/pingcap/tidb/pkg/domain.(*Domain).LoadSysVarCacheLoop()
      pkg/domain/domain.go:1730 +0x93
  github.com/pingcap/tidb/pkg/session.bootstrapSessionImpl()
      pkg/session/session.go:3386 +0x844
  github.com/pingcap/tidb/pkg/domain.(*Domain).GetSessionCache()
      pkg/domain/sysvar_cache.go:62 +0x50
  github.com/pingcap/tidb/pkg/session.(*session).loadCommonGlobalVariablesIfNeeded()
      pkg/session/session.go:3754 +0x2ce
  github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt()
      pkg/session/session.go:2107 +0x184
  github.com/pingcap/tidb/pkg/session.(*session).ExecuteInternal()
      pkg/session/session.go:1622 +0x3b9
  github.com/pingcap/tidb/pkg/domain.(*Domain).LoadPrivilegeLoop()
      pkg/domain/domain.go:1673 +0x108
  github.com/pingcap/tidb/pkg/session.bootstrapSessionImpl()
      pkg/session/session.go:3379 +0x7e4
  github.com/pingcap/tidb/pkg/session.BootstrapSession()
      pkg/session/session.go:3286 +0xad
  github.com/pingcap/tidb/pkg/testkit.bootstrap()
      pkg/testkit/mockstore.go:290 +0x94
  github.com/pingcap/tidb/pkg/testkit.CreateMockStoreAndDomain()
      pkg/testkit/mockstore.go:263 +0xd5
  github.com/pingcap/tidb/pkg/testkit.CreateMockStore()
      pkg/testkit/mockstore.go:71 +0x3bd
  pkg/table/tables/test/partition/partition_test.TestReorgPartExtensivePart()
      pkg/table/tables/test/partition/partition_test.go:2434 +0x4c
  testing.tRunner()
      GOROOT/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1648 +0x44
Goroutine 146784 (running) created at:
  github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run()
      pkg/util/wait_group_wrapper.go:155 +0xe4
  github.com/pingcap/tidb/pkg/ddl.(*ddl).delivery2Worker()
      pkg/ddl/job_table.go:422 +0x1b3
  github.com/pingcap/tidb/pkg/ddl.(*ddl).loadDDLJobAndRun()
      pkg/ddl/job_table.go:378 +0x4ef
  github.com/pingcap/tidb/pkg/ddl.(*ddl).startDispatchLoop()
      pkg/ddl/job_table.go:313 +0xe55
  github.com/pingcap/tidb/pkg/ddl.(*ddl).startDispatchLoop-fm()
      <autogenerated>:1 +0x33
  github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1()
      pkg/util/wait_group_wrapper.go:157 +0x86
==================

4. What is your TiDB version? (Required)
@hawkingrei hawkingrei added the type/bug This issue is a bug. label Mar 17, 2024
@hawkingrei hawkingrei mentioned this issue Mar 17, 2024
Open
@aytrack aytrack added the component/tablepartition This issue is related to Table Partition of TiDB. label Mar 18, 2024
@djshow832
Copy link
Contributor

The author of TestReorgPartExtensivePart is @mjonss so I'm assigning it to @mjonss

@mjonss
Copy link
Contributor

mjonss commented Apr 30, 2024

Still reproducable with (cd pkg/table/tables/test/partition ; go test -v -tags intest -race -run TestReorgPartExtensivePart -failfast -count 1000) on linux, using c24dca5

@mjonss mjonss linked a pull request May 2, 2024 that will close this issue
Open
12 tasks
@mjonss
Copy link
Contributor

mjonss commented May 2, 2024

The issue is the way the test works, and not a race in the server code. The test is using a test-only function for swapping the partitioning struct directly on the server global table cache, which then causes a race when the TTL loop is checking each table for syncing their ttl timers.

so I would not consider this as a major bug, since it is a test only, and not very frequently hit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mjonss mjonss

Labels
affects-7.5 affects-8.1 component/tablepartition This issue is related to Table Partition of TiDB. component/test severity/major sig/sql-infra SIG: SQL Infra type/bug This issue is a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ttl,partition: Stabilize test TestReorgPartExtensivePart mjonss/tidb
5 participants
@hawkingrei @mjonss @aytrack @djshow832 @ti-chi-bot