Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TestMRSystemDatabase timed out under stress race #122363

Closed
msbutler opened this issue Apr 15, 2024 · 1 comment · Fixed by #122386
Closed

TestMRSystemDatabase timed out under stress race #122363

msbutler opened this issue Apr 15, 2024 · 1 comment · Fixed by #122386
Assignees
@arulajmani
Labels
A-testing Testing tools and infrastructure C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). T-kv KV Team

Comments

@msbutler
Copy link
Collaborator

msbutler commented Apr 15, 2024 •
edited by cockroach-jira-scripts
Loading

See flake here. Goroutine dump suggests we're hanging in lock processing.

goroutine 26 [chan send]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).lockProcessing(...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:556
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).DrainQueue(0xc005d28b00, {0xdb07e40, 0x12f72b60}, 0xc0031fa1e0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:1447 +0xe5
github.com/cockroachdb/cockroach/pkg/kv/kvserver.forceScanAndProcess({0xdb07e40, 0x12f72b60}, 0xc002a15808, 0xc005d28b00)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue_helpers_testutil.go:42 +0x188
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).ForceReplicationScanAndProcess(...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue_helpers_testutil.go:55
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).WaitForFullReplication.func2(0xc002a15808)
	github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1451 +0x12b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).VisitStores.func1(0x93c75a0?, 0xc002a15808)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:150 +0x58
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*IntMap).Range(0xc0022f69c8, 0xc014993718)
	github.com/cockroachdb/cockroach/pkg/util/syncutil/int_map.go:385 +0x19d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).VisitStores(0xc0022f6990, 0xc00a649930)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:149 +0x8e
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).WaitForFullReplication(0xc003192708)
	github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1443 +0x5ae
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start(0xc003192708, {0xdb46e50, 0xc003fe01a0})
	github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:456 +0xa38
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster({_, _}, _, {{{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}, ...}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:238 +0x92
github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregionccltestutils.TestingCreateMultiRegionClusterWithRegionList({_, _}, {_, _, _}, _, {{0x0, 0x0}, {0x0, 0x0}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregionccltestutils/testutils.go:132 +0x295
github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregionccltestutils.TestingCreateMultiRegionCluster({_, _}, _, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregionccltestutils/testutils.go:89 +0x1f0
github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl.TestMrSystemDatabase(0xc003fe01a0)
	github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregion_system_table_test.go:51 +0x25c
testing.tRunner(0xc003fe01a0, 0xa26df60)
	GOROOT/src/testing/testing.go:1689 +0x21f
created by testing.(*T).Run in goroutine 1
	GOROOT/src/testing/testing.go:1742 +0x826

Jira issue: CRDB-37834
@msbutler msbutler added C-test-failure Broken test (automatically or manually discovered). T-kv KV Team branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 labels Apr 15, 2024
@arulajmani
Copy link
Collaborator

Looks like overload. Lots of log lines of the form:

I240412 23:18:22.563234 737 kv/kvserver/store_raft.go:699 ⋮ [T1,Vsystem,n1,s1,r4/1:‹/System{/tsd-tse}›,raft] 340  raft ready handling: 0.53s [append=0.00s, apply=0.41s, , other=0.12s], wrote [append-batch=49 B, apply=323 KiB (1)] pebble stats: [commit-wait 552ns sem 19.78µs]; node might be overloaded

We can skip this under stress race.

@arulajmani arulajmani added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-testing Testing tools and infrastructure and removed branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 labels Apr 15, 2024
@arulajmani arulajmani mentioned this issue Apr 15, 2024
Merged
@arulajmani arulajmani self-assigned this Apr 15, 2024
craig bot pushed a commit that referenced this issue Apr 15, 2024
@arulajmani
Merge #122386
3c643b0 
122386: multiregionccl: skip TestMRSystemDatabase under stress race r=arulajmani a=arulajmani

Closes #122363

Release note: None

Co-authored-by: Arul Ajmani <arulajmani@gmail.com>
@craig craig bot closed this as completed in 7db1338 Apr 15, 2024
michae2 pushed a commit that referenced this issue Jun 3, 2024
@arulajmani @michae2
multiregionccl: skip TestMRSystemDatabase under stress race
78d6cda 
Closes #122363

Release note: None
@michae2 michae2 mentioned this issue Jun 3, 2024
Merged
@michae2 michae2 mentioned this issue Jun 3, 2024
Merged
michae2 pushed a commit to michae2/cockroach that referenced this issue Jun 4, 2024
@arulajmani @michae2
multiregionccl: skip TestMRSystemDatabase under stress race
9a357f7 
Closes cockroachdb#122363

Release note: None
michae2 pushed a commit that referenced this issue Jun 4, 2024
@arulajmani @michae2
multiregionccl: skip TestMRSystemDatabase under stress race
1dfcb9a 
Closes #122363

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arulajmani arulajmani

Labels
A-testing Testing tools and infrastructure C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). T-kv KV Team
Projects
No open projects
KV
Status: Incoming
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

multiregionccl: skip TestMRSystemDatabase under stress race arulajmani/cockroach
2 participants
@arulajmani @msbutler