roachtest: loqrecovery/half-online/workload=tpcc/rangeSize=16mb failed #97912

cockroach-teamcity · 2023-03-02T11:15:56Z

roachtest.loqrecovery/half-online/workload=tpcc/rangeSize=16mb failed with artifacts on master @ 20e2adda3c76c7172dd986c871df0ae9a346918f:

test artifacts and logs in: /artifacts/loqrecovery/half-online/workload=tpcc/rangeSize=16mb/run_1
(cluster.go:1387).FailOnInvalidDescriptors: invalid descriptors check failed: pq: query execution canceled due to statement timeout

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/replication _{This test on roachdash | Improve this report!

Jira issue: CRDB-24961}

The text was updated successfully, but these errors were encountered:

pav-kv · 2023-03-06T17:35:23Z

teardownTest fails after timing out on trying to read from crdb_internal.invalid_objects. Can this timeout be related to the fact that there is a loss of quorum or somesuch? (the test is about LoQ)

aliher1911 · 2023-03-10T10:57:51Z

@herkolategan in #96949 you added a mandatory consistency check to the end of roachtest. Quorum recovery tests often leave cluster broken if it is impossible to recover all replicas.

Any ideas on how to make that check optional if test knows recovery didn't succeed? Unsuccessful recovery is a pass for the test.

herkolategan · 2023-03-16T20:06:40Z

@aliher1911 Thanks for pointing this out! I'll create an issue to add something to be able to opt out of the mandatory consistency check.

A few post validations are currently performed after a test completes. In most cases it applies to all tests, but there are some tests that are incompatible with particular post validations. This change adds the ability for a test to specify that it wants to opt-out of certain validations. Refs: cockroachdb#97912 Epic: None

98848: roachtest: allow opting out of post validation(s) r=aliher1911 a=herkolategan A few post validations are currently performed after a test completes. In most cases it applies to all tests, but there are some tests that are incompatible with particular post validations. This change adds the ability for a test to specify that it wants to opt-out of certain validations. Refs: #97912 Epic: None Co-authored-by: Herko Lategan <herko@cockroachlabs.com>

A few post validations are currently performed after a test completes. In most cases it applies to all tests, but there are some tests that are incompatible with particular post validations. This change adds the ability for a test to specify that it wants to opt-out of certain validations. Refs: cockroachdb#97912 Epic: None

cockroach-teamcity · 2023-03-25T16:40:47Z

roachtest.loqrecovery/half-online/workload=tpcc/rangeSize=16mb failed with artifacts on master @ 0fcc33bc2870961b9387999d8a9fed97fccbb2ae:

test artifacts and logs in: /artifacts/loqrecovery/half-online/workload=tpcc/rangeSize=16mb/run_1
(assertions.go:262).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/loss_of_quorum_recovery.go:418
	            				main/pkg/cmd/roachtest/monitor.go:105
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
	            				GOROOT/src/runtime/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	pq: could not validate zone config: RangeMaxBytes 16777216 less than minimum allowed 67108864
	Test:       	loqrecovery/half-online/workload=tpcc/rangeSize=16mb
	Messages:   	failed to set range limits configuration
(require.go:1264).NoError: FailNow called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-03-26T16:37:00Z

roachtest.loqrecovery/half-online/workload=tpcc/rangeSize=16mb failed with artifacts on master @ d107217dac5d817cc115bc0e97b7e53c0f2878bf:

test artifacts and logs in: /artifacts/loqrecovery/half-online/workload=tpcc/rangeSize=16mb/run_1
(assertions.go:262).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/loss_of_quorum_recovery.go:418
	            				main/pkg/cmd/roachtest/monitor.go:105
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
	            				GOROOT/src/runtime/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	pq: could not validate zone config: RangeMaxBytes 16777216 less than minimum allowed 67108864
	Test:       	loqrecovery/half-online/workload=tpcc/rangeSize=16mb
	Messages:   	failed to set range limits configuration
(require.go:1264).NoError: FailNow called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

aliher1911 · 2023-03-27T09:03:35Z

Looks like this is a completely new issue with changing range size.

erikgrinaker · 2023-03-27T09:04:36Z

Yes, see #96725.

cockroach-teamcity · 2023-03-27T11:02:07Z

roachtest.loqrecovery/half-online/workload=tpcc/rangeSize=16mb failed with artifacts on master @ 2bd2c806ab3044569b09e0a205b5bc0452ad4e2b:

test artifacts and logs in: /artifacts/loqrecovery/half-online/workload=tpcc/rangeSize=16mb/run_1
(assertions.go:262).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/loss_of_quorum_recovery.go:418
	            				main/pkg/cmd/roachtest/monitor.go:105
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
	            				GOROOT/src/runtime/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	pq: could not validate zone config: RangeMaxBytes 16777216 less than minimum allowed 67108864
	Test:       	loqrecovery/half-online/workload=tpcc/rangeSize=16mb
	Messages:   	failed to set range limits configuration
(require.go:1264).NoError: FailNow called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-03-28T16:45:25Z

roachtest.loqrecovery/half-online/workload=tpcc/rangeSize=16mb failed with artifacts on master @ 143b63a6a27aeb286637dd2e5abddafdf0c51874:

test artifacts and logs in: /artifacts/loqrecovery/half-online/workload=tpcc/rangeSize=16mb/run_1
(assertions.go:262).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/loss_of_quorum_recovery.go:418
	            				main/pkg/cmd/roachtest/monitor.go:105
	            				golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
	            				GOROOT/src/runtime/asm_amd64.s:1594
	Error:      	Received unexpected error:
	            	pq: could not validate zone config: RangeMaxBytes 16777216 less than minimum allowed 67108864
	Test:       	loqrecovery/half-online/workload=tpcc/rangeSize=16mb
	Messages:   	failed to set range limits configuration
(require.go:1264).NoError: FailNow called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

aliher1911 · 2023-03-29T13:51:18Z

Subsequent failure is now fixed by #99636. All good.

cockroach-teamcity added branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-kv-replication KV Replication Team labels Mar 2, 2023

cockroach-teamcity added this to the 23.1 milestone Mar 2, 2023

pav-kv assigned aliher1911 Mar 6, 2023

pav-kv added A-kv-replication Relating to Raft, consensus, and coordination. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-testing Testing tools and infrastructure labels Mar 6, 2023

aliher1911 mentioned this issue Mar 10, 2023

roachtest: add failure message to test if post validations fail #98366

Closed

herkolategan mentioned this issue Mar 17, 2023

roachtest: allow opting out of post validation(s) #98848

Merged

herkolategan mentioned this issue Mar 23, 2023

release-23.1: roachtest: allow opting out of post validation(s) #99398

Merged

aliher1911 mentioned this issue Mar 27, 2023

roachtest: loq recovery use new min range logic #99636

Merged

aliher1911 closed this as completed Mar 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: loqrecovery/half-online/workload=tpcc/rangeSize=16mb failed #97912

roachtest: loqrecovery/half-online/workload=tpcc/rangeSize=16mb failed #97912

cockroach-teamcity commented Mar 2, 2023 •

edited by cockroach-jira-scripts

pav-kv commented Mar 6, 2023 •

edited

aliher1911 commented Mar 10, 2023

herkolategan commented Mar 16, 2023

cockroach-teamcity commented Mar 25, 2023

cockroach-teamcity commented Mar 26, 2023

aliher1911 commented Mar 27, 2023

erikgrinaker commented Mar 27, 2023

cockroach-teamcity commented Mar 27, 2023

cockroach-teamcity commented Mar 28, 2023

aliher1911 commented Mar 29, 2023

roachtest: loqrecovery/half-online/workload=tpcc/rangeSize=16mb failed #97912

roachtest: loqrecovery/half-online/workload=tpcc/rangeSize=16mb failed #97912

Comments

cockroach-teamcity commented Mar 2, 2023 • edited by cockroach-jira-scripts

pav-kv commented Mar 6, 2023 • edited

aliher1911 commented Mar 10, 2023

herkolategan commented Mar 16, 2023

cockroach-teamcity commented Mar 25, 2023

cockroach-teamcity commented Mar 26, 2023

aliher1911 commented Mar 27, 2023

erikgrinaker commented Mar 27, 2023

cockroach-teamcity commented Mar 27, 2023

cockroach-teamcity commented Mar 28, 2023

aliher1911 commented Mar 29, 2023

cockroach-teamcity commented Mar 2, 2023 •

edited by cockroach-jira-scripts

pav-kv commented Mar 6, 2023 •

edited