roachtest: c2c/BulkOps/settings=none failed

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.c2c/BulkOps/settings=none failed with artifacts on release-26.1.1-rc @ 5abbbd24fcc2c7873fb0f985ccea68b7abaa6afe:

Failed with:

(assertions.go:369).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/backup.go:644
	            				pkg/cmd/roachtest/tests/backup.go:760
	            				pkg/cmd/roachtest/tests/cluster_to_cluster.go:433
	            				pkg/cmd/roachtest/tests/cluster_to_cluster.go:774
	            				pkg/cmd/roachtest/tests/cluster_to_cluster.go:1164
	            				pkg/cmd/roachtest/monitor.go:233
	            				external/org_golang_x_sync/errgroup/errgroup.go:93
	            				src/runtime/asm_amd64.s:1693
	Error:      	Received unexpected error:
	            	expected job state canceled, but got reverting
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runBackupImportRollback.func1.1
	            	  | 	pkg/cmd/roachtest/tests/backup.go:656
	            	  | github.com/cockroachdb/cockroach/pkg/util/retry.Options.Do
	            	  | 	pkg/util/retry/retry.go:298
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runBackupImportRollback.func1
	            	  | 	pkg/cmd/roachtest/tests/backup.go:644
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runBackupImportRollback
	            	  | 	pkg/cmd/roachtest/tests/backup.go:760
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.replicateBulkOps.runDriver
	            	  | 	pkg/cmd/roachtest/tests/cluster_to_cluster.go:433
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).runWorkload
	            	  | 	pkg/cmd/roachtest/tests/cluster_to_cluster.go:774
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main.func2
	            	  | 	pkg/cmd/roachtest/tests/cluster_to_cluster.go:1164
	            	  | main.(*monitorImpl).Go.func1
	            	  | 	pkg/cmd/roachtest/monitor.go:233
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:93
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1693
	            	Wraps: (2) expected job state canceled, but got reverting
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	c2c/BulkOps/settings=none
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/c2c/BulkOps/settings=none/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21245146-1773383045-23-n9cpu8-0001	34.73.17.221	10.142.1.255
teamcity-21245146-1773383045-23-n9cpu8-0002	34.24.172.33	10.142.0.211
teamcity-21245146-1773383045-23-n9cpu8-0003	35.185.126.67	10.142.1.18
teamcity-21245146-1773383045-23-n9cpu8-0004	34.24.107.177	10.142.0.236
teamcity-21245146-1773383045-23-n9cpu8-0005	34.75.112.137	10.142.1.23
teamcity-21245146-1773383045-23-n9cpu8-0006	34.23.199.14	10.142.1.34
teamcity-21245146-1773383045-23-n9cpu8-0007	34.73.205.232	10.142.1.24
teamcity-21245146-1773383045-23-n9cpu8-0008	35.190.164.18	10.142.1.22
teamcity-21245146-1773383045-23-n9cpu8-0009	34.24.151.147	10.142.3.79

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=8
• diskCount=0
• encrypted=false
• fs=ext4
• localSSD=false
• metamorphicLeases=expiration
• runtimeAssertionsBuild=true

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: c2c/BulkOps/settings=none failed #161366 roachtest: c2c/BulkOps/settings=none failed [A-disaster-recovery B-runtime-assertions-enabled C-test-failure O-roachtest O-robot T-disaster-recovery branch-master]

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-61550

[blathers-crl]

The release-blocker label has been automatically removed because this issue was assigned priority P-2. Only P-0 and P-1 issues are considered release blockers.