roachtest: acceptance/gossip/restart-node-one failed [slow replication] #68107

cockroach-teamcity · 2021-07-27T14:21:37Z

roachtest.acceptance/gossip/restart-node-one failed with artifacts on master @ 1e0daf805e2f0559add9a8a637c2338b478beeab:

The test failed on branch=master, cloud=local:
test artifacts and logs in: artifacts/acceptance/gossip/restart-node-one/run_1
	gossip.go:435,acceptance.go:104,test_runner.go:770: node 1 still has 1 replicas
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne.func3
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:426
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:415
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAcceptance.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/acceptance.go:104
		  | main.(*testRunner).runTest.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:770
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) node 1 still has 1 replicas
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Reproduce

See the corresponding section in the [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest)

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

The text was updated successfully, but these errors were encountered:

tbg · 2021-07-27T18:46:18Z

@AlexTalks could you take a look at this one? The link above should have "usable" artifacts. I'd start by downloading them (make sure you download only those for the test, not those for all tests - just the two zip files), and then trying to figure out what the test wants to do (code is be in ./pkg/roachtest/tests/ somewhere, searching for "restart-node-one" is going to be an easy way to find it), and what goes wrong. From the looks of it at some point in the test all replicas should move off a node but didn't, so you'll probably want to figure out which replica didn't (the debug.zip might be helpful - nodes/1/ranges/*.json), and then figure out from the logs (cockroach debug merge-logs . | grep -F 'rXXX/' where XXX is the rangeID) what might've happened.
Let me (or if I'm out, someone else) know if/where you get stuck!

cockroach-teamcity · 2021-07-27T19:50:04Z

roachtest.acceptance/gossip/restart-node-one failed with artifacts on master @ 4718beb99f1b90b4f64e769eb3d1a7dd64e51f64:

The test failed on branch=master, cloud=local:
test artifacts and logs in: artifacts/acceptance/gossip/restart-node-one/run_1
	gossip.go:435,acceptance.go:104,test_runner.go:770: node 1 still has 1 replicas
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne.func3
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:426
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:415
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAcceptance.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/acceptance.go:104
		  | main.(*testRunner).runTest.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:770
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) node 1 still has 1 replicas
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Reproduce

See the corresponding section in the [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest)

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-07-27T20:52:40Z

roachtest.acceptance/gossip/restart-node-one failed with artifacts on master @ 4718beb99f1b90b4f64e769eb3d1a7dd64e51f64:

The test failed on branch=master, cloud=local:
test artifacts and logs in: artifacts/acceptance/gossip/restart-node-one/run_1
	gossip.go:435,acceptance.go:104,test_runner.go:770: node 1 still has 2 replicas
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne.func3
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:426
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:415
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAcceptance.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/acceptance.go:104
		  | main.(*testRunner).runTest.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:770
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) node 1 still has 2 replicas
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Reproduce

See the corresponding section in the [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest)

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-07-27T23:46:09Z

roachtest.acceptance/gossip/restart-node-one failed with artifacts on master @ c7e039fa912c296b3ecf2bfb63679d7e6cbff8ca:

The test failed on branch=master, cloud=local:
test artifacts and logs in: artifacts/acceptance/gossip/restart-node-one/run_1
	gossip.go:435,acceptance.go:104,test_runner.go:770: node 1 still has 1 replicas
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne.func3
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:426
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:415
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAcceptance.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/acceptance.go:104
		  | main.(*testRunner).runTest.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:770
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) node 1 still has 1 replicas
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Reproduce

See the corresponding section in the [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest)

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-07-28T02:49:08Z

roachtest.acceptance/gossip/restart-node-one failed with artifacts on master @ 131db990b54f5fd526e1194ce1a4d6138bd04698:

The test failed on branch=master, cloud=local:
test artifacts and logs in: artifacts/acceptance/gossip/restart-node-one/run_1
	gossip.go:435,acceptance.go:104,test_runner.go:770: node 1 still has 1 replicas
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne.func3
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:426
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:415
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAcceptance.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/acceptance.go:104
		  | main.(*testRunner).runTest.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:770
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) node 1 still has 1 replicas
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Reproduce

See the corresponding section in the [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest)

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-07-28T03:51:03Z

roachtest.acceptance/gossip/restart-node-one failed with artifacts on master @ 131db990b54f5fd526e1194ce1a4d6138bd04698:

The test failed on branch=master, cloud=local:
test artifacts and logs in: artifacts/acceptance/gossip/restart-node-one/run_1
	gossip.go:435,acceptance.go:104,test_runner.go:770: node 1 still has 1 replicas
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne.func3
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:426
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:415
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAcceptance.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/acceptance.go:104
		  | main.(*testRunner).runTest.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:770
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) node 1 still has 1 replicas
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Reproduce

See the corresponding section in the [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest)

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

knz · 2021-07-28T12:10:39Z

skipping in #68158

cockroach-teamcity · 2021-07-28T14:16:49Z

roachtest.acceptance/gossip/restart-node-one failed with artifacts on master @ 15b5d06e0702dc0ca5f4aad98d09c6da06708b7b:

The test failed on branch=master, cloud=local:
test artifacts and logs in: artifacts/acceptance/gossip/restart-node-one/run_1
	gossip.go:435,acceptance.go:104,test_runner.go:770: node 1 still has 1 replicas
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne.func3
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:426
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:415
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAcceptance.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/acceptance.go:104
		  | main.(*testRunner).runTest.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:770
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) node 1 still has 1 replicas
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Reproduce

See the corresponding section in the [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest)

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-07-28T15:36:41Z

roachtest.acceptance/gossip/restart-node-one failed with artifacts on master @ 412117e27d3e11f1ea56982a6a95180ce0a5b7f5:

The test failed on branch=master, cloud=local:
test artifacts and logs in: artifacts/acceptance/gossip/restart-node-one/run_1
	gossip.go:435,acceptance.go:104,test_runner.go:770: node 1 still has 2 replicas
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne.func3
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:426
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:415
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAcceptance.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/acceptance.go:104
		  | main.(*testRunner).runTest.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:770
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) node 1 still has 2 replicas
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Reproduce

See the corresponding section in the [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest)

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-07-28T16:39:13Z

roachtest.acceptance/gossip/restart-node-one failed with artifacts on master @ 115928196de1beaa63e3e696c14dcd9392690ca2:

The test failed on branch=master, cloud=local:
test artifacts and logs in: artifacts/acceptance/gossip/restart-node-one/run_1
	gossip.go:435,acceptance.go:104,test_runner.go:770: node 1 still has 1 replicas
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne.func3
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:426
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:415
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAcceptance.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/acceptance.go:104
		  | main.(*testRunner).runTest.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:770
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) node 1 still has 1 replicas
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Reproduce

See the corresponding section in the [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest)

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-07-28T20:11:13Z

roachtest.acceptance/gossip/restart-node-one failed with artifacts on master @ c4702ef7d1959c8d1a231cc73ef5b2ccaefdfea4:

The test failed on branch=master, cloud=local:
test artifacts and logs in: artifacts/acceptance/gossip/restart-node-one/run_1
	gossip.go:435,acceptance.go:104,test_runner.go:770: node 1 still has 1 replicas
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne.func3
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:426
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:415
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAcceptance.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/acceptance.go:104
		  | main.(*testRunner).runTest.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:770
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) node 1 still has 1 replicas
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Reproduce

See the corresponding section in the [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest)

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

tbg · 2021-07-28T21:20:50Z

I would guess that this shares a root cause with #68171 (comment) (and the other tests linked to that issue).

It might be worth starting to pull on #68171 since that's a nice short test that can be run via --local and we should be able to bisect down the commit "easily".

cockroach-teamcity · 2021-07-29T05:30:18Z

roachtest.acceptance/gossip/restart-node-one failed with artifacts on master @ f19914b8c6281e463645580e1411774c3b0c20c9:

The test failed on branch=master, cloud=local:
test artifacts and logs in: artifacts/acceptance/gossip/restart-node-one/run_1
	gossip.go:435,acceptance.go:104,test_runner.go:770: node 1 still has 1 replicas
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne.func3
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:426
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:415
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAcceptance.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/acceptance.go:104
		  | main.(*testRunner).runTest.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:770
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) node 1 still has 1 replicas
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Reproduce

See the corresponding section in the [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest)

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-07-29T06:33:15Z

roachtest.acceptance/gossip/restart-node-one failed with artifacts on master @ f19914b8c6281e463645580e1411774c3b0c20c9:

The test failed on branch=master, cloud=local:
test artifacts and logs in: artifacts/acceptance/gossip/restart-node-one/run_1
	gossip.go:435,acceptance.go:104,test_runner.go:770: node 1 still has 1 replicas
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne.func3
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:426
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:415
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAcceptance.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/acceptance.go:104
		  | main.(*testRunner).runTest.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:770
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) node 1 still has 1 replicas
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Reproduce

See the corresponding section in the [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest)

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

Touches cockroachdb#68107. Release note: None

68156: roachtest: fix replicagc-changed-peers (again) r=erikgrinaker a=tbg I wrote buggy code. Since it isn't hit in every invocation, it slipped through my local testing. Fixes #68155 Fixes #68162 Release note: None 68157: CODEOWNERS: own catalog package to obs-inf-prs r=erikgrinaker a=tbg Noticed in #67866. Release note: None 68203: roachtest: avoid releasing alloc multiple times r=erikgrinaker a=tbg As of #68103, existing problems in the cluster creation retry logic are tickled reliably: on retry we are releasing an alloc twice, resulting in a panic. Unfortunately, the alloc is acquired several layers above the retry, so for now the best I can do is to avoid releasing the alloc if we are bound for a retry. The reason the above PR tickles it more reliably is because it accidentally removed a sensible failfast: when the cluster already exists, we shouldn't destroy & try to recreate it again (which would then explode on the double-release anyway). Release note: None 68224: roachtest: fix inconsistency r=erikgrinaker a=tbg The way that test was passing the args no longer works. Fixes #64806. Release note: None 68226: roachtest: allow local runs with roachstress r=erikgrinaker a=tbg It's appealing to use roachstress also as a tool to simply run a roachtest, since it avoids having to think much about the flags and incantation. This commit adds a prompt to roachstress about running in local mode. If selected, the binaries are built to target the local architecture, and the builder is not invoked. Release note: None 68227: roachtest: skip acceptance/gossip/restart-node-one r=erikgrinaker a=tbg Touches #68107. Release note: None Co-authored-by: Tobias Grieger <tobias.schottdorf@gmail.com> Co-authored-by: Tobias Grieger <tobias.b.grieger@gmail.com>

cockroach-teamcity · 2021-07-29T10:10:24Z

roachtest.acceptance/gossip/restart-node-one failed with artifacts on master @ baec9cc9d80faba95fdab715a5834d294df1062f:

The test failed on branch=master, cloud=local:
test artifacts and logs in: artifacts/acceptance/gossip/restart-node-one/run_1
	gossip.go:435,acceptance.go:104,test_runner.go:770: node 1 still has 1 replicas
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne.func3
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:426
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipRestartNodeOne
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:415
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAcceptance.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/acceptance.go:104
		  | main.(*testRunner).runTest.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:770
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) node 1 still has 1 replicas
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Reproduce

See the corresponding section in the [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest)

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

tbg · 2021-08-23T08:44:07Z

#68169 (comment)

tbg · 2021-08-24T10:14:29Z

The test is now unskipped (#69232), after I verified that it passes (10x).

cockroach-teamcity added this to roachtest/unit test backlog in KV Jul 27, 2021

tbg assigned AlexTalks Jul 27, 2021

tbg moved this from roachtest/unit test backlog to Current Milestone in KV Jul 27, 2021

knz mentioned this issue Jul 28, 2021

Revert "go.mod: bump cloud.google.com/go/storage to v1.16.0" #68158

Merged

tbg added the skipped-test label Jul 28, 2021

tbg changed the title ~~roachtest: acceptance/gossip/restart-node-one failed~~ roachtest: acceptance/gossip/restart-node-one failed [slow replication] Jul 29, 2021

tbg mentioned this issue Jul 29, 2021

roachtest: quit-all-nodes failed #65637

Closed

tbg added a commit to tbg/cockroach that referenced this issue Jul 29, 2021

roachtest: skip acceptance/gossip/restart-node-one

f6cc88c

Touches cockroachdb#68107. Release note: None

tbg mentioned this issue Jul 29, 2021

roachtest: skip acceptance/gossip/restart-node-one #68227

Merged

tbg removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Aug 23, 2021

tbg closed this as completed Aug 24, 2021

KV automation moved this from Current Milestone to Closed Aug 24, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: acceptance/gossip/restart-node-one failed [slow replication] #68107

roachtest: acceptance/gossip/restart-node-one failed [slow replication] #68107

cockroach-teamcity commented Jul 27, 2021

tbg commented Jul 27, 2021

cockroach-teamcity commented Jul 27, 2021

cockroach-teamcity commented Jul 27, 2021

cockroach-teamcity commented Jul 27, 2021

cockroach-teamcity commented Jul 28, 2021

cockroach-teamcity commented Jul 28, 2021

knz commented Jul 28, 2021

cockroach-teamcity commented Jul 28, 2021

cockroach-teamcity commented Jul 28, 2021

cockroach-teamcity commented Jul 28, 2021

cockroach-teamcity commented Jul 28, 2021

tbg commented Jul 28, 2021

cockroach-teamcity commented Jul 29, 2021

cockroach-teamcity commented Jul 29, 2021

cockroach-teamcity commented Jul 29, 2021

tbg commented Aug 23, 2021

tbg commented Aug 24, 2021

roachtest: acceptance/gossip/restart-node-one failed [slow replication] #68107

roachtest: acceptance/gossip/restart-node-one failed [slow replication] #68107

Comments

cockroach-teamcity commented Jul 27, 2021

tbg commented Jul 27, 2021

cockroach-teamcity commented Jul 27, 2021

cockroach-teamcity commented Jul 27, 2021

cockroach-teamcity commented Jul 27, 2021

cockroach-teamcity commented Jul 28, 2021

cockroach-teamcity commented Jul 28, 2021

knz commented Jul 28, 2021

cockroach-teamcity commented Jul 28, 2021

cockroach-teamcity commented Jul 28, 2021

cockroach-teamcity commented Jul 28, 2021

cockroach-teamcity commented Jul 28, 2021

tbg commented Jul 28, 2021

cockroach-teamcity commented Jul 29, 2021

cockroach-teamcity commented Jul 29, 2021

cockroach-teamcity commented Jul 29, 2021

tbg commented Aug 23, 2021

tbg commented Aug 24, 2021