storage: node gossip loop panic when own node descriptor deleted

[tbg]

https://sentry.io/cockroach-labs/cockroachdb/issues/798675882/?referrer=webhooks_plugin

stopper.go:182: gossip.go:969

github.com/cockroachdb/cockroach/pkg/server.(*Node).startGossip.func1

stacktrace: {u'frames': [{u'function': u'func1', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go', u'module': u'github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker', u'in_app': True, u'lineno': 199, u'filename': u'github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go'}, {u'function': u'func1', u'abs_path': u'/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go', u'module': u'github.com/cockroachdb/cockroach/pkg/server.(*Node).startGossip', u'in_app': True, u'lineno': 740, u'filename': u'github.com/cockroachdb/cockroach/pkg/server/node.go'}, {u'function': u'gopanic', u'abs_path': u'/usr/local/go/src/runtime/panic.go', u'module': u'runtime', u'in_app': False, u'lineno': 502, u'filename': u'runtime/panic.go'}, {u'function': u'call32', u'abs_path': u'/usr/local/go/src/runtime/asm_amd64.s', u'module': u'runtime', u'in_app': False, u'lineno': 573, u'filename': u'runtime/asm_amd64.s'}]}
type: *log.safeError
value: stopper.go:182: gossip.go:969

[tbg]

cockroach/pkg/server/node.go

Lines 738 to 741 in a7b24d6

	// Verify we've already gossiped our node descriptor.
	if _, err := n.storeCfg.Gossip.GetNodeDescriptor(n.Descriptor.NodeID); err != nil {
	panic(err)
	}

The error is

cockroach/pkg/gossip/gossip.go

Lines 966 to 970 in 7c3289e

	// Don't return node descriptors that are empty, because that's meant to
	// indicate that the node has been removed from the cluster.
	if nodeDescriptor.NodeID == 0 || nodeDescriptor.Address.IsEmpty() {
	return nil, errors.Errorf("n%d has been removed from the cluster", nodeID)
	}

[tbg]

@petermattis is this fixed by #34155?

[petermattis]

#34155 will certainly avoid this problem as our own node descriptor can no longer be removed by another node. This symptom of the underlying bug is relatively innocuous. The process will crash and hopefully something will restart it, but it isn't as bad as the other symptom where the process would remain running, but receive no traffic.

I think this can reasonably be closed as only a single event occurred months ago.