server,kv: ensure decommission works even when there's just a status record

[knz]

Some clusters have gotten into a state where there is a status record for a node but no liveness record.

When this happens, the node shows up in various places, but the node decommission command does not work (the deco process wants to know the liveness status before it touches the record).

We want to introduce a special case in decommission to "clean up" these stray node records.

[knz]

Erik proposes a more general KV edit tool in #70863, however I think we should still do this.

[tbg]

This shouldn't be possible any more on current CRDB versions (21.1+) as the liveness record is written before a new node even learns its assigned NodeID (and before it can possibly write a status record). So I don't think it's very useful to let decommissioning handle this case, as this would cause the reverse problem: it could create liveness records with user-defined NodeIDs, and users shouldn't be able to do this. I would rather go with #70863.

[knz]

The suggestion is to have deco remove the stray status record, not synthetize a new liveness record.

[tbg]

Thanks, that is a good idea.

[erikgrinaker]

There has been some pushback against this, as it would require relaxing the checks around decommissioning, so I'm going to close the issue. The current code should already make sure that status entries are handled properly on node creation and decommissioning, this state is a result of a bug in a previous version. I'm going to ship a custom binary to the affected users to clean up their status entry.

The reason this even comes up at all is because we have APIs that return node data based solely on the status entry, and these results are then shown in e.g. the DB Console. If these APIs joined/filtered the status data against the KV liveness data, such that only nodes with a liveness entry would be returned, users would not even know about this state at all. I opened #71033 to address this.