Bug Report: [vtgate] RemoveTablet during TabletExternallyReparented event can lead to healthcheck corruption

[arthurschreiber]

Overview of the Issue

During external reparent events, a shard can be in a state where to vttablet processes are running as PRIMARY tablets. vtgates consult a so called PrimaryTermStartTimestamp to break ties in this situation, and will exclusively prefer the PRIMARY tablet with the highest PrimaryTermStartTimestamp to sent queries to.

The duration for how long multiple vttablet processes can be seen in PRIMARY role is influenced by the --shutdown_grace_period flag, which until version 20.0 was 0, meaning potentially "indefinite" (but in practice is usually limited by the transaction timeout).

If, during the time that a vtgate is seeing two PRIMARY tablets running, a tablet deletion is recognized (this can be any other tablet in the shard), the vtgate healtcheck would go into an invalid state where both the demoted and promoted primary tablets are seen as valid targets for @primary queries. This could lead to queries being silently sent to the wrong tablet (and silently being retried, so most of the time this error state was completely invisible).

But we've also seen cases where vtgate processes ended up trying to send queries to the demoted primary exclusively, causing all DML queries processed by these vtgates to fail.

Once in this invalid state, I don't think the vtgate can leave it until the affected tablet is deleted from the topology. Restarting the vttablet process on the demoted primary would cause it to start back up as a REPLICA, but it would still be seen as a valid candidate for @primary queries. Restarting / replacing an affected vtgate process would be another option to get the affected vtgate processes into a healthy state again.

Reproduction Steps

N/A

Binary Version

v17 and later

Operating System and Environment details

N/A

Log Fragments

N/A