[docdb ] Leader-only tserver blacklisting mode for rolling upgrades

[mbautin]

Rolling upgrades should be done as follows:

• Move all leaders away from node 1 and load-balance them between other nodes.
• Upgrade node 1.
• Move leaders so that they are load-balanced between all nodes except node 2, and node 2 has no leaders.
• Upgrade node 2.
  etc.

This requires splitting the current node blacklisting logic into two parts: for leaders and for data.

The load balancer should be responsible for rebalancing the leader load between upgrade of individual nodes.

[bmatican]

Some pointers for this:

We have a generic mechanism for persisting cluster configuration:
src/yb/master/catalog_manager.cc::SetClusterConfig

This currently has a blacklisting functionality, that is used for draining data from the provided nodes. This needs to be persisted, in case of master failover, as the drain could take a while.

This blacklist is then used in the load balancer (see: src/yb/master/cluster_balance.cc, src/yb/master/cluster_balance_util.h), to determine some of the following:

• if tablets exists on a blacklisted node, they need to be moved
• if tablets are to be added, they should never go to a blacklisted node

There's also a yb-admin command to change the blacklist by adding/removing nodes from it: src/yb/tools/yb-admin_client.cc:ChangeBlacklist

[rao-vasireddy]

@rajukumaryb and I were discussing this issue -

1. If a new YW with this capability is performing a gflag update or SW upgrade on a old universe that does not have the new blacklist command, the operation should be a no-op and continue , we should not raise an error and abort
2. The blacklist needs to be only for a short duration and once the tserver restarts, it doesn't need to be black listed any longer.
3. Can the current leader affinity feature be used or leveraged for this operation?

[rao-vasireddy]

Also, we need to move the leaders off the node quickly, let's check the current rate limiting mechanism and see how we can speed it up.

[rajukumaryb]

@rajukumaryb and I were discussing this issue -

1. If a new YW with this capability is performing a gflag update or SW upgrade on a old universe that does not have the new blacklist command, the operation should be a no-op and continue , we should not raise an error and abort
2. The blacklist needs to be only for a short duration and once the tserver restarts, it doesn't need to be black listed any longer.
3. Can the current leader affinity feature be used or leveraged for this operation?

@rao-vasireddy -

1. Might need some help from yugaware - cc: @Arnav15
2. Can be achieved by "ADD" -> polling of "get_leader_blacklist_completion" -> "REMOVE"
3. Leader affinity is placement_{cloud+region+zone} based - so it is coarser than this task which is tserver id based.