[docs] Troubleshooting for how to replace a master

[bmatican]

This is the master equivalent of #4675 and uses parts of https://docs.yugabyte.com/latest/manage/change-cluster-config/#root.

Assumptions

• An initial set of masters M1, M2, M3
• We are removing M1
• We want to add M4
• We'll be using the default master RPC port of 7100

Steps

• Start M4 with --master_addresses= (empty string), this will put the master in what we refer to as ShellMode
• Add the new master into the quorum using yb-admin -master_addresses M1:7100,M2:7100,M3:7100 change_master_config ADD_SERVER M4 7100
• Remove the old master from the quorum using yb-admin -master_addresses M1:7100,M2:7100,M3:7100,M4:7100 change_master_config REMOVE_SERVER M1 7100 (note: the master addresses needs to include all 4 masters, in case the new one is suddenly the master leader!)
• Validate that your set of masters is now M2, M3 and M4 using yb-admin -master_addresses M2:7100,M3:7100,M4:7100 list_all_masters
• Until [docdb] Persist latest master config on TS #1542 is implemented, the TS will by default only know of whatever masters are encoded in the --tserver_master_addrs flag that they are started with. If any one of those masters is still part of the active quorum, then they will propagate the new master quorum over via heartbeats. If, however, none of the current masters are present in the TS flag, then the TS will not be able to join the cluster! So it is important to make sure to update tserver_master_addrs on every TS to the new set of master addresses, M2:7100,M3:7100,M4:7100!

Notes

• If the master you wish to replace is already dead (eg: VM was terminated), you might want to first do the REMOVE step, then do the ADD step