[Feature] Validate decision table for reconciliation as mentioned in ETCD design doc

[abdasgupta]

Feature (What you would like to be added):
We already enhanced reconciliation for multi node in ETCD Druid here. Now we have to validate the implementation with the decision table

This story will track the validation

Motivation (Why is this needed?):

Approach/Hint to the implement solution (optional):

[aaronfern]

After validating the decision table for reconciliation, here are my findings

Sr No	Event	Status	Comments
1	Pink of health	Complete
2	Member status is out of sync with their leases	Complete
3	All members are Ready but AllMembersReady condition is stale	Complete
4	Not all members are Ready but AllMembersReady condition is stale	Complete
5	Majority members are Ready but Ready condition is stale	Complete
6	Majority members are NotReady but Ready condition is stale	Complete
7	Some members have been in Unknown status for a while	Complete
8	Some member pods are not Ready but have not had the chance to update their status	Complete	The proposal asks that a status PodNotReady be used, however a status ContainerNotReady is used instead.
9	Quorate cluster with a minority of members NotReady	Complete
10	Quorum lost with a majority of members NotReady	Under development
11	Scale up of a healthy cluster	Feature complete	- Feature is ready from etcd druid - Not yet enabled from g/g perspective
12	Scale down of a healthy cluster	Out of scope	No use cases of scale-down right now
13	Superfluous member entries in Etcd status	Not required	- Cluster are not expected to dynamically change sizes - Single member restoration already takes care of removing a member from the cluster if that is deemed necessary - The feature of superfluous member deletion is already developed in etcd-backup-restore and can be enabled via a flag if deemed necessary in the future

tl;dr:
Etcd druid reconciliation is generally working as expected. The caveats are the following

• Scale-down is deemed out of scope,
• Quorum loss is still under development
• Scale-up is feature ready, but is yet to be enabled on g/g

[abdasgupta]

Should we close the issue as we are done with validating the decision table?

[abdasgupta]

We have already validated recovery from transient quorum loss ( see here #436) . For a non quorate cluster, we will need human intervention. The human operator will decide how to recover a non quorate cluster. We will be providing a playbook for their guidance. Please follow #437 for more details. As the scope of this issue is finished , I am closing this issue.