Leader controller loses all the callback handlers after leadership switch

[jiajunwang]

A problem was found that the leader controller may lose all the callback handlers after leadership switch.
To reproduce the issue, the cluster must be using leader election mode (DistributedLeaderElection). Then frequent leadership switch caused by ZK session expiring may trigger the problem.

The symptom is that, although the leader controller exists, it won't process any ZK notification. So the cluster will not be managed.

[i3wangyi]

Do I understand it right? I just want to understand the whole picture more clearly since there's a bunch of race-condition issues happened recently.

1. The "frequent" leadership switch is caused by ZK session expiration.
2. What caused the "frequent" ZK session expiration?
3. Why the new controller didn't successfully register the callback handlers?
4. Is it because the 1st new controller hasn't finished the registration and it lost leadership already then the 2nd new controller came to place?

[jiajunwang]

Do I understand it right? I just want to understand the whole picture more clearly since there's a bunch of race-condition issues happened recently.

1. The "frequent" leadership switch is caused by ZK session expiration.

Yes.

2. What caused the "frequent" ZK session expiration?

This is not confirmed. Maybe GC, maybe network issue. Or combined. The result is obvious though.

3. Why the new controller didn't successfully register the callback handlers?

Please refer to the fix. The previous design cannot handle more than one leader node change event in a graceful way.
This use case is just not considered.

4. Is it because the 1st new controller hasn't finished the registration and it lost leadership already then the 2nd new controller came to place?

The 1st controller does lose the leadership, but that does not cause the problem. The issue was in one controller always. If this one has a leftover controller change event unprocessed, it will for sure fall into this bad situation.