Handle replication-cluster list change for peer-cluster when zk-watch…

[rdhabalia]

Motivation

Addressing: #3425

Issue

There could be possible data-loss when user is using peer-cluster feature and changes replication cluster from peer1-cluster to peer2-cluster.

• Sometimes, broker never receives a zk-watch for namespace policies change so, if replication cluster list has been changed then there is a possibility that broker which is owning bundle of that namespace, might not see the change and still serving the namespace bundle.
• if the replication cluster has been changed from peer-cluster1 to peer-cluster2 then peer-cluster1 will continue to serve namespace and will not redirect lookup-request to peer-cluster2. in that case there will be ownership discrepancy and peer-cluster2 might not receive the traffic which will cause in data-loss.

Fix

• this issue happens when one of broker doesn't receive the zk-watch.
• To fix the issue, whichever broker receives the watch, that unloads the namespace if it's already owned by local cluster and broker invalidates namespace-policy cache before owning the namespace bundle to avoid any discrepancy.

[merlimat]

👍

[rdhabalia]

retest this please

[rdhabalia]

rerun java8 tests
rerun integration tests

[rdhabalia]

rerun java8 tests

[rdhabalia]

rerun java8 tests

[rdhabalia]

rerun java8 tests

[rdhabalia]

rerun java8 tests