Forced join when network split

[patriknw]

imported from https://www.assembla.com/spaces/akka/tickets/3584

The leader cannot move joining nodes to Up when there are unreachable nodes (no convergence). Some users might want to add more nodes also when there is a network split.
Probably those nodes should have a special Up status, but be taken in use by for example routers. They are then moved to real Up by the leader on convergence.

[patriknw]

Returning to backlog due to lack of interest/priority. My prototype can be revived from #17625

[patriknw]

Taking note of that we got a user request for something similar as this issue. Discussed in #18067

[hveiga]

Hello @patriknw, I see this got moved to Backlog due to lack of interest/priority. I would like to see this implemented as part of 2.4.0 release (if possible). I am happy to help wrapping up whatever is remaining and create a pull request, only thing is that I would just need a bit of guidance. Is that something you would be interested in?

[patriknw]

Thanks for wanting to help out. We need that! Have you looked at my prototype in #17625 ?

The big question is what we want? The immediate answer is of course "we want it all, let members join and leave however they want, also during network partitions". That is a rather big change to how the cluster membership is currently designed and would be major undertaking.

However, if the answer is more specific we might be able to support some scenarios, as I tried in the prototype.

WDYT?

[hveiga]

I have been looking into the code and I believe that's what I had in mind. Another Member status where a node could join the cluster even if there is no convergence. Once convergence is reached, leader will move the Member from Weakly Up to Up. I have created this transition schema:

I believe that is what we are trying to achieve. Can you double check?

I believe everything is pretty much done except for the config parameter that will be required to enable or disable this feature to don't break Gossips coming from 2.3.X that don't know about WeaklyUp. Also, I believe there are some tests failing and maybe we need to merge the base 2.4-M3 (or the latest) because I believe there was some changes from where you implemented this.

Thanks and let me know your thoughts on this :)

[patriknw]

Sounds good to me. I'll try to talk Roland and team next week to see if we can come to some conclusion.
Thanks!

The Distributed Data module is another thing that could benefit from this. It could start using weakly up nodes in gossip replication, but not count them in consistency concerns such as WriteMajority.

[hveiga]

Cool. I am already working on it. Please let me know if there is any update.

[patriknw]

Great, thanks for picking it up

[hveiga]

Hi Patrik, should we move this ticket to 2.4.X to add it in 2.4.1?