How to allow multiple instances of replicated service to be leader?

[sixty-two62]

Hey, thanks for open-sourcing TernFS! I've been taking a look at the code and doing some testing, and I'm confused on how to start up replicated services like ternregistry and ternshard such that more than one instance is allowed to be the leader. It looks like ternrun only passes -logsdb-leader to instance 0 for all of the replicated services. If I try to override this and start all instances with -logsdb-leader, I start getting assertion errors.

I took a look at the source and found this chunk of code to be odd: https://github.com/XTXMarkets/ternfs/blob/main/cpp/core/LogsDB.cpp#L732-L738. If I'm understanding things, _noReplication is true iff -logsdb-no-replication is passed, and _avoidBeingLeader is true iff -logsdb-leader is passed, so it looks like this function will always make any instance started with -logsdb-leader behave as if -logsdb-no-replication was passed. Am I missing something here?

[mcrnic]

You are right, it's not possible to enable it through flags alone.

It has been disabled in code on purpose.

When we need to failover we start another instance with -logsdb-leader

I'll work on removing code restriction, test and enabling in our production.
Should be a week or so to test it properly