Stateful failover

[rosik]

This patch introduces new failover mode - stateful.

There are 3 main concepts:

Internal coordinator is a new role, which makes decisions regarding leadership. Earlier it was a part of every instances' failover module, but now it's split away. There may be only one active coordinator in cluster at a time. Its uniqueness is ensured by external storage which manages the lock and saves appointments.

External storage (kingdom.lua) is a stand-alone Tarantool instance which provides locking mechanism and keeps decisions made by the coordinator.

Failover module (the old one) operates on every instance in cluster and gathers leadership information for others modules. It was refactored too, and now it can be described with 4 functions with clearly separated responsibilities:

• _get_appointments_* generates leadership map by itself or polls it from external storage depending on the mode setting (disabled/eventual/stateful).
• accept_appointments() just refreshes the cache and tracks if anything changed.
• failover_loop (a fiber) repeatedly gets new appointments and accepts them using corresponding functions from the above.
• cfg is called from confapplier.apply_config() (on restart or on committing new clusterwide configuration). At first it gets appointments synchronously and then starts the failover loop.

I didn't forget about

• Tests
• Changelog
• Documentation

Close #148

[Kasen]

I suggest allowing the node to check its status and push it into membership. It may be made with some kind of callback. It may be necessary to check node replication status if this node is storage.

[HustonMmmavr]

Maybe use pool.format_uri, something like that:

local _, err = pool.format_uri(uri)
e_config:assert(
    not err,
    '%s.failover.storage_uri: %s',
    field, err.err
)