Stateful failover (#604)

This patch introduces new failover mode - stateful.

There are 3 main concepts:

**Internal coordinator** is a new role, which makes decisions regarding
leadership. Earlier it was a part of every instances' failover module,
but now it's split away. There may be only one active coordinator in
cluster at a time. Its uniqueness is ensured by external storage which
manages the lock and saves appointments.

**External storage** (`kingdom.lua`) is a stand-alone Tarantool instance
which provides locking mechanism and keeps decisions made by the
coordinator.

**Failover module** (the old one) operates on every instance in cluster
and gathers leadership information for others modules. It was refactored
too, and now it can be described with 4 functions with clearly separated
responsibilities:

- `_get_appointments_*` generates leadership map by itself or polls it
  from external storage depending on the mode setting
  (disabled/eventual/stateful).
- `accept_appointments()` just refreshes the cache and tracks if
  anything changed.
- `failover_loop` (a fiber) repeatedly gets new appointments and accepts
  them using corresponding functions from the above.
- `cfg` is called from `confapplier.apply_config()` (on restart or on
  committing new clusterwide configuration). At first it gets
  appointments synchronously and then starts the failover loop.

.luacheckrc

@@ -11,3 +11,10 @@ exclude_files = {
     'cartridge/graphql.lua',
     'cartridge/graphql/*.lua',
 }
+new_read_globals = {
+    box = { fields = {
+        session = { fields = {
+            storage = {read_only = false, other_fields = true}
+        }}
+    }}
+}

CHANGELOG.md

@@ -14,6 +14,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
   can be either absolute or relative - in the later case it's calculated
   relative to `cartridge.workdir`.
 
+- Implement stateful failover mode.
+
 ### Deprecated
 
 Lua API:

CMakeLists.txt

@@ -165,6 +165,11 @@ install(
   DESTINATION ${TARANTOOL_INSTALL_LUADIR}
 )
 
+install(
+  FILES ${CMAKE_CURRENT_SOURCE_DIR}/kingdom.lua
+  DESTINATION ${TARANTOOL_INSTALL_BINDIR}
+)
+
 install(
   FILES
     ${CMAKE_CURRENT_BINARY_DIR}/VERSION.lua

cartridge-scm-1.rockspec

@@ -29,6 +29,7 @@ build = {
         TARANTOOL_DIR = '$(TARANTOOL_DIR)',
         TARANTOOL_INSTALL_LIBDIR = '$(LIBDIR)',
         TARANTOOL_INSTALL_LUADIR = '$(LUADIR)',
+        TARANTOOL_INSTALL_BINDIR = '$(BINDIR)',
     },
     copy_directories = {'doc'},
 }

cartridge.lua

@@ -509,6 +509,10 @@ local function cfg(opts, box_opts)
         service_registry.set('httpd', httpd)
     end
 
+    local ok, err = roles.register_role('cartridge.roles.coordinator')
+    if not ok then
+        return nil, err
+    end
     for _, role in ipairs(opts.roles or {}) do
         local ok, err = roles.register_role(role)
         if not ok then