`replication.autoexpel` configuration option

[TarantoolBot]

Dev. issue: tarantool/tarantool#10823

Product: Tarantool
Since: 3.3.0
Root document:

• https://www.tarantool.io/en/doc/latest/reference/configuration/configuration_reference/
• (possibly somewhere in) https://www.tarantool.io/en/doc/latest/platform/replication/

SME: @ Totktonada

Details

The new option is useful for management of dynamic clusters using the
YAML configuration. The option allows to automatically expel instances
that are removed from the YAML configuration.

A ready-to-run example:

credentials:
  users:
    guest:
      roles: [super]

replication:
  failover: manual
  autoexpel:
    enabled: true
    by: prefix
    prefix: '{{ replicaset_name }}'

iproto:
  listen:
    - uri: 'unix/:./var/run/{{ instance_name }}.iproto'

groups:
  g-001:
    replicasets:
      r-001:
        leader: r-001-i-001
        instances:
          r-001-i-001: {}
          r-001-i-002: {}
          r-001-i-003: {}

How to run the example:

1. Start all the three instances:

   $ tarantool --name r-001-i-001 --config config.yaml -i
   $ tarantool --name r-001-i-002 --config config.yaml -i
   $ tarantool --name r-001-i-003 --config config.yaml -i

2. Remove r-001-i-003 from config.yaml.

3. Reload the configuration on r-001-i-001 (leader) and check the
   effect:

   tarantool> box.space._cluster:fselect()
   <..r-001-i-003 is here..>
   
   tarantool> config = require('config')
   tarantool> config:reload()
   
   tarantool> box.space._cluster:fselect()
   <..r-001-i-003 is deleted..>

Only instances whose names start from the given prefix are taken
into account, all the others are ignored. Also, instances without a
persistent name set are ignored too.

If an instance is in read-write mode and has a latest database schema,
it performs expelling of the instances:

• with the given prefix, and
• not present in the YAML configuration.

The expelling process the usual one: deletion from the _cluster system
space.

The autoexpel logic works on startup and reacts on the reconfiguration
and the box.status watcher event. If a new instance is joined and
neither of these two events occur, autoexpel does not perform any
actions on it. IOW, it doesn't forbid joining of an instance that met
the autoexpel criterion.

From the configuration point of view, the new replication.autoexpel
option of type record is added to the replicaset, group and global
levels. It has no much sense to set it for individual instances, so it
is forbidden on the instance level of the cluster configuration.

The record contains the following fields:

• enabled (boolean, default false)
• by (string, default nil)
  • allows only prefix value
• prefix (string, default nil)

replication.autoexpel.enabled determines, whether the autoexpelling
logic is enabled at all. If the option is set,
replication.autoexpel.by and replication.autoexpel.prefix are
required.

replication.autoexpel.by is the autoexpel criterion: it defines how to
determine that an instance is part of the cluster configuration and is
not an external service that uses the replication channel (such as a CDC
tool). Now, only replication.autoexpel.by = prefix criterion is
supported. A user have to set it explicitly. In future we can provide
other criteria and set one of them as default.

replication.autoexpel.prefix defines a pattern for instance names that
are considered a part of the cluster (not some external services). For
example, if all the instances in the cluster configuration are prefixed
with the replicaset name, one can use prefix: '{{ replicaset_name }}'.
If all the instances follow the i-\d\d\d pattern, the option can be
set to i-.
Requested by @Totktonada in tarantool/tarantool@b2c1153.