config: allow to configure an anonymous replica #9418

Totktonada · 2023-11-28T01:06:59Z

replication.anon: true option is now working.

Mostly, it is the only thing that a user needs to know about this changeset.

However, technically speaking, several related configuration checks are introduced here and dependent defaults were adjusted to make usage of anonymous replicas smoother.

The new configuration constrainsts are the following.

A replicaset must contain at least one non-anonymous instance.
An anonymous replica can't be configured as writable instance using database.mode or <replicaset>.leader options.
An anonymous replica can't be configured with replication.election_mode equals to candidate, voter or manual (only off is allowed).

A few more nuances about anonymous replicas:

Anonymous replicas are filtered out from default upstream list.
A replication.failover: election replicaset can contain anonymous replicas, but replication.election_mode defaults to off for them (unlike non-anonymous instances, where the default is candidate).
replication.failover: supervised skips anonymous replicas, when choosing a bootstrap leader.
A anonymous replica can joined a replicaset, which has all the instances in read-only mode (unlike a non-anonymous instance).

Please, check out commits of the patchset for more detailed description of the constrains and nuances.

Fixes #9432

coveralls · 2023-11-28T01:15:51Z

coverage: 86.821% (+0.005%) from 86.816%
when pulling e068c48 on Totktonada:gh-xxxx-config-fix-anonymous-replica
into a6d5eb7
on tarantool:master.

Serpentian

Looks great! I have no objections with the following patch, just some nits. Let's beautify the patch with commit messages and everything

src/box/lua/config/configdata.lua

test/config-luatest/replicaset_test.lua

@TarantoolBot

This commit allows to bootstrap an anonymous replica from a replicaset, where all the instances are in read-only mode. The reason of the change is that there are no technical reasons to forbid this action. An anonymous replica is not registered in `_cluster` system space, so it can join a replicaset even if there are no writable instances. Fixes tarantool#9432 @TarantoolBot document Title: config: anonymous replica is now supported `replication.anon: true` option is now working. There are configuration constraints that are related to anonymous replicas. * A replicaset must contain at least one non-anonymous instance. * An anonymous replica can't be configured as writable instance using `database.mode` or `<replicaset>.leader` options. * An anonymous replica can't be configured with `replication.election_mode` equals to `candidate`, `voter` or `manual` (only `off` is allowed). A few more nuances about anonymous replicas: * Anonymous replicas are filtered out from default upstream list. * A `replication.failover: election` replicaset can contain anonymous replicas, but `replication.election_mode` defaults to `off` for them (unlike non-anonymous instances, where the default is `candidate`). * `replication.failover: supervised` skips anonymous replicas, when choosing a bootstrap leader. * A anonymous replica can joined a replicaset, which has all the instances in read-only mode (ulike a non-anonymous instance). See details in [1] and [2]. [1]: tarantool#9432 [2]: tarantool#9418

@TarantoolBot

This commit allows to bootstrap an anonymous replica from a replicaset, where all the instances are in read-only mode. The reason of the change is that there are no technical reasons to forbid this action. An anonymous replica is not registered in `_cluster` system space, so it can join a replicaset even if there are no writable instances. Fixes tarantool#9432 @TarantoolBot document Title: config: anonymous replica is now supported `replication.anon: true` option is now working. There are configuration constraints that are related to anonymous replicas. * A replicaset must contain at least one non-anonymous instance. * An anonymous replica can't be configured as writable instance using `database.mode` or `<replicaset>.leader` options. * An anonymous replica can't be configured with `replication.election_mode` equals to `candidate`, `voter` or `manual` (only `off` is allowed). A few more nuances about anonymous replicas: * Anonymous replicas are filtered out from default upstream list. * A `replication.failover: election` replicaset can contain anonymous replicas, but `replication.election_mode` defaults to `off` for them (unlike non-anonymous instances, where the default is `candidate`). * `replication.failover: supervised` skips anonymous replicas, when choosing a bootstrap leader. * A anonymous replica can joined a replicaset, which has all the instances in read-only mode (unlike a non-anonymous instance). See details in [1] and [2]. [1]: tarantool#9432 [2]: tarantool#9418

changelogs/unreleased/config-anonymous-replica.md

@TarantoolBot

This commit allows to bootstrap an anonymous replica from a replicaset, where all the instances are in read-only mode. The reason of the change is that there are no technical reasons to forbid this action. An anonymous replica is not registered in `_cluster` system space, so it can join a replicaset even if there are no writable instances. Fixes tarantool#9432 @TarantoolBot document Title: config: anonymous replica is now supported `replication.anon: true` option is now working. There are configuration constraints that are related to anonymous replicas. * A replicaset must contain at least one non-anonymous instance. * An anonymous replica can't be configured as writable instance using `database.mode` or `<replicaset>.leader` options. * An anonymous replica can't be configured with `replication.election_mode` equals to `candidate`, `voter` or `manual` (only `off` is allowed). A few more nuances about anonymous replicas: * Anonymous replicas are filtered out from default upstream list. * A `replication.failover: election` replicaset can contain anonymous replicas, but `replication.election_mode` defaults to `off` for them (unlike non-anonymous instances, where the default is `candidate`). * `replication.failover: supervised` skips anonymous replicas, when choosing a bootstrap leader. * A anonymous replica can joined a replicaset, which has all the instances in read-only mode (unlike a non-anonymous instance). See details in [1] and [2]. [1]: tarantool#9432 [2]: tarantool#9418

ImeevMA

Thanks for the PR! I don't see any problem with PR, except that it seems a little strange to add a list to the changelog and then remove it. But I do not think that this is a bit deal.

changelogs/unreleased/config-anonymous-replica.md

@TarantoolBot

This commit allows to bootstrap an anonymous replica from a replicaset, where all the instances are in read-only mode. The reason of the change is that there are no technical reasons to forbid this action. An anonymous replica is not registered in `_cluster` system space, so it can join a replicaset even if there are no writable instances. Fixes tarantool#9432 @TarantoolBot document Title: config: anonymous replica is now supported `replication.anon: true` option is now working. There are configuration constraints that are related to anonymous replicas. * A replicaset must contain at least one non-anonymous instance. * An anonymous replica can't be configured as writable instance using `database.mode` or `<replicaset>.leader` options. * An anonymous replica can't be configured with `replication.election_mode` equals to `candidate`, `voter` or `manual` (only `off` is allowed). A few more nuances about anonymous replicas: * Anonymous replicas are filtered out from default upstream list. * A `replication.failover: election` replicaset can contain anonymous replicas, but `replication.election_mode` defaults to `off` for them (unlike non-anonymous instances, where the default is `candidate`). * `replication.failover: supervised` skips anonymous replicas, when choosing a bootstrap leader. * A anonymous replica can joined a replicaset, which has all the instances in read-only mode (unlike a non-anonymous instance). See details in [1] and [2]. [1]: tarantool#9432 [2]: tarantool#9418

Serpentian

Thank you for the patch! Looks great. Feel free to skip nits

src/box/lua/config/applier/box_cfg.lua

src/box/lua/config/configdata.lua

The commit effectively enables support of anonymous replicas in the declarative configuration. It has several caveats (see the changelog entry), which will be resolved in the following commits of the patchset. An attempt to persist an instance name of an anonymous replica can't be successful, because it has no entry in `_cluster` system space. Such an attempt leads to ER_INSTANCE_NAME_MISMATCH error. This commit patches the configuration applying logic to skip attempt to set `box.cfg({instance_name = <...>})` if the instance is configured as an anonymous replica using `replication.anon: true` option. Part of tarantool#9432 NO_DOC=replication.anon option is already documented in the scope of tarantool/doc#3851. The bugfix shouldn't affect the documentation pages much, however related constraints are summarized in a documentation request in the last commit of the series.

This commit effectively allows to set `replication.anon: true` without specifying `replication.peers`. Without filtering out anonymous replicas from the list of upstreams we get an error regarding attempt to use an anonymous replica as an upstream for a non-anonymous instance. Also, anonymous replicas are excluded from autogenerated upstream list for other anonymous replicas. It makes the list the same on all the peers. A user can configure a custom data flow using `replication.peers` option. Part of tarantool#9432 NO_DOC=The documentation request is in the last commit of the series.

Filter out anonymous replicas when choosing a bootstrap leader in `replication.failover: supervised` mode. An anonymous replica can't be in read-write mode, so it can't be a replicaset bootstrap leader. Part of tarantool#9432 NO_DOC=It is bugfix. However, this detail is mentioned in the documentation request is in the last commit of the series just in case.

A replicaset that contains only anonymous replicas can't be bootstrapped, because all the instances must be in read-only mode. Part of tarantool#9432 NO_DOC=The documentation request is in the last commit of the series.

This commit adds several checks that are specific for `replication.failover` mode. * `replication.failover: off`: an anonymous replica shouldn't be set to read-write mode using `database.mode` option. * `replication.failover: manual`: an anonymous replica shouldn't be configured as a replicaset leader using `<replicaset>.leader` option. * `replication.failover: election`: an anonymous replica can't be configured with `replication.election_mode` other than `off`. This commit also adjusts default `replication.election_mode` to `off` for an anonymous replica if it is part of a `replication.failover: election` replicaset (the default for a non-anonymous instance is `candidate`). Part of tarantool#9432 NO_DOC=The documentation request is in the last commit of the series.

@TarantoolBot

This commit allows to bootstrap an anonymous replica from a replicaset, where all the instances are in read-only mode. The reason of the change is that there are no technical reasons to forbid this action. An anonymous replica is not registered in `_cluster` system space, so it can join a replicaset even if there are no writable instances. Fixes tarantool#9432 @TarantoolBot document Title: config: anonymous replica is now supported `replication.anon: true` option is now working. There are configuration constraints that are related to anonymous replicas. * A replicaset must contain at least one non-anonymous instance. * An anonymous replica can't be configured as writable instance using `database.mode` or `<replicaset>.leader` options. * An anonymous replica can't be configured with `replication.election_mode` equals to `candidate`, `voter` or `manual` (only `off` is allowed). A few more nuances about anonymous replicas: * Anonymous replicas are filtered out from default upstream list. * A `replication.failover: election` replicaset can contain anonymous replicas, but `replication.election_mode` defaults to `off` for them (unlike non-anonymous instances, where the default is `candidate`). * `replication.failover: supervised` skips anonymous replicas, when choosing a bootstrap leader. * A anonymous replica can joined a replicaset, which has all the instances in read-only mode (unlike a non-anonymous instance). See details in [1] and [2]. [1]: tarantool#9432 [2]: tarantool#9418

Serpentian reviewed Nov 28, 2023

View reviewed changes

src/box/lua/config/configdata.lua Outdated Show resolved Hide resolved

src/box/lua/config/configdata.lua Show resolved Hide resolved

src/box/lua/config/configdata.lua Outdated Show resolved Hide resolved

test/config-luatest/replicaset_test.lua Outdated Show resolved Hide resolved

Totktonada mentioned this pull request Nov 29, 2023

Improve XlogGapError log message verbosity #9430

Open

Totktonada force-pushed the gh-xxxx-config-fix-anonymous-replica branch from a45bc5c to 63030ce Compare December 5, 2023 00:56

Totktonada changed the title ~~WIP: config: allow to configure an anonymous replica~~ config: allow to configure an anonymous replica Dec 5, 2023

Totktonada force-pushed the gh-xxxx-config-fix-anonymous-replica branch from 63030ce to 4711e53 Compare December 5, 2023 01:06

Totktonada marked this pull request as ready for review December 5, 2023 01:07

Totktonada requested a review from a team as a code owner December 5, 2023 01:07

Totktonada requested review from ImeevMA and a team December 5, 2023 01:07

Totktonada assigned Serpentian and ImeevMA Dec 5, 2023

p7nov approved these changes Dec 5, 2023

View reviewed changes

changelogs/unreleased/config-anonymous-replica.md Outdated Show resolved Hide resolved

Totktonada force-pushed the gh-xxxx-config-fix-anonymous-replica branch from 4711e53 to 603162d Compare December 5, 2023 10:29

ImeevMA approved these changes Dec 6, 2023

View reviewed changes

changelogs/unreleased/config-anonymous-replica.md Outdated Show resolved Hide resolved

Totktonada force-pushed the gh-xxxx-config-fix-anonymous-replica branch from 603162d to 89c3b82 Compare December 6, 2023 12:49

Totktonada unassigned ImeevMA Dec 6, 2023

Totktonada requested a review from Serpentian December 6, 2023 12:49

Serpentian approved these changes Dec 6, 2023

View reviewed changes

src/box/lua/config/applier/box_cfg.lua Show resolved Hide resolved

src/box/lua/config/configdata.lua Outdated Show resolved Hide resolved

Totktonada added 6 commits December 6, 2023 19:22

config: forbid a replicaset of only anon replicas

151edc0

A replicaset that contains only anonymous replicas can't be bootstrapped, because all the instances must be in read-only mode. Part of tarantool#9432 NO_DOC=The documentation request is in the last commit of the series.

Totktonada force-pushed the gh-xxxx-config-fix-anonymous-replica branch from 89c3b82 to e068c48 Compare December 6, 2023 16:23

Totktonada unassigned Serpentian Dec 6, 2023

ylobankov approved these changes Dec 6, 2023

View reviewed changes

Totktonada added the full-ci Enables all tests for a pull request label Dec 6, 2023

Totktonada merged commit 5ba3ace into tarantool:master Dec 6, 2023
106 checks passed

Totktonada deleted the gh-xxxx-config-fix-anonymous-replica branch December 6, 2023 23:53

TarantoolBot mentioned this pull request Dec 6, 2023

config: anonymous replica is now supported tarantool/doc#3896

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

config: allow to configure an anonymous replica #9418

config: allow to configure an anonymous replica #9418

Totktonada commented Nov 28, 2023 •

edited

coveralls commented Nov 28, 2023 •

edited

Serpentian left a comment

ImeevMA left a comment

Serpentian left a comment

config: allow to configure an anonymous replica #9418

config: allow to configure an anonymous replica #9418

Conversation

Totktonada commented Nov 28, 2023 • edited

coveralls commented Nov 28, 2023 • edited

Serpentian left a comment

Choose a reason for hiding this comment

ImeevMA left a comment

Choose a reason for hiding this comment

Serpentian left a comment

Choose a reason for hiding this comment

Totktonada commented Nov 28, 2023 •

edited

coveralls commented Nov 28, 2023 •

edited