Allow to set instance/replicaset/cluster names on recovery at first `box.cfg` #8978

Gerold103 · 2023-08-11T21:34:16Z

The names are supposed to be set on instance bootstrap and never change. However this is problematic when it comes to setting them first time on already existing instances which were upgraded to >= 3.0.

Right now those instances have to be started without names, then the names have to be set in a non-first box.cfg{}.

Apparently it is not enough and people want to set them first time on first box.cfg{} on recovery.

There is no an easy way to do that. The problems are on read-only instances:

They can't set replicaset and cluster names. So such box.cfg{} with new names would fair or hang, no other options.
Instance name can't be set because the instance have to wait for the master to register the new name for this instance. The waiting might contradict with options like replication_sync_timeout.

Besides, if there are multiple read-write instances, it is not clear what happens if they all set replicaset and cluster names.

The solution approved by PMs is this: if force_recovery is set and at least one stored name doesn't match the name in the config, then try to fix them. box.cfg{} will hang until the instance becomes writable. The writable instance persists new replicaset and cluster names, and own instance name. The read-only instances have to wait until the writeable one sets the new names. For the instance name they will have to retry their IPROTO_REGISTER request until they get the new name. The timeout for this waiting could be something like replication_timeout x 4 or replication_connect_timeout or something. That isn't too important, up to the implementation.

This should only work if there was no old name at all (i.e. it was NULL). Otherwise the behaviour remains like now.

The text was updated successfully, but these errors were encountered:

Currently it's impossible to recover from the snapshot, which doesn't have names set in it. This makes upgrading to 3.0.0 difficult and requires starting Tarantool manually in interactive mode, invoking box.cfg without names, upgrading schema and setting names manually. We need to make Tarantool set names automatically, when it's configured via yaml file. Even with invoking box.cfg two times (without names and with them) it's impossible to set names before the time, schema is upgraded. But we consider schema auto-upgrade unacceptable and expect from users to do it, when they're ready. So, let's allow setting names in box.cfg on the first configuration and recover from snaps, which don't have names set. As we cannot apply such names before schema upgrade, setting names on schema version below 3.0.0 has no effect. Needed for tarantool#8978 NO_DOC=tarantool/doc#3661

This commit moves all code, related to working with versions and which was used in box/lua/upgrade.lua, into a separate module and exports it to Lua API as 'internal.version' This is needed, as in the following commit we start to broadcast the dd_version_id variable, which shows the version of schema, stored in _schema['version']. This module allows us to write a trigger for config module and avoid code duplication. Needed for tarantool#8978 NO_DOC=internal NO_TEST=<already tested> NO_CHANGELOG=internal

@TarantoolBot

This commit starts to broadcast dd_version_id, which will be used to write a trigger, which will be run on schema upgrade. We cannot use on_replace/on_commit triggers for that, as it's either forbidden or not safe to yield in them. Moreover, on_replace trigger forbids making DDL changes in it. Needed for tarantool#8978 NO_TEST=<in the following commit> @TarantoolBot document Title: broadcast dd_version_id in box.schema event One new field have been added to box.schema system event: dd_version_id, which is represented as bit mask. Use 'internal.mkversion' to get more convenient way of version representation: ``` tarantool> mkversion = require('internal.mkversion') --- ... tarantool> version = mkversion.from_id(dd_version_id) --- ... tarantool> version --- - patch: 0 minor: 0 id: 196608 major: 3 ... ```

This commit adds a trigger, which sets names via box.cfg, when dd_version_id, seen from box.schema event, is changed to the version >= 3.0.0 from version < 3.0.0. On master instances we set replicaset_name and instance_name. On replicas - only instance_name, they'll automatically register themselves in _cluster space by making IPROTO_REGISTER to master. This behavior allows live upgrade to Tarantool 3.0.0. Closes tarantool#8978 NO_DOC=tarantool/doc#3661

Currently it's impossible to recover from the snapshot, which doesn't have names set in it. This makes upgrading to 3.0.0 difficult and requires starting Tarantool manually in interactive mode, invoking box.cfg without names, upgrading schema and setting names manually. We need to make Tarantool set names automatically, when it's configured via yaml file. Even with invoking box.cfg two times (without names and with them) it's impossible to set names before the time, schema is upgraded. But we consider schema auto-upgrade unacceptable and expect from users to do it, when they're ready. So, let's allow setting names in box.cfg on the first configuration and recover from snaps, which don't have names set. As we cannot apply such names before schema upgrade, setting names on schema version below 3.0.0 has no effect. Needed for tarantool#8978 NO_DOC=tarantool/doc#3661

This commit moves all code, related to working with versions and which was used in box/lua/upgrade.lua, into a separate module and exports it to Lua API as 'internal.version' This is needed, as in the following commit we start to broadcast the dd_version_id variable, which shows the version of schema, stored in _schema['version']. This module allows us to write a trigger for config module and avoid code duplication. Needed for tarantool#8978 NO_DOC=internal NO_TEST=<already tested> NO_CHANGELOG=internal

@TarantoolBot

This commit starts to broadcast dd_version_id, which will be used to write a trigger, which will be run on schema upgrade. We cannot use on_replace/on_commit triggers for that, as it's either forbidden or not safe to yield in them. Moreover, on_replace trigger forbids making DDL changes in it. Needed for tarantool#8978 NO_TEST=<in the following commit> @TarantoolBot document Title: broadcast dd_version_id in box.schema event One new field have been added to box.schema system event: dd_version_id, which is represented as bit mask. Use 'internal.mkversion' to get more convenient way of version representation: ``` tarantool> mkversion = require('internal.mkversion') --- ... tarantool> version = mkversion.from_id(dd_version_id) --- ... tarantool> version --- - patch: 0 minor: 0 id: 196608 major: 3 ... ```

This commit adds a trigger, which sets names via box.cfg, when dd_version_id, seen from box.schema event, is changed to the version >= 3.0.0 from version < 3.0.0. On master instances we set replicaset_name and instance_name. On replicas - only instance_name, they'll automatically register themselves in _cluster space by making IPROTO_REGISTER to master. This behavior allows live upgrade to Tarantool 3.0.0. Closes tarantool#8978 NO_DOC=tarantool/doc#3661

Config's box.cfg applier scans snapshot_dir in order to find out, whether recovery is going to be done. It's needed in order to determine, whether the instance should be started into ro mode firstly. Let's move info about snapshot into configdata, as it accumulates configuration data and this part should be there. The commit also introduces snapshot_path, which will be used in the following commits in order to validate names. Needed for tarantool#8978 NO_DOC=refactoring NO_TEST=refactoring NO_CHANGELOG=refactoring

@TarantoolBot

This commit introduces the new method for xlog module: xlog.meta(). It opens an xlog file, reads and returns the meta block of the file, which includes its filetype, instance_uuid and vclocks. It's needed in order to introduce names cheking in the following commit. @TarantoolBot document Title: xlog.meta([file-name]) method Description: Open an xlog file, and return its meta block. Possible errors: File does not contain properly formatted snapshot or write-ahead-log information. Needed for tarantool#8978 Example: ```lua tarantool> xlog = require('xlog') --- ... tarantool> xlog.meta('00000000000000000000.snap') --- - filetype: SNAP prev_vclock: {} instance_uuid: 87b2e60f-275c-4efa-9b0e-e9562e309692 vclock: {} ... ```

Currently only instance_uuid is validated before recovery process. All names and replicaset_uuid are checked only when recovery is done, which can take a long time. It can be frustrating to users, which have been waiting for several hours only to get name mismatch error. Let's read the small part of snapshot file before calling box.cfg in order to figure out, whether the names and uuids, passed to configuration match the ones, saved inside the snapshot. Let's also save all names, which are missing from snapshot in order to set them automatically in the future. Needed for tarantool#8978 NO_DOC=tarantool/doc#3661

For now it's impossible to drop created alert in any way except manual searching for _alerts table. However, we need to drop alerts on missing names, when the names are set. Let's introduce simple key-value alerts in order to easily drop them by key. Needed for tarantool#8978 NO_DOC=internal NO_TEST=following commit NO_CHANGELOG=internal

For now it's impossible to use config module in order to recover from snaps, which don't have names set in them. Calling box.cfg with names on recovery fails with MISMATCH error, which is caused by difficult implementation of setting names on first box.cfg, as names can be set only on rw instance. This commit doesn't call box.cfg with names, if these names are missing from the snapshot file. Instead it creates a fiber, which will set names, when it's possible to do so. Only master sets the names for the whole cluster. Closes tarantool#8978 NO_DOC=tarantool/doc#3661 NO_CHANGELOG=following commits

This commit moves all code, related to working with versions and which was used in box/lua/upgrade.lua, into a separate module and exports it to Lua API as 'internal.version' This is needed, as in the following commit we set names automatically only when schema version is more than 3.0.0. This module is used their in order to avoid code duplication. Follow-up tarantool#8978 NO_DOC=internal NO_TEST=<already tested> NO_CHANGELOG=internal

It's impossible to set names on Tarantool below 3.0.0, as all DDL is forbidden before schema upgrade. Let's make names NoOp on schema below Tarantool 3.0.0 and set names automatically only schema upgrade is done. Follow-up tarantool#8978 NO_DOC=tarantool/doc#3661 NO_CHANGELOG=following commits

@TarantoolBot

This commit introduces the new method for xlog module: xlog.meta(). It opens an xlog file, reads and returns the meta block of the file, which includes its filetype, instance_uuid and vclocks. It's needed in order to introduce names checking in the following commit. @TarantoolBot document Title: xlog.meta([file-name]) method Description: Open an xlog file, and return its meta block. Possible errors: File does not contain properly formatted snapshot or write-ahead-log information. Needed for tarantool#8978 Example: ```lua tarantool> xlog = require('xlog') --- ... tarantool> xlog.meta('00000000000000000000.snap') --- - filetype: SNAP prev_vclock: {} instance_uuid: 87b2e60f-275c-4efa-9b0e-e9562e309692 vclock: {} ... ```

Currently only instance_uuid is validated before recovery process. All names and replicaset_uuid are checked only when recovery is done, which can take a long time. It can be frustrating to users, which have been waiting for several hours only to get name mismatch error. Let's read the small part of snapshot file before calling box.cfg in order to figure out, whether the names and uuids, passed to configuration match the ones, saved inside the snapshot. Let's also save all names, which are missing from snapshot in order to set them automatically in the future. Needed for tarantool#8978 NO_DOC=tarantool/doc#3661

For now it's impossible to drop created alert in any way except manual searching for _alerts table. However, we need to drop alerts on missing names, when the names are set. Let's introduce simple key-value alerts in order to easily drop them by key. Needed for tarantool#8978 NO_DOC=internal NO_TEST=following commit NO_CHANGELOG=internal

For now it's impossible to use config module in order to recover from snaps, which don't have names set in them. Calling box.cfg with names on recovery fails with MISMATCH error, which is caused by difficult implementation of setting names on first box.cfg, as names can be set only on rw instance. This commit doesn't call box.cfg with names, if these names are missing from the snapshot file. Instead it creates a fiber, which will set names, when it's possible to do so. Only master sets the names for the whole cluster. Closes tarantool#8978 NO_DOC=tarantool/doc#3661 NO_CHANGELOG=following commits

This commit moves all code, related to working with versions and which was used in box/lua/upgrade.lua, into a separate module and exports it to Lua API as 'internal.version' This is needed, as in the following commit we set names automatically only when schema version is more than 3.0.0. This module is used their in order to avoid code duplication. Follow-up tarantool#8978 NO_DOC=internal NO_TEST=<already tested> NO_CHANGELOG=internal

It's impossible to set names on Tarantool below 3.0.0, as all DDL is forbidden before schema upgrade. Let's make names NoOp on schema below Tarantool 3.0.0 and set names automatically only schema upgrade is done. Follow-up tarantool#8978 NO_DOC=tarantool/doc#3661 NO_CHANGELOG=following commits

For now it's impossible to use config module in order to recover from snaps, which don't have names set in them. Calling box.cfg with names on recovery fails with MISMATCH error, which is caused by difficult implementation of setting names on first box.cfg, as names can be set only on rw instance. This commit doesn't call box.cfg with names, if these names are missing from the snapshot file. Instead it creates a fiber, which will set names, when it's possible to do so. Only master sets the names for the whole cluster. Closes tarantool#8978 NO_DOC=tarantool/doc#3661 NO_CHANGELOG=following commits

This commit moves all code, related to working with versions and which was used in box/lua/upgrade.lua, into a separate module and exports it to Lua API as 'internal.version' This is needed, as in the following commit we set names automatically only when schema version is more than 3.0.0. This module is used their in order to avoid code duplication. Follow-up tarantool#8978 NO_DOC=internal NO_TEST=<already tested> NO_CHANGELOG=internal

When the name is manually set on master by replace in _cluster space, calling box.cfg on replica with the same name causes its hang. The problem is the fact, that resubscribe is initiated and waiting for APPLIER_REGISTERED status is started. As applier knows, that no registration should be done, this never happens. Let's don't initiate registration, when instance name is already set. Needed for tarantool#8978 NO_DOC=bugfix NO_CHANGELOG=not released yet

For now it's impossible to use config module in order to recover from snaps, which don't have names set in them. Calling box.cfg with names on recovery fails with MISMATCH error, which is caused by difficult implementation of setting names on first box.cfg, as names can be set only on rw instance. This commit doesn't call box.cfg with names, if these names are missing from the snapshot file. Instead it creates a fiber, which will set names, when it's possible to do so. Only master sets the names for the whole cluster. Closes tarantool#8978 NO_DOC=tarantool/doc#3661 NO_CHANGELOG=following commits

This commit moves all code, related to working with versions and which was used in box/lua/upgrade.lua, into a separate module and exports it to Lua API as 'internal.version' This is needed, as in the following commit we set names automatically only when schema version is more than 3.0.0. This module is used their in order to avoid code duplication. Follow-up tarantool#8978 NO_DOC=internal NO_TEST=<already tested> NO_CHANGELOG=internal

It's impossible to set names on Tarantool below 3.0.0, as all DDL is forbidden before schema upgrade. Let's make names NoOp on schema below Tarantool 3.0.0 and set names automatically only when schema upgrade is done. Follow-up tarantool#8978 NO_DOC=tarantool/doc#3661

Currently only instance_uuid is validated before recovery process. All names and replicaset_uuid are checked only when recovery is done, which can take a long time. It can be frustrating to users, which have been waiting for several hours only to get name mismatch error. Let's read the small part of snapshot file before calling box.cfg in order to figure out, whether the names and uuids, passed to configuration match the ones, saved inside the snapshot. During config reload there's no sense in reading snapshot file, as data is already saved inside spaces, let's read them. We still check that names in config and names in spaces don't contradict during config reload. This commit also introduces methods, for getting names, which are not set in snap (or memory), this'll be used in consequent commits to set names automatically. Needed for tarantool#8978 NO_DOC=tarantool/doc#3661