Unable to connect with net.box to 2.2 if box.schema.upgrade() has not been called. #4691

Kasen · 2019-12-18T11:26:28Z

Tarantool version:
2.2.1-137

OS version:
CentOS Linux release 7.6.1810
Bug description:
If I start 2.2 as a read-only replica of 1.10, then I can't connect to this node with error:

error: Space '277' does not exist

Steps to reproduce:

Start tarantool 1.10
Start the second instance of 1.10
Configure replication between these instances. The second instance must be read-only.
Restart the second instance with tarantool 2.2
Try to connect to the second instance with net.box

The text was updated successfully, but these errors were encountered:

rosik · 2020-01-10T20:37:23Z

@kyukhin could you please set a milestone and help with fixing that.

Totktonada · 2020-01-10T20:42:33Z

It is the next incarnation of #4307. We missed the case when a remote instance is run on a new tarantool version, but don't upgraded yet and so lack of _vcollation view.

A trivial fix is possible: we can just ignore an error of fetching _vcollation. We may try to strength this way with some kind of verification whether the error is transient or not. I'm not sure that this way is right, but I don't see any other one for now.

This patch introduces automatic schema upgrade. The main idea to apply box.schema.upgrade on the leader instance. We can't do it on each instance - it lead to replication conflicts. We can't delegate it to users becouse a bug: "Space 277 doesn't exist" (tarantool/tarantool#4691) still is not resolved. Closes #640

This patch introduces automatic schema upgrade. The main idea to apply box.schema.upgrade on the leader instance. We can't do it on each instance - it lead to replication conflicts. We can't delegate it to users because a bug: "Space 277 doesn't exist" (tarantool/tarantool#4691) still is not resolved. Closes #640

This patch introduces an option - `upgrade_schema` to perform an upgrade procedure if we want to upgrade tarantool version. The main idea to apply box.schema.upgrade on the leader instance. We can't do it on each instance - it lead to replication conflicts if we have 2 or more writable instances. We don't do it automatically to give an ability for user to check that instances works fine with new tarantool version but old schema. This thesis is not relevant just now because of "Space 277 doesn't exist" (tarantool/tarantool#4691). But I hope it will be solved soon. Closes #640

This patch introduces an option - `upgrade_schema` to perform an upgrade procedure if we want to upgrade tarantool version. The main idea to apply `box.schema.upgrade` on the leader (according to failover priority in topology confgig). We can't do it on each instance - it lead to replication conflicts if we have 2 or more writable instances. We don't do it automatically to give an ability for user to check that instances works fine with new tarantool version but old schema. This thesis is not relevant just now because of "Space 277 doesn't exist" error (tarantool/tarantool#4691). But I hope it will be solved soon. Also introduce new type of tests - "upgrade". The main idea it to try to bootstrap server from existing snapshots with older schema version. See also #587 Co-authored-by: Yaroslav Dynnikov <yaroslav.dynnikov@gmail.com>

After 2.2.0-633-gaa0964ae1 ('net.box: fix schema fetching from 1.10/2.1 servers') net.box expects that _vcollation system view exists on a tarantool server of 2.2.1+ version. This is however not always so: a server may be run on a new version of tarantool, but work on a schema of an old version. The situation with non last schema is usual for replication cluster in process of upgrading: all instances run on the new version of tarantool first (no auto-upgrade is performed by tarantools in a cluster). Then box.schema.upgrade() should be called, but the instances should be operable even before the call. Before the commit net.box was unable to connect a server if it is run on a schema without _vcollation system view (say, 2.1.3), but the server executable is of 2.2.1 version or newer. Follows up #4307 Fixes #4691

After 2.2.0-633-gaa0964ae1 ('net.box: fix schema fetching from 1.10/2.1 servers') net.box expects that _vcollation system view exists on a tarantool server of 2.2.1+ version. This is however not always so: a server may be run on a new version of tarantool, but work on a schema of an old version. The situation with non last schema is usual for replication cluster in process of upgrading: all instances run on the new version of tarantool first (no auto-upgrade is performed by tarantools in a cluster). Then box.schema.upgrade() should be called, but the instances should be operable even before the call. Before the commit net.box was unable to connect a server if it is run on a schema without _vcollation system view (say, 2.1.3), but the server executable is of 2.2.1 version or newer. Note: I trim tests from the commit to polish them a bit more, but include the fix itself to 2.4.1 release. Follows up #4307 Fixes #4691

After 2.2.0-633-gaa0964ae1 ('net.box: fix schema fetching from 1.10/2.1 servers') net.box expects that _vcollation system view exists on a tarantool server of 2.2.1+ version. This is however not always so: a server may be run on a new version of tarantool, but work on a schema of an old version. The situation with non last schema is usual for replication cluster in process of upgrading: all instances run on the new version of tarantool first (no auto-upgrade is performed by tarantools in a cluster). Then box.schema.upgrade() should be called, but the instances should be operable even before the call. Before the commit net.box was unable to connect a server if it is run on a schema without _vcollation system view (say, 2.1.3), but the server executable is of 2.2.1 version or newer. Note: I trim tests from the commit to polish them a bit more, but include the fix itself to 2.4.1 release. Follows up #4307 Fixes #4691 (cherry picked from commit 06edcbe)

See tarantool#4691 for the problem description. In brief: before the fix net.box expected _vcollation presence if tarantool's runtime on the server is new enough disregarding a server's schema version. The old schema version has no this view, so net.box was unable to connect. After tarantool#4691 net.box tolerates lack of the view. The fix was pushed without a test (see [1]) and now I want to pay this debt. [1]: https://lists.tarantool.org/pipermail/tarantool-patches/2020-April/016207.html Follows up tarantool#4307 Follows up tarantool#4691 Fixes tarantool#5482 NO_DOC=just a new test NO_CHANGELOG=see NO_DOC

See #4691 for the problem description. In brief: before the fix net.box expected _vcollation presence if tarantool's runtime on the server is new enough disregarding a server's schema version. The old schema version has no this view, so net.box was unable to connect. After #4691 net.box tolerates lack of the view. The fix was pushed without a test (see [1]) and now I want to pay this debt. [1]: https://lists.tarantool.org/pipermail/tarantool-patches/2020-April/016207.html Follows up #4307 Follows up #4691 Fixes #5482 NO_DOC=just a new test NO_CHANGELOG=see NO_DOC

kyukhin added the bug Something isn't working label Jan 14, 2020

kyukhin added this to the 2.2.3 milestone Jan 14, 2020

Totktonada self-assigned this Jan 23, 2020

rosik mentioned this issue Feb 18, 2020

Describe tarantool 1.10->2.x migration process tarantool/cartridge#587

Closed

2 tasks

Totktonada added the upgrade label Mar 13, 2020

Totktonada mentioned this issue Mar 13, 2020

Ensure that an instance with non-last schema is operational #4801

Open

olegrok mentioned this issue Mar 20, 2020

automatic schema upgrade support tarantool/cartridge#671

Merged

3 tasks

kyukhin closed this as completed in 06edcbe Apr 20, 2020

kyukhin mentioned this issue Nov 2, 2020

test for 'net.box: fix fetching of schema of an old version' #5482

Closed

Totktonada mentioned this issue Jan 9, 2023

test: verify net.box on a schema of version 2.1.3 #8119

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to connect with net.box to 2.2 if box.schema.upgrade() has not been called. #4691

Unable to connect with net.box to 2.2 if box.schema.upgrade() has not been called. #4691

Kasen commented Dec 18, 2019

rosik commented Jan 10, 2020

Totktonada commented Jan 10, 2020 •

edited

Unable to connect with net.box to 2.2 if box.schema.upgrade() has not been called. #4691

Unable to connect with net.box to 2.2 if box.schema.upgrade() has not been called. #4691

Comments

Kasen commented Dec 18, 2019

rosik commented Jan 10, 2020

Totktonada commented Jan 10, 2020 • edited

Totktonada commented Jan 10, 2020 •

edited