tarantool does not start: Tuple field 6 type does not match...

[stepan-mitkin]

tarantool server does not start

This is a catastrophe. Please help.

How it happened:

1. Stop tarantool using tarantoolctl
2. Start tarantool using tarantoolctl
3. tarantool refuses to start.

No modifications to the schema have been made recently.

Here is the log:

2017-11-23 21:29:42.436 [2633] main/101/dbg.lua I> systemd: NOTIFY_SOCKET variable is empty, skipping
2017-11-23 21:29:42.436 [2633] main/101/dbg.lua C> Tarantool 1.7.6-9-g4e960f7
2017-11-23 21:29:42.436 [2633] main/101/dbg.lua C> log level 5
2017-11-23 21:29:42.436 [2633] main/101/dbg.lua I> mapping 268435456 bytes for memtx tuple arena...
2017-11-23 21:29:42.484 [2633] main/101/dbg.lua I> mapping 134217728 bytes for vinyl tuple arena...
2017-11-23 21:29:42.486 [2633] iproto/101/main I> binary: bound to 0.0.0.0:3301
2017-11-23 21:29:42.487 [2633] main/101/dbg.lua I> recovery start
2017-11-23 21:29:42.487 [2633] main/101/dbg.lua I> recovering from `/dewt/data/00000000000006562404.snap'
2017-11-23 21:29:42.519 [2633] main/101/dbg.lua key_def.h:343 E> ER_FIELD_TYPE: Tuple field 6 type does not match one required by operation: expected string
2017-11-23 21:29:42.519 [2633] main/101/dbg.lua F> can't initialize storage: Tuple field 6 type does not match one required by operation: expected string

[stepan-mitkin]

Let me explain what "does not start" means in practice.
It means:

1. All data is lost forever.
2. The application does not work anymore.

This is the worst thing that can happen to a database

Thanks God it happened in the test environment, not in production.
But currently, if the production server stops, it will stop forever.

[stepan-mitkin]

I did some debugging. Here is what the stack looks like:

tuple_validate_raw (i == 5)
tuple_validate
memtx_space_check_format (the space name is "_space")
space_check_format_xc
ModifySpace::alter
alter_space_do
on_replace_dd_space

[kostja]

First, please, this is not a data loss, this looks like an upgrade issue. Beginning with 1.7.6 we started to enforce space:format data types, so if you have a string column in space:format, you won't be able to insert a number in it.

Please downgrade to the original version you used, fix the conflicting data or remove space formatting, then you should be able to upgrade OK.

[kostja]

You could also use 'force_recovery' to skip the offending tuple, but it is of course the last resort.

[stepan-mitkin]

First, please, this is not a data loss, this looks like an upgrade issue.

Huff... Thanks.

fix the conflicting data or remove space formatting

It's not data. It's metadata. A description of spaces in the database. The space name that tarantool complains on is "_space".

So 'force_recovery' does not help. I end up with an empty database if I use 'force_recovery'.
I'll try to downgrade and see what happens.

[stepan-mitkin]

Downgrade to 1.7.5 did not work.

2017-11-27 21:15:23.531 [16508] main/101/dbg.lua I> recovery start
2017-11-27 21:15:23.531 [16508] main/101/dbg.lua I> recovering from `/home/stipan/tmp/tara//00000000000093350726.snap'
2017-11-27 21:15:23.555 [16508] main/101/dbg.lua schema.h:184 E> ER_NO_SUCH_SPACE: Space '276' does not exist
2017-11-27 21:15:23.555 [16508] main/101/dbg.lua F> can't initialize storage: Space '276' does not exist

this is not a data loss, this looks like an upgrade issue.

This is an upgrade issue that lead to a data loss.

[alyapunov]

* stepan-mitkin <notifications@github.com> [17/11/27 23:21]:

Downgrade to 1.7.5 did not work. ``` 2017-11-27 21:15:23.531 [16508] main/101/dbg.lua I> recovery start 2017-11-27 21:15:23.531 [16508] main/101/dbg.lua I> recovering from `/home/stipan/tmp/tara//00000000000093350726.snap' 2017-11-27 21:15:23.555 [16508] main/101/dbg.lua schema.h:184 E> ER_NO_SUCH_SPACE: Space '276' does not exist 2017-11-27 21:15:23.555 [16508] main/101/dbg.lua F> can't initialize storage: Space '276' does not exist ``` > this is not a data loss, this looks like an upgrade issue. This _is_ an upgrade issue that lead to a data loss.

Could you send us your snapshot? We'll see what can be done. Thanks.

…

-- Konstantin Osipov, Moscow, Russia, +7 903 626 22 32 http://tarantool.org - www.twitter.com/kostja_osipov

[Gerold103]

It seems like you store in options of one of your spaces some data not allowed by a format. Can you please show your _space content? (You said, that you still have the same schema working on a production). It can be done by box.space._space:select{}.

[Gerold103]

I right now had tried to run 1.7.5 from 1.7.6 snapshot using force_recovery = true, and it has started ok.

[Gerold103]

Write me in the Telegram (@Gerold103). I have wrote a script that can help.

[stepan-mitkin]

The script helped. Thanks a lot!
Here is the script.
Run tarantool 1.7.5 with this:

box.cfg{force_recovery = true}

box.space._index:run_triggers(false)
box.space._space:run_triggers(false)

box.space._index:delete{276, 1}
box.space._index:delete{276, 0}
box.space._space:delete{276}
box.space._priv:delete{2, 'space', 276}

box.space._index:delete{284, 2}
box.space._index:delete{284, 1}
box.space._index:delete{284, 0}
box.space._space:delete{284}

box.space._index:delete{285, 0}
box.space._space:delete{285}

box.space._index:delete{340, 1}
box.space._index:delete{340, 0}
box.space._space:delete{340}

box.space._index:delete{330, 0}
box.space._space:delete{330}

box.space._schema:replace{'version', 1,7,4}
box.snapshot()

Stop tarantool.
Run tarantool 1.7.6 from that snapshot.
box.cfg { ... }
box.schema.upgrade()

[stepan-mitkin]

Here is how I will upgrade tarantool from now on:

1. Upgrade tarantool.
2. Stop tarantool.
3. Run tarantool with only this script

box.cfg { ... }
box.schema.upgrade()

4. Stop tarantool
5. Start tarantool with the normal application scripts.

Any deviation from this scenario may lead to upgrade issues.