Recovery: force recovery should try to recover mixed transactions from WAL #7932

elhimov · 2022-11-14T10:14:01Z

Transaction boundaries in WAL were introduced in 2.1.2 (872d6f1), but replication applier started to apply transaction atomically only in 2.2.1(e94ba2e), so within this range it is possible that transactions are mixed in WAL. There was no problem with them until 2.8.1 because recovering process was still working row-by-row, but in 2.8.1 (9311113) it became transactional as well so starting from 2.8.1 tarantool fails to handle mixed transactions from WAL.

Recently this problem arose here

I find it difficult to provide 100% reproducer here, but at least you need:

Setup full mesh cluster of master replicas (the more the better) of any version from the range [2.1.2, 2.2.0]
Write various multi-component transactions simultaneously on different replicas
Upgrade one instance to 2.8.1 or later (w/o snapshots). If mixed transactions have been produced in WAL at step 2, upgraded instance will fail to recover from WAL

Mixed transactions is obviously the one of unexpected scenarios in modern tarantool, so we should not handle it in normal mode, and we take care of it only in force recovery mode. But current 'force recovery' treats all the rows of the unexpected transaction as unexpected and thus just skips the entire transaction row-by-row. However it seems we have enough information in this case to handle it more data-friendly, so I'd suggest to improve force recovery to avoid data loss, or at least try to avoid

The text was updated successfully, but these errors were encountered:

@TarantoolBot

See the docbot request for details. Closes tarantool#7932 @TarantoolBot document Title: correct recovery of mixed transactions Transaction boundaries in WAL were introduced in 2.1.2 (`872d6f1`), but replication applier started to apply transaction atomically only in 2.2.1(`e94ba2e`), so within this range it is possible that transactions are mixed in WAL. There was no problem with them until 2.8.1 because recovering process was still working row-by-row, but in 2.8.1 (`9311113`) it became transactional as well so starting from 2.8.1 tarantool fails to handle mixed transactions from WAL. Mixed transactions is obviously the one of unexpected scenarios in modern tarantool, so we should not handle it in normal mode, and we take care of it only in force recovery mode. But current 'force recovery' treats all the rows of the unexpected transaction as unexpected and thus just skips the entire transaction row-by-row. However we have enough information in this case to handle it more data-friendly.

@TarantoolBot

See the docbot request for details. Closes tarantool#7932 @TarantoolBot document Title: correct recovery of mixed transactions Transaction boundaries in WAL were introduced in 2.1.2 (`872d6f1`), but replication applier started to apply transaction atomically only in 2.2.1(`e94ba2e`), so within this range it is possible that transactions are mixed in WAL. There was no problem with them until 2.8.1 because recovering process was still working row-by-row, but in 2.8.1 (`9311113`) it became transactional as well so starting from 2.8.1 tarantool fails to handle mixed transactions from WAL. Mixed transactions is obviously the one of unexpected scenarios in modern tarantool, so we should not handle it in normal mode, and we take care of it only in force recovery mode. But current 'force recovery' treats all the rows of the unexpected transaction as unexpected and thus just skips the entire transaction row-by-row. However we have enough information in this case to handle it more data-friendly. Note: To replication do the following. Recover the broken xlog node by setting `force_recovery` to `true`. Then do replica rebootstrap.

A new macro has been introduced because OOM errors are not handled in the code. Needed for tarantool#7932 NO_DOC=internal NO_TEST=internal NO_CHANGELOG=internal

@TarantoolBot

See the docbot request for details. Closes tarantool#7932 @TarantoolBot document Title: correct recovery of mixed transactions Transaction boundaries in WAL were introduced in 2.1.2 (`872d6f1`), but replication applier started to apply transaction atomically only in 2.2.1(`e94ba2e`), so within this range it is possible that transactions are mixed in WAL. There was no problem with them until 2.8.1 because recovering process was still working row-by-row, but in 2.8.1 (`9311113`) it became transactional as well so starting from 2.8.1 tarantool fails to handle mixed transactions from WAL. Mixed transactions is obviously the one of unexpected scenarios in modern tarantool, so we should not handle it in normal mode, and we take care of it only in force recovery mode. But current 'force recovery' treats all the rows of the unexpected transaction as unexpected and thus just skips the entire transaction row-by-row. However we have enough information in this case to handle it more data-friendly. Note: Let there be two nodes (`node#1` and `node#2`). And let the data be replicated from `node#1` to `node#2`. Suppose that at some point in time, `node#1` is restoring data from an xlog containing mixed transactions. To replicate data from `node#1` to `node#2`, do the following: 1) Disable `node#2` 2) Start `node#1` by setting `force_recovery` to `true` 3) Connect `node#2`

@TarantoolBot

See the docbot request for details. Closes tarantool#7932 @TarantoolBot document Title: correct recovery of mixed transactions Transaction boundaries in WAL were introduced in 2.1.2 (`872d6f1`), but replication applier started to apply transaction atomically only in 2.2.1(`e94ba2e`), so within this range it is possible that transactions are mixed in WAL. There was no problem with them until 2.8.1 because recovering process was still working row-by-row, but in 2.8.1 (`9311113`) it became transactional as well so starting from 2.8.1 tarantool fails to handle mixed transactions from WAL. Mixed transactions is obviously the one of unexpected scenarios in modern tarantool, so we should not handle it in normal mode, and we take care of it only in force recovery mode. But current 'force recovery' treats all the rows of the unexpected transaction as unexpected and thus just skips the entire transaction row-by-row. However we have enough information in this case to handle it more data-friendly. Note: Let there be two nodes (`node#1` and `node#2`). And let the data be replicated from `node#1` to `node#2`. Suppose that at some point in time, `node#1` is restoring data from an xlog containing mixed transactions. To replicate data from `node#1` to `node#2`, do the following: 1) Disable `node#2` 2) Start `node#1` by setting `force_recovery` to `true` 3) Connect `node#2`

@TarantoolBot

See the docbot request for details. Closes tarantool#7932 @TarantoolBot document Title: correct recovery of mixed transactions Transaction boundaries in WAL were introduced in 2.1.2 (`872d6f1`), but replication applier started to apply transaction atomically only in 2.2.1(`e94ba2e`), so within this range it is possible that transactions are mixed in WAL. There was no problem with them until 2.8.1 because recovering process was still working row-by-row, but in 2.8.1 (`9311113`) it became transactional as well so starting from 2.8.1 tarantool fails to handle mixed transactions from WAL. Mixed transactions is obviously the one of unexpected scenarios in modern tarantool, so we should not handle it in normal mode, and we take care of it only in force recovery mode. But current 'force recovery' treats all the rows of the unexpected transaction as unexpected and thus just skips the entire transaction row-by-row. However we have enough information in this case to handle it more data-friendly. Note: Let there be two nodes (`node#1` and `node#2`). And let the data be replicated from `node#1` to `node#2`. Suppose that at some point in time, `node#1` is restoring data from an xlog containing mixed transactions. To replicate data from `node#1` to `node#2`, do the following: 1. Stop `node#2` and delete all xlog files from it 2. Restart `node#1` by setting `force_recovery` to `true` 3. Make `node#2` rejoin to `node#1` again (tarantoolgh-7932).

A new macro has been introduced because OOM errors are not handled in the code. Needed for tarantool#7932 NO_DOC=internal NO_TEST=internal NO_CHANGELOG=internal

@TarantoolBot

See the docbot request for details. Closes tarantool#7932 @TarantoolBot document Title: correct recovery of mixed transactions Transaction boundaries in WAL were introduced in 2.1.2 (`872d6f1`), but replication applier started to apply transaction atomically only in 2.2.1(`e94ba2e`), so within this range it is possible that transactions are mixed in WAL. There was no problem with them until 2.8.1 because recovering process was still working row-by-row, but in 2.8.1 (`9311113`) it became transactional as well so starting from 2.8.1 tarantool fails to handle mixed transactions from WAL. Mixed transactions is obviously the one of unexpected scenarios in modern tarantool, so we should not handle it in normal mode, and we take care of it only in force recovery mode. But current 'force recovery' treats all the rows of the unexpected transaction as unexpected and thus just skips the entire transaction row-by-row. However we have enough information in this case to handle it more data-friendly. Note: Let there be two nodes (`node#1` and `node#2`). And let the data be replicated from `node#1` to `node#2`. Suppose that at some point in time, `node#1` is restoring data from an xlog containing mixed transactions. To replicate data from `node#1` to `node#2`, do the following: 1. Stop `node#2` and delete all xlog files from it 2. Restart `node#1` by setting `force_recovery` to `true` 3. Make `node#2` rejoin to `node#1` again (tarantoolgh-7932).

@TarantoolBot

See the docbot request for details. Closes tarantool#7932 @TarantoolBot document Title: correct recovery of mixed transactions Transaction boundaries in WAL were introduced in 2.1.2 (`872d6f1`), but replication applier started to apply transaction atomically only in 2.2.1(`e94ba2e`), so within this range it is possible that transactions are mixed in WAL. There was no problem with them until 2.8.1 because recovering process was still working row-by-row, but in 2.8.1 (`9311113`) it became transactional as well so starting from 2.8.1 tarantool fails to handle mixed transactions from WAL. Mixed transactions is obviously the one of unexpected scenarios in modern tarantool, so we should not handle it in normal mode, and we take care of it only in force recovery mode. But current 'force recovery' treats all the rows of the unexpected transaction as unexpected and thus just skips the entire transaction row-by-row. However we have enough information in this case to handle it more data-friendly. Note: Let there be two nodes (`node#1` and `node#2`). And let the data be replicated from `node#1` to `node#2`. Suppose that at some point in time, `node#1` is restoring data from an xlog containing mixed transactions. To replicate data from `node#1` to `node#2`, do the following: 1. Stop `node#2` and delete all xlog files from it 2. Restart `node#1` by setting `force_recovery` to `true` 3. Make `node#2` rejoin to `node#1` again.

A new macro has been introduced because OOM errors are not handled in the code. Needed for tarantool#7932 NO_DOC=internal NO_TEST=internal NO_CHANGELOG=internal

@TarantoolBot

See the docbot request for details. Closes tarantool#7932 @TarantoolBot document Title: correct recovery of mixed transactions Transaction boundaries in WAL were introduced in 2.1.2 (`872d6f1`), but replication applier started to apply transaction atomically only in 2.2.1(`e94ba2e`), so within this range it is possible that transactions are mixed in WAL. There was no problem with them until 2.8.1 because recovering process was still working row-by-row, but in 2.8.1 (`9311113`) it became transactional as well so starting from 2.8.1 tarantool fails to handle mixed transactions from WAL. Mixed transactions is obviously the one of unexpected scenarios in modern tarantool, so we should not handle it in normal mode, and we take care of it only in force recovery mode. But current 'force recovery' treats all the rows of the unexpected transaction as unexpected and thus just skips the entire transaction row-by-row. However we have enough information in this case to handle it more data-friendly. Note: Let there be two nodes (`node#1` and `node#2`). And let the data be replicated from `node#1` to `node#2`. Suppose that at some point in time, `node#1` is restoring data from an xlog containing mixed transactions. To replicate data from `node#1` to `node#2`, do the following: 1. Stop `node#2` and delete all xlog files from it 2. Restart `node#1` by setting `force_recovery` to `true` 3. Make `node#2` rejoin to `node#1` again.

A new macro has been introduced because OOM errors are not handled in the code. Needed for tarantool#7932 NO_DOC=internal NO_TEST=internal NO_CHANGELOG=internal

@TarantoolBot

See the docbot request for details. Closes tarantool#7932 @TarantoolBot document Title: correct recovery of mixed transactions In this patch implemented correct recovery of mixed transactions. To do this, set `box.cfg.force_recovery` to `true`. If you need to revert to the old behavior, don't set the `force_recovery` option. What to do when one node feeds the other a xlog with mixed transactions? Let there be two nodes (`node#1` and `node#2`). And let the data be replicated from `node#1` to `node#2`. Suppose that at some point in time, `node#1` is restoring data from an xlog containing mixed transactions. To replicate data from `node#1` to `node#2`, do the following: 1. Stop `node#2` and delete all xlog files from it 2. Restart `node#1` by setting `force_recovery` to `true` 3. Make `node#2` rejoin to `node#1` again.

A new macro has been introduced because OOM errors are not handled in the code. Needed for tarantool#7932 NO_DOC=internal NO_TEST=internal NO_CHANGELOG=internal

@TarantoolBot

See the docbot request for details. Closes tarantool#7932 @TarantoolBot document Title: correct recovery of mixed transactions In this patch implemented correct recovery of mixed transactions. To do this, set `box.cfg.force_recovery` to `true`. If you need to revert to the old behavior, don't set the `force_recovery` option. What to do when one node feeds the other a xlog with mixed transactions? Let there be two nodes (`node#1` and `node#2`). And let the data be replicated from `node#1` to `node#2`. Suppose that at some point in time, `node#1` is restoring data from an xlog containing mixed transactions. To replicate data from `node#1` to `node#2`, do the following: 1. Stop `node#2` and delete all xlog files from it 2. Restart `node#1` by setting `force_recovery` to `true` 3. Make `node#2` rejoin to `node#1` again.

A new macro has been introduced because OOM errors are not handled in the code. Needed for tarantool#7932 NO_DOC=internal NO_TEST=internal NO_CHANGELOG=internal

@TarantoolBot

See the docbot request for details. Closes tarantool#7932 @TarantoolBot document Title: correct recovery of mixed transactions In this patch implemented correct recovery of mixed transactions. To do this, set `box.cfg.force_recovery` to `true`. If you need to revert to the old behavior, don't set the `force_recovery` option. What to do when one node feeds the other a xlog with mixed transactions? Let there be two nodes (`node#1` and `node#2`). And let the data be replicated from `node#1` to `node#2`. Suppose that at some point in time, `node#1` is restoring data from an xlog containing mixed transactions. To replicate data from `node#1` to `node#2`, do the following: 1. Stop `node#2` and delete all xlog files from it 2. Restart `node#1` by setting `force_recovery` to `true` 3. Make `node#2` rejoin to `node#1` again.

A new macros have been introduced because OOM errors are not handled in the code. Needed for tarantool#7932 NO_DOC=internal NO_TEST=internal NO_CHANGELOG=internal

@TarantoolBot

See the docbot request for details. Closes tarantool#7932 @TarantoolBot document Title: correct recovery of mixed transactions In this patch implemented correct recovery of mixed transactions. To do this, set `box.cfg.force_recovery` to `true`. If you need to revert to the old behavior, don't set the `force_recovery` option. What to do when one node feeds the other a xlog with mixed transactions? Let there be two nodes (`node#1` and `node#2`). And let the data be replicated from `node#1` to `node#2`. Suppose that at some point in time, `node#1` is restoring data from an xlog containing mixed transactions. To replicate data from `node#1` to `node#2`, do the following: 1. Stop `node#2` and delete all xlog files from it 2. Restart `node#1` by setting `force_recovery` to `true` 3. Make `node#2` rejoin to `node#1` again.

@TarantoolBot

See the docbot request for details. Closes tarantool#7932 @TarantoolBot document Title: correct recovery of mixed transactions In this patch implemented correct recovery of mixed transactions. To do this, set `box.cfg.force_recovery` to `true`. If you need to revert to the old behavior, don't set the `force_recovery` option. What to do when one node feeds the other a xlog with mixed transactions? Let there be two nodes (`node#1` and `node#2`). And let the data be replicated from `node#1` to `node#2`. Suppose that at some point in time, `node#1` is restoring data from an xlog containing mixed transactions. To replicate data from `node#1` to `node#2`, do the following: 1. Stop `node#2` and delete all xlog files from it 2. Restart `node#1` by setting `force_recovery` to `true` 3. Make `node#2` rejoin to `node#1` again.

A new macros have been introduced because OOM errors are not handled in the code. Needed for #7932 NO_DOC=internal NO_TEST=internal NO_CHANGELOG=internal

@TarantoolBot

See the docbot request for details. Closes #7932 @TarantoolBot document Title: correct recovery of mixed transactions In this patch implemented correct recovery of mixed transactions. To do this, set `box.cfg.force_recovery` to `true`. If you need to revert to the old behavior, don't set the `force_recovery` option. What to do when one node feeds the other a xlog with mixed transactions? Let there be two nodes (`node#1` and `node#2`). And let the data be replicated from `node#1` to `node#2`. Suppose that at some point in time, `node#1` is restoring data from an xlog containing mixed transactions. To replicate data from `node#1` to `node#2`, do the following: 1. Stop `node#2` and delete all xlog files from it 2. Restart `node#1` by setting `force_recovery` to `true` 3. Make `node#2` rejoin to `node#1` again.

sergepetrenko · 2023-06-19T09:01:43Z

Decided with @sergos this shouldn't go to 2.10, because it's too big of a change.
Still pushing to 2.11 to fix the possible problems with migration from 2.1-2.2 to the latest release of 2.x series

A new macros have been introduced because OOM errors are not handled in the code. Needed for #7932 NO_DOC=internal NO_TEST=internal NO_CHANGELOG=internal (cherry picked from commit 5385714)

@TarantoolBot

See the docbot request for details. Closes #7932 @TarantoolBot document Title: correct recovery of mixed transactions In this patch implemented correct recovery of mixed transactions. To do this, set `box.cfg.force_recovery` to `true`. If you need to revert to the old behavior, don't set the `force_recovery` option. What to do when one node feeds the other a xlog with mixed transactions? Let there be two nodes (`node#1` and `node#2`). And let the data be replicated from `node#1` to `node#2`. Suppose that at some point in time, `node#1` is restoring data from an xlog containing mixed transactions. To replicate data from `node#1` to `node#2`, do the following: 1. Stop `node#2` and delete all xlog files from it 2. Restart `node#1` by setting `force_recovery` to `true` 3. Make `node#2` rejoin to `node#1` again. (cherry picked from commit 2b1c871)

Added a new parameter `is_sync` to `box.begin()`, `box.commit()` and `box.atomic`. Thanks to this parameter can make the transaction synchronous. To do this set `is_sync = true`. `box.commit()` cannot change `is_sync` to `false` if the transaction was opened as synchronous or has writes to synchro spaces. By default, `is_sync = false`. Examples: ```Lua -- Sync transactions box.atomic({is_sync = true}, function() ... end) box.begin({is_sync = true}) ... box.commit({is_sync = false}) box.begin({is_sync = false}) ... box.commit({is_sync = true}) box.begin({is_sync = true}) ... box.commit({is_sync = true}) -- Async transactions box.atomic({is_sync = false}, function() ... end) box.begin({is_sync = false}) ... box.commit({is_sync = false}) box.begin() ... box.commit() ``` Closes tarantool#7932 NO_DOC=internal

Force recovery first tries to collect all rows of a transaction into a single list, and only then applies those rows. The problem was that it collected rows based on the row replica_id. For local rows replica_id is set to 0, but actually such rows can be part of a transaction coming from any instance. Fix recovery of such rows Follow-up tarantool#8746 Follow-up tarantool#7932 NO_DOC=bugfix NO_CHANGELOG=the broken behaviour couldn't be seen due to bug tarantool#8746

Force recovery first tries to collect all rows of a transaction into a single list, and only then applies those rows. The problem was that it collected rows based on the row replica_id. For local rows replica_id is set to 0, but actually such rows can be part of a transaction coming from any instance. Fix recovery of such rows Follow-up #8746 Follow-up #7932 NO_DOC=bugfix NO_CHANGELOG=the broken behaviour couldn't be seen due to bug #8746

Force recovery first tries to collect all rows of a transaction into a single list, and only then applies those rows. The problem was that it collected rows based on the row replica_id. For local rows replica_id is set to 0, but actually such rows can be part of a transaction coming from any instance. Fix recovery of such rows Follow-up #8746 Follow-up #7932 NO_DOC=bugfix NO_CHANGELOG=the broken behaviour couldn't be seen due to bug #8746 (cherry picked from commit 85df1c9)

A new macros have been introduced because OOM errors are not handled in the code. Needed for tarantool#7932 NO_DOC=internal NO_TEST=internal NO_CHANGELOG=internal (cherry picked from commit 5385714)

elhimov added the feature A new functionality label Nov 14, 2022

sergos assigned yanshtunder Jan 9, 2023

yanshtunder mentioned this issue Mar 30, 2023

replication: recovery mixed transacrtions #8518

Merged

sergos added the 2.10 Target is 2.10 and all newer release/master branches label May 17, 2023

sergepetrenko closed this as completed in #8518 Jun 19, 2023

sergepetrenko pushed a commit that referenced this issue Jun 19, 2023

trivia: add xlsregion_alloc macros and xlsregion_alloc_object

5385714

A new macros have been introduced because OOM errors are not handled in the code. Needed for #7932 NO_DOC=internal NO_TEST=internal NO_CHANGELOG=internal

sergepetrenko added 2.11 Target is 2.11 and all newer release/master branches and removed 2.10 Target is 2.10 and all newer release/master branches labels Jun 19, 2023

andreyaksenov mentioned this issue Jul 18, 2023

correct recovery of mixed transactions tarantool/doc#3526

Closed

yanshtunder mentioned this issue Jul 24, 2023

box: add is_sync option #8907

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Recovery: force recovery should try to recover mixed transactions from WAL #7932

Recovery: force recovery should try to recover mixed transactions from WAL #7932

elhimov commented Nov 14, 2022 •

edited

sergepetrenko commented Jun 19, 2023 •

edited

Recovery: force recovery should try to recover mixed transactions from WAL #7932

Recovery: force recovery should try to recover mixed transactions from WAL #7932

Comments

elhimov commented Nov 14, 2022 • edited

sergepetrenko commented Jun 19, 2023 • edited

elhimov commented Nov 14, 2022 •

edited

sergepetrenko commented Jun 19, 2023 •

edited