Сannot recover from an anonymous replica after writing to a local space due to an on_replace trigger for a global space. #8746

yanshtunder · 2023-06-07T09:36:04Z

If at an anonymous replica due to an on_replace trigger on the global space write is made to the local space, then recovering will be impossible. Run the tarantool like below:

------------
-- Master
------------

-- Step 1
box.cfg{
    listen = 3301,
    replication = {3301, 3302},
}
box.schema.user.grant('guest', 'replication')

-- Step 3   
box.schema.space.create('loc', {is_local = true})
box.space.loc:create_index('pk')
box.schema.space.create('glob')
box.space.glob:create_index('pk')

-- Step 5
box.space.glob:insert{1}


---------------------
-- Anonymous replica
---------------------

-- Step 2
box.cfg{
    listen = 3302,
    replication = {3301, 3302},
    replication_anon = true,
    read_only = true,
}

-- Step 4
box.space.glob:on_replace(function()
    box.space.loc:replace{42}
end)

Now try to recover from the anonymous replica. If you recover without force_recovery, you will get error:

2023-06-07 10:53:32.787 [8125] main/103/interactive I> recover from `./00000000000000000004.xlog'
2023-06-07 10:53:32.788 [8125] main/103/interactive box.cc:822 E> error at request: {type: 'INSERT', replica_id: 1, lsn: 6, space_id: 513, index_id: 0, tuple: [42]}
2023-06-07 10:53:32.788 [8125] main/103/interactive box.cc:765 E> XlogError: found a first global row in a transaction with LSN/TSN mismatch
2023-06-07 10:53:32.788 [8125] main/103/interactive F> can't initialize storage: found a first global row in a transaction with LSN/TSN mismatch

If you recover with force_recovery = true, you will get error:

2023-06-07 12:32:19.031 [16600] main/103/interactive I> recover from `./00000000000000000004.xlog'
2023-06-07 12:32:19.031 [16600] main/103/interactive box.cc:822 E> error at request: {type: 'INSERT', replica_id: 1, lsn: 6, space_id: 513, index_id: 0, tuple: [42]}
2023-06-07 12:32:19.031 [16600] main/103/interactive recovery.cc:292 E> skipping row {1: 6}
2023-06-07 12:32:19.031 [16600] main/103/interactive box.cc:765 E> XlogError: found a first global row in a transaction with LSN/TSN mismatch
tarantool: ./src/box/recovery.cc:280: void recover_xlog(recovery*, xstream*, const vclock*): Assertion `row.replica_id != 0 || row.group_id == GROUP_LOCAL' failed.

The text was updated successfully, but these errors were encountered:

Closes tarantool#8746

Transaction boundaries were not updated correctly for transactions in which local space writes were made from a replication trigger. Fix transaction boundary calculations for such cases. Closes tarantool#8746 NO_DOC=bugfix NO_CHANGELOG=covered by the next commit

Force recovery first tries to collect all rows of a transaction into a single list, and only then applies those rows. The problem was that it collected rows based on the row replica_id. For local rows replica_id is set to 0, but actually such rows can be part of a transaction coming from any instance. Fix recovery of such rows Follow-up tarantool#8746 Follow-up tarantool#7932 NO_DOC=bugfix NO_CHANGELOG=the broken behaviour couldn't be seen due to bug tarantool#8746

Transaction boundaries were not updated correctly for transactions in which local space writes were made from a replication trigger. Fix transaction boundary calculations for such cases. Closes tarantool#8746 NO_DOC=bugfix NO_CHANGELOG=covered by the next commit

Force recovery first tries to collect all rows of a transaction into a single list, and only then applies those rows. The problem was that it collected rows based on the row replica_id. For local rows replica_id is set to 0, but actually such rows can be part of a transaction coming from any instance. Fix recovery of such rows Follow-up tarantool#8746 Follow-up tarantool#7932 NO_DOC=bugfix NO_CHANGELOG=the broken behaviour couldn't be seen due to bug tarantool#8746

Transaction boundaries were not updated correctly for transactions in which local space writes were made from a replication trigger. Existing transaction boundaries and row flags from the master were written as is on the replica. Actually, the replica should recalculate transaction boundaries and even WAIT_SYNC/WAIT_ACK flags. Transaction boundaries should be recalculated when a replica appends a local write at the end of the master's transaction, and WAIT_SYNC/WAIT_ACK should be overwritten when nopifying synchronous transactions coming from an old term. The latter fix has uncovered the bug in skipping outdated synchronous transactions: if one replica replaces a transaction from an old term with NOPs and then passes that transaction to the other replica, the other replica raises a split brain error. It believes the NOPs are an async transaction form an old term. This worked before the fix, because the rows were written with the original WAIT_ACK = true bit. Now this is fixed properly: we allow fully NOP async tranasctions from the old term. Closes tarantool#8746 NO_DOC=bugfix NO_CHANGELOG=covered by the next commit