LSN for 1 is used twice or COMMIT order is broken when registering a replica

[Serpentian]

Run with ./test-run.py $(yes commit_order | head -200). Fails in approximately 1 out of 100 runs on my machine. Don't have stable reproducer, didn't investigate the issue.

Repro:

local t = require('luatest')
local cluster = require('luatest.replica_set')
local server = require('luatest.server')

local g = t.group('commit-order')

g.before_all(function()
    g.cluster = cluster:new({})
    local cfg = {
        replication_timeout = 0.1,
        replication = {
            server.build_listen_uri('replica_a', g.cluster.id),
            server.build_listen_uri('replica_b', g.cluster.id),
        },
    }
    g.replica_a = g.cluster:build_and_add_server({alias = 'replica_a', box_cfg = cfg})
    cfg.read_only = true
    g.replica_b = g.cluster:build_and_add_server({alias = 'replica_b', box_cfg = cfg})
    g.cluster:start()
end)

g.after_all(function()
    g.cluster:stop()
end)

g.test_commit_order = function(g)
    g.replica_a:exec(function()
        box.ctl.wait_rw()
        box.cfg{instance_name = 'replica_1_a', replicaset_name = 'replicaset_1'}
    end)
    g.replica_b:exec(function()
        box.cfg{instance_name = 'replica_1_b'}
    end)
end

Result:

[002] Test failed! Output from reject file /tmp/t/rejects/replication-luatest/commit_order.reject:
[002] Tarantool version is 3.1.0-entrypoint-227-gde80e0264f
[002] TAP version 13
[002] 1..1
[002] # Started on Mon Apr  8 16:30:00 2024
[002] # Starting group: commit-order
[002] replica_b | 2024-04-08 16:30:01.012 [329342] main/119/applier/unix/:/tmp/t/002_replication-luatest/rs-Q4DJjnmIy6tX/replica_a.sock F> LSN for 1 is used twice or COMMIT order is broken: confirmed: 4, new: 3, req: {type: 'REPLACE', replica_id: 1, lsn: 3, space_id: 272, index_id: 0, tuple: ["replicaset_name", "replicaset_1"]}
[002] not ok 1	commit-order.test_commit_order
[002] #   Peer closed
[002] #   stack traceback:
[002] #   	...tarantool/test/replication-luatest/commit_order_test.lua:31: in function 'commit-order.test_commit_order'
[002] #   	...
[002] #   	[C]: in function 'xpcall'
[002] #   artifacts:
[002] #   	replica_b -> /tmp/t/002_replication-luatest/artifacts/rs-Q4DJjnmIy6tX/replica_b-yEc9SYoQrXRw
[002] #   	replica_a -> /tmp/t/002_replication-luatest/artifacts/rs-Q4DJjnmIy6tX/replica_a-inKWP1n63_kp
[002] # Ran 1 tests in 0.864 seconds, 0 succeeded, 1 errored
[002] 

Probably other DDLs are also affected by this. Related closed issues: #4749, #2951

---

Reproduced in vshard's router-luatest/router_test.lua in test_named_config_identification

[Gerold103]

Here is a 100% test.

--
-- Instance 1
--
-- Step 1
--
box.cfg{
    listen = 3313,
    replication = {3313, 3314},
}
box.schema.user.grant('guest', 'super')
s = box.schema.create_space('test')
s:create_index('pk')
--
-- Step 4
--
s:replace{1}



--
-- Instance 2
--
-- Step 2
--
fiber = require('fiber')
box.cfg{
    listen = 3314,
    replication = {3313, 3314},
    read_only = true,
}
--
-- Step 3
--
box.error.injection.set("ERRINJ_WAL_DELAY", true)
--
-- Step 5
--
f = fiber.create(function() box.cfg{instance_name = 'replica'} end)
fiber.sleep(2)
box.error.injection.set("ERRINJ_WAL_DELAY", false)

That will crash in debug build here:

    frame #4: 0x00000001001c5177 tarantool`wal_assign_lsn(vclock_diff=0x0000000121d7fdd8, base=0x000000010094efc8, entry=0x00000001050273c0) at wal.c:1074:5
   1071						 " new lsn %lld", (*row)->replica_id,
   1072						 (long long)confirmed_lsn,
   1073						 (long long)(*row)->lsn);
-> 1074					assert(false);
   1075				} else {
   1076					vclock_follow(vclock_diff, (*row)->replica_id, diff);
   1077				}

and in release it would result in double LSN apply.

Same happens if the replica is anon and is trying to do replication_anon = false instead of setting a name.

The common problem is applier_register() which asks to send data starting from WAL vclock, but when it is started, there might be a WAL write in progress from the master which will do the registration. As a result, the master will send this row again, because it wasn't included into register request's vclock.

It seems the register must not use apply_final_join_tx() and should use the same mechanisms as the subscribe. I.e. lock the applier latch to wait until all the writing txns are fully written before trying to apply another one from the same replica. Or whatever else the subscribe is doing to avoid the same problem.