txm: tuning TX_MANAGER_GC_STEPS_SIZE results in various bugs

[Korablev77]

TX_MANAGER_GC_STEPS_SIZE (equals to 2 by default) is a constant that determines the number of iterations allowed for transaction manager to remove unused stories. Changing it should not result in visible changes of TXM. However, attempt to change it to different values leads to failing box/tx_man.test.lua:

TX_MANAGER_GC_STEPS_SIZE = 1:

#0  0x5630b5121719 in crash_collect+c5
#1  0x5630b512372a in crash_signal_cb+51
#2  0x7fce7432d980 in funlockfile+50
#3  0x5630b54430c8 in mslab_alloc+63
#4  0x5630b5443732 in mempool_alloc+188
#5  0x5630b5445083 in smalloc+b2
#6  0x5630b504ecd6 in memtx_tuple_new+3e4
#7  0x5630b5051297 in memtx_space_execute_update+1c4
#8  0x5630b4f31c01 in space_execute_dml+148
#9  0x5630b4f5f436 in box_process_rw+11c
#10 0x5630b4f65df8 in box_process1+172
#11 0x5630b4f8c26a in tarantoolsqlIncrementMaxid+b9
#12 0x5630b4ffcac3 in sqlVdbeExec+c01d
#13 0x5630b4ffe1d4 in sqlStep+19b
#14 0x5630b4ffe27b in sql_step+46
#15 0x5630b50c601d in sql_execute+b5
#16 0x5630b50c6418 in sql_prepare_and_execute+d5
#17 0x5630b503ad44 in lbox_execute+13b
#18 0x5630b51f1e2f in lj_BC_FUNCC+34
#19 0x5630b516c2db in lua_pcall+34d
#20 0x5630b5104e14 in luaT_call+29
#21 0x5630b50ffd62 in lua_fiber_run_f+74
#22 0x5630b4f1b8e3 in _ZL16fiber_cxx_invokePFiP13__va_list_tagES0_+1e
#23 0x5630b512b5cc in fiber_loop+84
#24 0x5630b5464e18 in coro_init+4c
[001] box/tx_man.test.lua                                             
[001] 
[001] [Instance "tx_man" killed by signal: 6 (SIGABRT)]

TX_MANAGER_GC_STEPS_SIZE = 4:

[001] box/tx_man.test.lua                                             [ fail ]
[001] 
[001] Test failed! Result content mismatch:
[001] --- box/tx_man.result	Tue Jul 20 21:09:34 2021
[001] +++ var/rejects/box/tx_man.reject	Tue Jul 20 21:13:34 2021
[001] @@ -428,11 +428,11 @@
[001]   | ...
[001]  tx1("i2:select{}")
[001]   | ---
[001] - | - - [[3, 1], [2, 2], [1, 3]]
[001] + | - - [[3, 1], [2, 2], [1, 3], [1, 11]]
[001]   | ...
[001]  tx2("i2:select{}")
[001]   | ---
[001] - | - - [[3, 1], [2, 2], [1, 3]]
[001] + | - - [[3, 1], [2, 2], [1, 3], [1, 11]]
[001]   | ...
[001]  tx1("s:replace{2, 4}")
[001]   | ---
[001] @@ -440,11 +440,11 @@
[001]   | ...
[001]  tx1("i2:select{}")
[001]   | ---
[001] - | - - [[3, 1], [1, 3], [2, 4]]
[001] + | - - [[3, 1], [1, 3], [2, 4], [1, 11]]
[001]   | ...
[001]  tx2("i2:select{}")
[001]   | ---
[001] - | - - [[3, 1], [2, 2], [1, 3]]
[001] + | - - [[3, 1], [2, 2], [1, 3], [1, 11]]
[001]   | ...
[001]  tx1("s:delete{1}")
[001]   | ---
[001] @@ -452,11 +452,11 @@
[001]   | ...
[001]  tx1("i2:select{}")
[001]   | ---
[001] - | - - [[3, 1], [2, 4]]
[001] + | - - [[3, 1], [2, 4], [1, 11]]
[001]   | ...

TX_MANAGER_GC_STEPS_SIZE = 6:

[001] box/tx_man.test.lua                                             [ fail ]
[001] 
[001] Test failed! Result content mismatch:
[001] --- box/tx_man.result	Tue Jul 20 21:09:34 2021
[001] +++ var/rejects/box/tx_man.reject	Tue Jul 20 21:46:22 2021
[001] @@ -512,7 +512,8 @@
[001]   | ...
[001]  tx1("s:replace{3, 3}")
[001]   | ---
[001] - | - - [3, 3]
[001] + | - - {'error': 'Duplicate key exists in unique index "sec" in space "test" with old
[001] + |       tuple - [1, 3] and new tuple - [3, 3]'}
[001]   | ...
[001]  tx2("s2:replace{4, 4}")
[001]   | ---

[alyapunov]

Crash with TX_MANAGER_GC_STEPS_SIZE = 1 was reproduced in #6274.
Bug with TX_MANAGER_GC_STEPS_SIZE = 4 is not reproducing anymore, perhaps some other patch fixes it.
Bug with TX_MANAGER_GC_STEPS_SIZE = 6+ (I've checked up to 10) can be reproduced with:

s = box.schema.create_space('test')
primary = s:create_index('pk', {parts={1, 'uint'}})
secondary = s:create_index('sk', {parts={2, 'uint'}})

-- Make a big transaction to make lots of garbage at once
tx1:begin()
for i = 1,10 do tx1('s:replace{i+100,i+100}') tx1('s:delete{i+100}') end
tx1:commit()

tx1:begin()
tx1("s:replace{1, 2, 'magic'}")
s:replace{1, 1, 'magic'} -- now {1, 2 ,'magic'} overwrites {1, 1, 'magic'}
tx1('s:delete{1}')
tx1:commit() -- succeeds

-- Make lost of changes to cause GC to run
for i = 11,50 do s:replace{i+100,i+100} s:delete{i+100} end

secondary:select{1} -- must be empty since we deleted {1}
s:replace{3, 1, 'magic'} -- must be OK.
s:drop()