INDY-2025: Improve tests #1156
INDY-2025: Improve tests #1156
Conversation
Signed-off-by: Sergey Khoroshavin <sergey.khoroshavin@dsr-corporation.com>
Signed-off-by: Sergey Khoroshavin <sergey.khoroshavin@dsr-corporation.com>
| # Delay some commits on lagging node | ||
| some_commits = start_delaying(lagging_stasher, delay_some_commits(0, [2, 3])) | ||
| # Setup delayers | ||
| lagging_mid_commits = start_delaying(lagging_stasher, delay_3pc(after=3, before=6, msgs=Commit)) |
There was a problem hiding this comment.
- delay Commits for ppSeqNo=4 on NodeX
- increase the timeout for out of order commits on NodeX to 10000 secs
- send 5 batches
- make sure that NodeX ordered 3 of them
- make sure that others ordered all 5
- make sure that NodeX has out of order commits for ppSeqNo=5
- process delayed commits for view4 on NodeX
- make sure that NodeX ordered till batch ppSeqNo=4
- delay POOL catchup on NodeX
- start catchup on Node X
- make sure that audit catchup is done, and pool one is started
- call processOutOfOrderCommits by hand (emulating call by timeout)
- it should not do anything on the latest master
- it will order ppSeqNo=5 before starting Domain catchup on old code
- finish catchup (process delayed catchup msgs)
- call processOutOfOrderCommits by hand again (emulating call by timeout)
=> ppSeqNo=5 is caught up, all nodes have equal data, and NodeX can participate in more ordering
- on old code, NodeX will have duplicate txns for ppSeqNo=5: one from catchup, and one from outOfOrderCommits
Signed-off-by: Sergey Khoroshavin <sergey.khoroshavin@dsr-corporation.com>
…tchup Signed-off-by: Sergey Khoroshavin <sergey.khoroshavin@dsr-corporation.com>
Signed-off-by: Sergey Khoroshavin <sergey.khoroshavin@dsr-corporation.com>
| # Make sure pool is in expected state | ||
| for node in other_nodes: | ||
| assert node.master_replica.last_ordered_3pc == (0, init_pp_seq_no + 4) | ||
| assert lagging_node.master_replica.last_ordered_3pc == (0, init_pp_seq_no) |
There was a problem hiding this comment.
Why last ordered is not (0, init_pp_seq_no + 2) right after we unstashed and processed delayed commits?
There was a problem hiding this comment.
At this point lagging commits are just put into node inbox, but not processed. See next line where we actually wait until they are processed.
| with delay_rules(lagging_stasher, delay_catchup(ledger_id)): | ||
| # Start catchup | ||
| lagging_node.start_catchup() | ||
|
|
There was a problem hiding this comment.
Should we check that we call looper.runFor only when we are catching up the given ledger, not earlier?
There was a problem hiding this comment.
In this case I think additional check is not needed. Logic is as follows:
process_stashed_out_of_order_commitsis already scheduled to run in future- catchup is triggered
- during
runForcatch-up progresses as far as it can and stops runForruns long enough so that scheduled action fires
It is possible to implement this in a more robust way (but at the cost of being more tied to node internals) by doing following things:
- set
process_stashed_out_of_order_commitstimeout to "infinity" - start catchup
- wait until catchup enters expected stage
- manually fire
process_stashed_out_of_order_commits - remove catchup delayer and let it progress further
Signed-off-by: Sergey Khoroshavin <sergey.khoroshavin@dsr-corporation.com>
No description provided.