[Bug]: Unable to retrieve an order from a datanode restored from network history

[MuthuVega]

Problem encountered

Unable to retrieve an order from a datanode restored from network history from a block higher than the block when the limit order was cerated.

Scenario:
Create a limit order on block 200
Let that order be unfilled
Let time pass by and the network reached block 500
Now create a new datanode from latest network history , say block 500
Now retrieve for the order

Observed behaviour

Unable to retrieve an order from a datanode restored from network history from a block higher than the block when the limit order was cerated.

Expected behaviour

Should be able to retrieve the order details form a restored datanode as the order is unfilled and is current.

Steps to reproduce

Create a limit order on block 200
Let that order be unfilled
Let time pass by and the network reached block 500
Now create a new datanode from latest network history , say block 500
Now retrieve for the order

Run the below system test:

System Test Branch - fix_failing_datanode_iceberg_test
System Test Name -test_datanode_normal_orders

Software version

System Test Branch - fix_failing_datanode_iceberg_test

Failing test

test_datanode_normal_orders

Jenkins run

https://jenkins.ops.vega.xyz/job/common/job/system-tests-wrapper/77714/testReport/junit/tests.datanode/datanode_test/test_datanode_normal_orders/

Configuration used

No response

Relevant log output

2023-06-23 13:05:40.342:[INFO] [MainProcess] [MainThread] [vega_commands.py]:[datanode_decentralised_history_list_history_segments]:[ 837] /jenkins/workspace/common/system-tests-wrapper/networkdata/testnet/../vega datanode network-history show --home /jenkins/workspace/common/system-tests-wrapper/networkdata/testnet/data-node/node6 -s
2023-06-23 13:05:40.721:[INFO] [MainProcess] [MainThread] [vega_commands.py]:[datanode_decentralised_history_list_history_segments]:[ 842] History Segments for node /jenkins/workspace/common/system-tests-wrapper/networkdata/testnet/data-node/node6: Available contiguous history spans:

Contiguous history from block height 639 to 814, from segment id: QmRKY2SysjKzFPTSJ8rXaM8iL6eHv2KAxEdY2CYCbQzF1D to QmUvW1MrBkQp5JFrUZg9Axq8HZpxhoLuboAbbamYqP9QZs

639 to 660, id: QmRKY2SysjKzFPTSJ8rXaM8iL6eHv2KAxEdY2CYCbQzF1D, previous segment id: QmP4oHWGybRzevZP1UKYMsFj5SrSh4cgFgD9SDsBHxSkxU
661 to 682, id: QmRCXyUhhAutPftNwtjm5eEBCWZcXubHE46xtUmXZAVEWX, previous segment id: QmRKY2SysjKzFPTSJ8rXaM8iL6eHv2KAxEdY2CYCbQzF1D
683 to 704, id: QmfL8ytEByGCz1fUSWc11aPeCfDa3KnWTnm5ZrwagDyjPN, previous segment id: QmRCXyUhhAutPftNwtjm5eEBCWZcXubHE46xtUmXZAVEWX
705 to 726, id: QmX7RT5Xgzj1wwygMv5epZbGvKP6dBbMeLFCNBWgdPzYW8, previous segment id: QmfL8ytEByGCz1fUSWc11aPeCfDa3KnWTnm5ZrwagDyjPN
727 to 748, id: QmRJxrTJfPTmccu1Rrt557cxDUiUVgMLBW2bDBXwhiiPfN, previous segment id: QmX7RT5Xgzj1wwygMv5epZbGvKP6dBbMeLFCNBWgdPzYW8
749 to 770, id: QmeUqm9cspsqnL3BasEBe8DKrFGXQmBqaKH1DLKr7X14RH, previous segment id: QmRJxrTJfPTmccu1Rrt557cxDUiUVgMLBW2bDBXwhiiPfN
771 to 792, id: QmdRnYfkDZe6fF4EzCJfchjAQFYJamDs2ojvQFCQyb7t4N, previous segment id: QmeUqm9cspsqnL3BasEBe8DKrFGXQmBqaKH1DLKr7X14RH
793 to 814, id: QmUvW1MrBkQp5JFrUZg9Axq8HZpxhoLuboAbbamYqP9QZs, previous segment id: QmdRnYfkDZe6fF4EzCJfchjAQFYJamDs2ojvQFCQyb7t4N

Datanode currently has data from block height 639 to 814

2023-06-23 13:05:40.739:[INFO] [MainProcess] [MainThread] [    network.py]:[wait_until_ready]:[ 778] waiting for testnet-nodeset-full-6-full to replay chain, max timeout 837 seconds
2023-06-23 13:05:41.743:[INFO] [MainProcess] [MainThread] [    network.py]:[core_connection]:[ 479] connecting to testnet-nodeset-full-6-full
2023-06-23 13:05:41.758:[INFO] [MainProcess] [MainThread] [    network.py]:[wait_until_ready]:[ 784] no block height in statistics, node is maybe still starting
2023-06-23 13:05:42.763:[INFO] [MainProcess] [MainThread] [    network.py]:[wait_until_ready]:[ 784] no block height in statistics, node is maybe still starting
2023-06-23 13:05:43.766:[INFO] [MainProcess] [MainThread] [    network.py]:[wait_until_ready]:[ 784] no block height in statistics, node is maybe still starting
2023-06-23 13:05:44.769:[INFO] [MainProcess] [MainThread] [    network.py]:[wait_until_ready]:[ 784] no block height in statistics, node is maybe still starting
2023-06-23 13:05:45.773:[INFO] [MainProcess] [MainThread] [    network.py]:[wait_until_ready]:[ 784] no block height in statistics, node is maybe still starting
2023-06-23 13:05:46.775:[INFO] [MainProcess] [MainThread] [    network.py]:[wait_until_ready]:[ 784] no block height in statistics, node is maybe still starting
2023-06-23 13:05:47.780:[INFO] [MainProcess] [MainThread] [    network.py]:[wait_until_ready]:[ 784] no block height in statistics, node is maybe still starting
2023-06-23 13:05:48.784:[INFO] [MainProcess] [MainThread] [    network.py]:[wait_until_ready]:[ 784] no block height in statistics, node is maybe still starting
2023-06-23 13:05:49.787:[INFO] [MainProcess] [MainThread] [    network.py]:[wait_until_ready]:[ 784] no block height in statistics, node is maybe still starting
2023-06-23 13:05:50.791:[INFO] [MainProcess] [MainThread] [    network.py]:[wait_until_ready]:[ 784] no block height in statistics, node is maybe still starting
2023-06-23 13:05:51.793:[INFO] [MainProcess] [MainThread] [    network.py]:[wait_until_ready]:[ 784] no block height in statistics, node is maybe still starting
2023-06-23 13:05:52.795:[INFO] [MainProcess] [MainThread] [    network.py]:[wait_until_ready]:[ 784] no block height in statistics, node is maybe still starting
2023-06-23 13:05:53.798:[INFO] [MainProcess] [MainThread] [    network.py]:[wait_until_ready]:[ 784] no block height in statistics, node is maybe still starting
2023-06-23 13:05:54.802:[INFO] [MainProcess] [MainThread] [    network.py]:[wait_until_ready]:[ 787] node testnet-nodeset-full-6-full block-height 822 target 837
2023-06-23 13:05:55.805:[INFO] [MainProcess] [MainThread] [    network.py]:[wait_until_ready]:[ 787] node testnet-nodeset-full-6-full block-height 840 target 837
2023-06-23 13:05:55.811:[INFO] [MainProcess] [MainThread] [    network.py]:[datanode_connection]:[ 519] connecting to datanode testnet-nodeset-full-6-full at 127.0.0.1:3067
2023-06-23 13:06:25.816:[INFO] [MainProcess] [MainThread] [    network.py]:[get_datanode_order_by_ref]:[ 756] Order not found via order_ref='test_datanode_normal_orders-bc368b65'
FAILED                                                                   [100%]2023-06-23 13:06:26.383:[INFO] [MainProcess] [MainThread] [   conftest.py]:[pytest_runtest_makereport]:[  62] Marking test as failed for use in tearDown

[MuthuVega]

The iceberg order retrieved from the Live orders does NOT contain the iceberg details

FYI - Jenkins Link

https://jenkins.ops.vega.xyz/job/common/job/system-tests-wrapper/78040/testReport/tests.datanode/datanode_test/test_datanode_iceberg_orders/