Additional fixes/tests for irreversible mode

[arhag]

PR #6624 made changes to enable irreversible mode to work.

However, that code does not support various corner cases of irreversible mode and particular switching to and from different modes.

-) Replaying in irreversible mode does not work properly. Duplicate blocks problems are encountered in the fork database if there are reversible blocks available to replay.
-) Even without reversible blocks, replay does not end with the latest irreversible considered as the last_irreversible_block.

There are also likely issues with switching between irreversible mode and speculative mode without a replay.
-) It should be possible to cleanly shutdown nodeos running in speculative, head, or read-only mode, and then start nodeos up again without a replay in irreversible mode.
-) It should also be possible to cleanly shutdown nodeos running in irreversible mode, and then start nodeos up again without a replay in speculative, head, or read-only mode.

The code changes supporting the above behaviors should also be accompanies by additional automated tests:

1. A test that shuts down a controller in speculative mode and then restarts it in irreversible mode. After startup, the fork DB pending head should be the same as before and the state should be at the irreversible block height which should be the same as what it was when the controller was originally shut down.

2. A test that shuts down a controller in irreversible mode and then restarts it in speculative mode. After startup, the fork DB pending head should be the same as before and the start should have automatically replayed the reversible blocks that were available in the saved fork database.

3. A test that shuts down a controller in speculative mode and then restarts with a forced replay in irreversible mode. The replay should only go up to the last irreversible block. However, the reversible blocks should still be added to the fork database. The pending fork DB head should end up being the same as what it was before the controller was originally shutdown. The state should remain at the irreversible block height which should be the same as what it was when the controller was originally shut down.

4. A variant of test 3 in which after shutdown but prior to restarting the reversible blocks database is removed. In this case after resuming the controller with a forced replay, it should replay up to the last irreversible block, which should be the same as what it was when the controller was originally shut down. However, this time the pending fork DB head should just be pointing to the root, which represent the last irreversible block.

The tests above should first change producers from eosio to at least four producers and producer enough blocks for the active schedule to switch over and then some more so that the blocks of the new producer round become irreversible, prior to shutting down the controller.

These combinations should also then be tested with actual nodeos instances rather than only building tests using the unit testing framework.

[ramtej]

I played a bit with the irreversible mode, especially with replaying blocklog. Besides other issues it seems that nodeos in irreversible mode leaks GBs of memory.

[crazybits]

when would the irreversible mode feature be available?

[andriantolie]

Based on the above description, 8 test cases are added in #6843

[andriantolie]

Additional test case:

1. Generate some blocks with a multiple producer schedule (its fine if all producers are on the same node). Backup the blocks directory.
2. Switch to irreversible mode successfully. Make sure the head block is correctly pointing to the LIB. Take a portable snapshot.
3. Switch to a clean data directory and copy over the backup blocks directory. Then start nodeos from the portable snapshot in speculative mode. Ensure it automatically replays the reversible blocks.
4. Allow more blocks to be generated for the nodeos from step 3 such that LIB advances. Then cleanly shut down that nodeos.
5. Restart the nodeos once again from the same portable snapshot that was taken earlier. Ensure it automatically replays some of the irreversible blocks as well as the reversible blocks.

[arhag]

Resolved by #6842 and #6843.