Fix Mismatched BlockRoots In State Replay

[nisdas]

What type of PR is this?

Bug Fix

What does this PR do? Why is it needed?

• During times without finality the database doesn't return the same number of blocks and its
  respective roots in-order. This is especially true for unfinalized blocks.

Follows on from #7019 and #7028

Which issues(s) does this PR fix?

Other notes for review

[codecov]

Codecov Report

Merging #7559 into master will decrease coverage by 0.15%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #7559      +/-   ##
==========================================
- Coverage   61.86%   61.71%   -0.16%     
==========================================
  Files         424      424              
  Lines       30068    29884     -184     
==========================================
- Hits        18602    18443     -159     
+ Misses       8487     8481       -6     
+ Partials     2979     2960      -19     

[terencechain]

Then the issue is on the block roots retrieval in db? Should we fix it on that end? Can we reproduce this with a DB test?

Not 100% sure about this because it feels like the real issue is getting masked away and we should fix it instead or at least open a tracking issue

[prestonvanloon]

Please add regression test if possible

[nisdas]

Then the issue is on the block roots retrieval in db? Should we fix it on that end? Can we reproduce this with a DB test?

So there is a slight race with retrieval of roots and their respective blocks. In times of network turbulence, you could see occasionally the 2 calls retrieving differing results. We shouldn't be using it this way, as there is always a chance of a mismatch. What I could instead do is expose the Blocks API to return the roots also.