Limit number of blocks per level (2nd attempt)

[davxy]

Closes #432

---

This PR prevents the StateDbError::TooManySiblingBlocks error from being triggered by eagerly removing leaves from the backend on block import and before the error condition is met.

---

⚠ SUPERSEEDS: #1544

The main difference is this is a portable solution WRT the leaves order read from the backend.

[davxy]

(Updated)

Following #1559 (comment), let's recap the problems to be solved:

• issue 1. We need to fetch the list of all the blocks at the same level of the imported block.
• issue 2. if this list length is >= the threshold then we have to choose which block to remove.
• issue 3. The block to remove may not be a leaf (e.g. because what has been said in the linked comment). In this case we also have to remove all its descendants.

I don't see a direct support from the Backend to solve the issues, let's see if we can somehow work around the limitations.

• issue 1. This is probably the most annoying. Is there already an easy and possibly low cost way to get all blocks at one level?
• issue 2. The choice will be again driven by the "age". Probably the best thing to do is: for each block at the same height of the imported one its "age" is set as the age of its most recent descendant leaf (for example if a block is the parent a very old leaf and a very fresh one, then we consider the block age equal to the most fresh leaf).
• issue 3. We can remove all the block's descendants by starting from the leaves and using the Backend::remove_leaf_block.

A block descendants can be recovered using the leaves can be found using the sp_blockchain::tree_route (is expensive... but also a very sporadic operation).

So the main question here is: how to get the blocks list at one level?

I'm open to more simple suggestions :-)

[davxy]

@davxy what is the status with merging this earlier?

Confident to merge this very soon. I just need to add a small safety net

[sandreim]

bot rebase

[paritytech-processbot]

Error: Command 'Command { std: "git" "merge" "origin/master" "--no-ff" "--no-edit", kill_on_drop: false }' failed with status Some(1); output: no output

[bkchr]

@davxy can we merge this?

[davxy]

bot merge

[paritytech-processbot]

Error: Github API says #1559 is not mergeable