Index blocks by number in the DB

[arkpar]

With #760 The blocks are indexed by hash in the database which has a negative impact on performance. Keeping blocks close to each other in the DB allows for better utilisation of the database cache and disk IO when querying ranges of blocks. We need to bring back the block-by-number DB scheme. The problem is that two blocks that are not yet finalized can share the same number. I can think of two ways to get around that:

1. Store current unfinalized best blocks by number. Any extra blocks are stored with additional suffix or by hash. In case of chain reorg the blocks are re-inserted with new keys. This implementation looks more preferable unless chain reorgs are too frequent.

2. Keep the blocks in the journal similar to state changes in state-db. On finalization the block is moved from the journal to the main storage with a proper key.

[snd]

The problem is that two blocks that are not yet finalized can share the same number.

this suggests that we don't store any additional blocks with the same number as a finalized one.
in other words if there's a finalized block with number n, then we store no additional blocks with number n.

should we remove all other blocks with the same number as the finalized one on finalization?
or is there a reason to keep the extra blocks around?

[snd]

1. seems to have pretty good properties overall. a downside is that reorgs are expensive.

[arkpar]

this suggests that we don't store any additional blocks with the same number as a finalized one.

After finalization the non-canonical blocks can only be accessed by hash. They can be stored with hash-keys. I'd also agree that they should be pruned away by default on finalization unless the node is in archive mode.

[snd]

we could store all blocks by number + hash. blocks would still be close due to the number prefix. it would require a bit more storage space for the keys of the canonical chain. it wouldn't require removal and reinsert on reorgs. nevermind. that would require an additional way to distinguish canonical blocks from other blocks.