Add support for generating multiple block proofs at once

[BGluth]

It has come up during benchmarking that it would be useful to test what it is like to generate multiple block proofs simultaneously.

The motivation for this comes from the fact that the amount of parallelizable work diminishes as generation progresses for a block proof. The reason for this is due to the trie structure from aggregating txn proofs together into a final block proof, the amount of work units available quickly decreases as we move up the proof aggregation trie. This quickly leads to a situation where CPU utilization drops off after a good portion of the proof tree becomes completed.

Because it's useful to benchmark a more realistic scenario where we have multiple blocks being proved at once, we probably should add a new mode that support generating a range of blocks. This should be pretty straight forward to implement I think.

[vladimir-trifonov]

@BGluth this issue is a duplicate of #10, right?

[BGluth]

@vladimir-trifonov Yeah you're right. Like I guess that PR focuses more on caching block hashes, but this one is more about explicit multi-blocks.

[BGluth]

Ok... So just want to note that while #297 does add support to generate multiple blocks at once, it does not generate multiple blocks in parallel, so this is still something that we should do (as a separate PR probably).

[vladimir-trifonov]

When we prove a range, let's say blocks 10..13= for processing block 10 we use checkpoint-block-number set to 9, but for processing block 11 we use the block proof of block 10 as a parent, i.e. every next proof relies on the previous one, i.e. that's why proving a range happens synchronously. We can skip that, we have in the readme:

- Because incorporating the previous block proof requires a chain of proofs back to the last checkpoint height, you can also disable this requirement by passing `true` for `<IGNORE_PREVIOUS_PROOFS>` (which internally just sets the current checkpoint height to the previous block height).

and then the proving blocks could be made to run in parallel... @Nashtare

[Nashtare]

I'm not sure how much work this would be with the current zero-bin (cc @cpubot) as it is currently built around proving one block at a time, but we need to be able to process blocks concurrently and to connect them if required, by passing the previous block proof when proving the next one. 99.99% of the proving time (all txn + aggregation proofs) remains fully independent of the previous block, so we could still have the leader push a next block to the queue, to be processed ASAP by available workers.
We'd just need the final block_proof producer for block N to:

• wait for proof N-1 to be available if checkpoint-block-number is smaller than N-1
• ignore previous proofs and just process the current block if checkpoint-block-number == N-1.

[vladimir-trifonov]

That sounds fine to me, should we make some plan, it sounds like a pretty big change to me if I'm not mistaken...

[BGluth]

I remember I did take a decent look at what an implementation might look like a while back, and IIRC it looked pretty doable with how Paladin was setup. I would take a look at prove in prover/src/lib.rs. You can probably adjust this so it takes in a block range and creates the same task dependencies for multiple blocks. You might also need to make one more task type to merge two block proofs together. @cpubot might have more to add.

[Nashtare]

Done in 0xPolygonZero/zero-bin#90