feat(sequencer): dynamic batch production

[Mirko-von-Leipzig]

Summary

This PR changes batch production from a fixed interval, to producing as many full batches as possible. The current batch.interval: Duration CLI is instead now used to configure the time after which a non-full batch will be scheduled if no full batch can be produced.

The new algorithm is essentially:

• Whenever a batch completes, attempt to build a full batch if possible.
• If no batch has been produced for batch.interval seconds, attempt to build a batch of any size.
• If idle, check for full batches every block.interval / 10 seconds.

In addition, I've removed the failure injection we had in place for batch and block production. These have actually never been used, so they're not really worth the LoC.

Closes #2287

Changelog

[[entry]]
scope       = "node"
impact      = "changed"
description = "Batch production is now dynamic, creating full batches as often as possible, while maintaining a minimum batch rate"

[Mirko-von-Leipzig]

Note that the batch selection process is now quite unoptimized. This will be improved as a separate PR.

[kkovaacs]

This is still slightly suboptimal.

If the budget is not exhausted but all remaining transactions would make the selection go over the budget budget.is_exhausted() will be false and we're returning BatchSelection::Partial here. That in turn makes the scheduler wait for the partial-batch timeout.

If we were to return BatchSelection::Full instead then we could immediately create this batch and then continue with the rest of the candidates.

[Mirko-von-Leipzig]

I did debate back and forth on this. But perhaps I settled on the wrong conclusion.

At the moment this only really matters if we have transactions with massive amounts of notes (since the tx count restriction is more binary).

As an example, if we have two extreme txs each with close to the maximum number of output notes, then each batch can only contain one of these. But you could in theory squeeze additional smaller txs in if they arrive.

So I guess the question is, do we want to wait, or do we want to process these asap even if better opportunities might arrive.

[kkovaacs]

It really just depends on whether we're optimizing for latency or for throughput.

I think that in most cases it's latency that matters (better UX, as transactions get included faster). But maybe even that is a premature optimization at this point -- I'm fairly sure the block builder will need quite a few tweaks once these decisions start to matter...

[kkovaacs]

LGTM!