Respect Sequential Updates of Merkle Trees #193
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue
In the current version of the backfiller transactions can be processed out of order from transaction responses coming out of order. The transactions are immediately pushed to redis for processing which may also result of out of order with multiple consumers processing the redis stream.
Fix
Switch to a fan-out-fan in strategy where transaction request threads are scoped to the tree. In this updated strategy trees can be processed in parallel. Within a tree gaps and transactions threads are managed by the tree thread so max workers are now scoped. Once all transactions for the tree are retrieved sort and process in order.
Changes