Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Respect Sequential Updates of Merkle Trees #193

Conversation

kespinola
Copy link
Collaborator

@kespinola kespinola commented May 15, 2024

Issue

In the current version of the backfiller transactions can be processed out of order from transaction responses coming out of order. The transactions are immediately pushed to redis for processing which may also result of out of order with multiple consumers processing the redis stream.

Fix

Switch to a fan-out-fan in strategy where transaction request threads are scoped to the tree. In this updated strategy trees can be processed in parallel. Within a tree gaps and transactions threads are managed by the tree thread so max workers are now scoped. Once all transactions for the tree are retrieved sort and process in order.

Changes

  • Resolve out of order backfill issues.
  • Drop plerkle as a dependency for das-ops.
  • Ops command now manage thread pools for running program transform handlers instead of pushing to a _FILL redis stream.

…e is on out of order processing. process the transaction by ops instead of queueing to TXN_FILL redis stream.
@kespinola kespinola force-pushed the espi/backill-tree-transactions-sequentially branch from 5721335 to f0f75b7 Compare May 15, 2024 13:04
@kespinola
Copy link
Collaborator Author

work moved here.

#197

@kespinola kespinola closed this Jun 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant