Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backfill Crate and Ops CLI No {TXN,ACC}_FILL Redis Streams #197

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

kespinola
Copy link
Collaborator

@kespinola kespinola commented Jun 11, 2024

Problem

In the current backfiller version, transactions are processed out of order due to out-of-sequence responses and immediate pushing to Redis, resulting in potential out-of-order processing by multiple consumers.

Fix

Implement a fan-out-fan-in strategy. Transaction request threads are scoped to a tree, allowing parallel processing. Within a tree, threads are managed to ensure ordered processing after retrieving all transactions.

Changes

  • Resolve backfill order issues.
  • Remove plerkle dependency.
  • Manage thread pools in ops commands.
  • Move backfill logic to crate for continuous ingest process.
  • LightDAS integration with RPC-only version of the bubblegum ingester.
  • Metadata json worker pool used where needed to fetch JSON along side the backfill.
  • OPS CLI account single and program use program_transform instead of _FILL streams.

Conversations

  1. Metrics Refactoring: Remove statsd; refactor to expose reports for different backends.
  2. Ops CLI Update: Use program_transformers crate directly for backfilling instead of pushing to Redis.

@kespinola kespinola changed the title Bubblegum Backfill Crate Bubblegum Backfill Crate and Ops CLI No {TXN,ACC}_FILL Redis Streams Jun 11, 2024
@WilfredAlmeida
Copy link

WilfredAlmeida commented Jul 1, 2024

The backfill crate is now integrated into LightDAS.

LightDAS performs all writes to the database via program_transformers and bubblegum-backfill

Thanks for the changes!

…e is on out of order processing. process the transaction by ops instead of queueing to TXN_FILL redis stream.
@kespinola kespinola force-pushed the espi/backfill-accounts-program-transformer branch from 7528633 to b0fefbb Compare July 18, 2024 09:24
@kespinola kespinola marked this pull request as ready for review July 18, 2024 09:24
@kespinola kespinola changed the title Bubblegum Backfill Crate and Ops CLI No {TXN,ACC}_FILL Redis Streams Backfill Crate and Ops CLI No {TXN,ACC}_FILL Redis Streams Jul 18, 2024
…cts eg LightDAS and be included in the grpc-ingest to execute backfills on restart then pick up tasks from the stream to insure no out of order indexing
@kespinola kespinola force-pushed the espi/backfill-accounts-program-transformer branch from b0fefbb to 138bf2e Compare July 18, 2024 09:43
@kespinola kespinola requested a review from danenbm July 18, 2024 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants