Backfill Crate and Ops CLI No {TXN,ACC}_FILL Redis Streams

[kespinola]

Problem

In the current backfiller version, transactions are processed out of order due to out-of-sequence responses and immediate pushing to Redis, resulting in potential out-of-order processing by multiple consumers.

Fix

Implement a fan-out-fan-in strategy. Transaction request threads are scoped to a tree, allowing parallel processing. Within a tree, threads are managed to ensure ordered processing after retrieving all transactions.

Changes

• Resolve backfill order issues.
• Remove plerkle dependency.
• Manage thread pools in ops commands.
• Move backfill logic to crate for continuous ingest process.
• LightDAS integration with RPC-only version of the bubblegum ingester.
• Metadata json worker pool used where needed to fetch JSON along side the backfill.
• OPS CLI account single and program use program_transform instead of _FILL streams.

Conversations

1. Metrics Refactoring: Remove statsd; refactor to expose reports for different backends.
2. Ops CLI Update: Use program_transformers crate directly for backfilling instead of pushing to Redis.

[WilfredAlmeida]

The backfill crate is now integrated into LightDAS.

LightDAS performs all writes to the database via program_transformers and bubblegum-backfill

Thanks for the changes!