Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Implement Historical Backfill using Bitmap Indexer #803

Merged
merged 14 commits into from
Jul 4, 2024

Conversation

darunrs
Copy link
Collaborator

@darunrs darunrs commented Jun 14, 2024

Replaces Delta Lake Stream with Bitmap Indexer Stream as the method for backfilling matching block heights before switching to real time processing. Bitmap Indexer resolves many issues the Delta Lake approach had:

  1. Can be extended more easily.
  2. Can be queried much faster thanks to storage in a database as well as in a compressed format.
  3. Queries can be customized, enabling better matching.
  4. Supports wildcard contract filters much better than the previous approach.
  5. Is much harder to overload with too much data, which was a problem with the previous approach.

@darunrs darunrs force-pushed the replace-delta-lake-with-bitmap branch from f96cb07 to 0dc529f Compare June 18, 2024 14:35
@@ -336,24 +345,40 @@ mod tests {

mock_lake_s3_client
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For some reason I kept getting an error from the mock saying the function has no matching expectation, even though it is never used. I had no choice but to keep it in, which felt odd as a default mock should have a default expect for each function right? Maybe I'm doing something wrong?

block-streamer/src/main.rs Outdated Show resolved Hide resolved
@darunrs darunrs force-pushed the replace-delta-lake-with-bitmap branch from ce4307c to 631c6fc Compare June 22, 2024 00:50
@darunrs darunrs marked this pull request as ready for review June 22, 2024 01:14
@darunrs darunrs requested a review from a team as a code owner June 22, 2024 01:14
@morgsmccauley
Copy link
Collaborator

Updates:

  • Did some minor clean up, mostly clippy lints
  • Use graphql endpoint from env rather than hard code
  • Rename BitmapProcessor to ReceiverBlocksProcessor to align with indexer name, adjusted file system to accommodate
  • Merged main and resolved conflicts

@morgsmccauley morgsmccauley linked an issue Jul 4, 2024 that may be closed by this pull request
@morgsmccauley morgsmccauley merged commit c64f8ff into main Jul 4, 2024
4 checks passed
@morgsmccauley morgsmccauley deleted the replace-delta-lake-with-bitmap branch July 4, 2024 05:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Replace Index File Backfill with Bitmap Indexer Backfill
2 participants