-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Implement Historical Backfill using Bitmap Indexer #803
Conversation
f96cb07
to
0dc529f
Compare
@@ -336,24 +345,40 @@ mod tests { | |||
|
|||
mock_lake_s3_client |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For some reason I kept getting an error from the mock saying the function has no matching expectation, even though it is never used. I had no choice but to keep it in, which felt odd as a default mock should have a default expect for each function right? Maybe I'm doing something wrong?
ce4307c
to
631c6fc
Compare
Updates:
|
Replaces Delta Lake Stream with Bitmap Indexer Stream as the method for backfilling matching block heights before switching to real time processing. Bitmap Indexer resolves many issues the Delta Lake approach had: