feat: Support Star Contract Filter in Block Streamer #572
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I did a simple manual test: I called start_stream with a modified filter of * and observed that it began pulling block heights immediately. I also verified manually that the heights were in the expected sequential order, including skipping skipped heights. |
morgsmccauley
approved these changes
Feb 20, 2024
|
As per discussion in team meeting, I'll expand the list of things to skip delta to *, *.near, *.kaiching, *.tg to start. |
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Block Streamer will fail to process a * contract filter as it will try to fetch index files for all account IDs. This is not only extremely inefficient since we aren't planning to skip anything, but will also fail as we limit the number of accounts fetched as a sanity check.
Block Streamer, if the filter is "*" specifically, now skips fetching index files and begins reading from Lake directly instead.
This retains the behavior of block streamer reading block data before pushing the height to Redis. There is an intended migration of Runner's prefetch into Block Streamer. This change works well towards that goal. However, obviously there is double work performed here: We fetch the block, only to just put the height into Redis. If, in the short term and before we migrate prefetch from Runner to Block Streamer, we need to speed things up, we can list files form S3 instead. However, this work would be removed after the mentioned migration of work.