Merged
Conversation
Collaborator
dwerner
commented
Oct 2, 2025
dwerner
commented
- Add historical log streaming support to erigon-bridge
- Implements historical log data streaming using custom Erigon's Logs RPC.
- Completes the bridge's ability to serve all three data types (blocks, transactions, logs)
Implements historical log data streaming using Erigon's Logs RPC. Completes the bridge's ability to serve all three data types (blocks, transactions, logs) in both historical and live modes.
Adds DataScanner module that: - Detects existing parquet files and finds gaps in block coverage - Identifies historical/live sync boundaries to prevent overlap - Enables resumable sync by cleaning failed historical temp files - Preserves active live sync temp files using modification time checks - Validates filename ranges to reject corrupted data Sync improvements: - Workers continue on failure, tracking failed segments for retry - 10-minute timeout per segment prevents hung workers - Temp filenames show actual block ranges being written - Three-phase sync per segment: blocks → transactions → logs Includes 8 comprehensive tests covering edge cases: - Live vs historical temp file detection - Gap finding and boundary calculation - Temp file cleanup with race condition protection - Complete vs incomplete segment detection
Progress now correctly represents work across blocks, transactions, and logs (3 phases per segment). Updated CLI display to clarify that progress includes all data types.
- Remove deprecated BIT_PACKED parquet encoding - Change ToString to Display implementation for StreamType - Remove unused struct fields (endpoint, query, batch_size) - Replace .get(0) with .first() for clarity - Prefix intentionally unused parameters with underscore - Fix useless format!() calls
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.