Merged
Conversation
…eaming files Added is_live flag to PhaserMetadata (version 2) to distinguish between: - Historical sync workers: Fetch specific block ranges (is_live = false) - Live streaming workers: Subscribe to current head (is_live = true) Changes: - Bump PhaserMetadata version from 1 to 2 - Add is_live boolean field with #[serde(default)] for backward compatibility - Version 1 files remain readable (is_live defaults to false)
…andling ParquetWriter now tracks two distinct ranges: - Segment range: Logical 500K segment boundaries (used for filename) - Responsibility range: Actual range this worker is responsible for (used for metadata) This allows files to be grouped by segment in filenames while metadata accurately reflects the capped responsibility range when workers are stopped at the live boundary. Changes: - Split requested_range into segment_range and responsibility_range - Added set_ranges() method to set both independently - Added validation to ensure end >= start for both ranges - Filenames use segment range, metadata uses responsibility range - Logs show both ranges for debugging
…nges Workers now cap their sync range at the historical/live boundary and correctly report their responsibility range in metadata. Example: Worker syncing segment 23500000-23999999 capped at boundary 23550000: - Filename: blocks_from_23500000_to_23999999_0.parquet (segment grouping) - Metadata segment: 23500000-23999999 (logical segment) - Metadata responsibility: 23500000-23549999 (actual coverage) Changes: - SyncWorker caps to_block at boundary-1 when boundary is present - Workers calculate segment boundaries independently of capping - Workers pass both segment range and responsibility range to ParquetWriter - SyncService passes historical_boundary to workers - DataScanner handles both historical and live_ prefixed files - Removed unused _historical_boundary storage field
Updated parquet-files.md to explain: - live_ filename prefix indicates file written by live streaming worker - is_live metadata flag tracks worker type (historical vs live) - PhaserMetadata version 2 adds is_live field
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.