-
Notifications
You must be signed in to change notification settings - Fork 3
feat: bulk ingestor WoC #685
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…ctor synchronization logic
Manual Testsℹ️ Remember to ask team members to perform manual tests and to assign |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR introduces a new bulk ingestor for WhatsOnChain (WoC) and refactors the existing WoC polling ingestor to use a shared HTTP client abstraction. The changes enable fetching bulk block header files from WhatsOnChain's bulk endpoints while maintaining backward compatibility with the polling-based approach.
Key changes include:
- Addition of a new
BulkIngestorWOCimplementation that fetches bulk header files from WhatsOnChain - Refactoring of WoC-related HTTP calls into a shared
wocClientfor better code reuse - Migration from config-based initialization to an options builder pattern for both poll and bulk ingestors
- Interface signature update to
BulkIngestor.Synchronize()to accept a singleHeightRangeinstead ofHeightRanges
Reviewed Changes
Copilot reviewed 17 out of 19 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
pkg/services/chaintracks/models/height_range.go |
Adds Overlaps method to check range overlap, supporting bulk file filtering |
pkg/services/chaintracks/ingest/woc_dto.go |
Adds prevChainWorkForGenesis constant and WOCHeadersResourcesDTO for bulk endpoint responses |
pkg/services/chaintracks/ingest/live_ingestor_woc_poll_test.go |
Updates tests to use new options builder pattern instead of config structs |
pkg/services/chaintracks/ingest/live_ingestor_woc_poll_options.go |
Introduces options builder pattern with IngestorWocPollOpts for flexible configuration |
pkg/services/chaintracks/ingest/live_ingestor_woc_poll.go |
Refactors to use shared wocClient instead of direct HTTP calls |
pkg/services/chaintracks/ingest/chaintraks_woc_client.go |
New shared HTTP client for WhatsOnChain API interactions across live and bulk ingestors |
pkg/services/chaintracks/ingest/cdn_reader_test.go |
Comprehensive tests for CDN reader functionality with error scenarios |
pkg/services/chaintracks/ingest/cdn_reader.go |
Implements CDN reader for fetching bulk header files from Project Babbage CDN |
pkg/services/chaintracks/ingest/cdn_data.go |
Adds ToHeightRange method to convert bulk file metadata to height ranges |
pkg/services/chaintracks/ingest/bulk_ingestor_woc_test.go |
Adds placeholder test (currently skipped) for WoC bulk ingestor |
pkg/services/chaintracks/ingest/bulk_ingestor_woc_options.go |
Options builder pattern for configuring bulk WoC ingestor |
pkg/services/chaintracks/ingest/bulk_ingestor_woc.go |
Core implementation of bulk ingestor fetching header files from WhatsOnChain |
pkg/services/chaintracks/ingest/bulk_ingestor_cdn.go |
Updates signature to accept single HeightRange parameter |
pkg/services/chaintracks/chaintracks_storage.interface.go |
Updates BulkIngestor interface to use HeightRange instead of HeightRanges |
pkg/services/chaintracks/chaintracks_service_test.go |
Updates service test to use new options pattern |
pkg/services/chaintracks/chaintracks_initializers.go |
Updates factory to pass API key using options builder |
pkg/services/chaintracks/bulk_manager.go |
Implements logic to track missing ranges and iteratively fetch from multiple ingestors |
pkg/defs/woc_poll_ingestor.go |
Removes deprecated WOCPollIngestorConfig struct (replaced by options pattern) |
pkg/defs/chaintracks.go |
Adds WocAPIKey field to main chaintracks config |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|



Description of Changes
Provide a brief description of the changes you've made.
Linked Issues / Tickets
Reference any related issues or tickets, e.g. "Closes #123".
Testing Procedure
Describe the tests you've added or any testing steps you've taken.
Checklist: