improve start block determination for poller

[iuwqyir]

TL;DR

Improved block selection logic in the poller to use the highest available block number from either staging or main storage.

What changed?

• Enhanced the NewPoller function to check both staging and main storage for the highest block number
• Added logic to compare block numbers from staging storage, main storage, and the configured start block
• Always selects the highest block number as the starting point for polling
• Added comprehensive unit tests for the poller initialization logic covering various scenarios:
  • Force from block enabled
  • Staging block higher than configured start
  • Main storage block higher than configured start
  • Main storage block higher than staging block
  • Configured start block highest
  • Error handling for storage failures
  • Edge cases with zero or negative block numbers
  • Default config values
  • Boundless poller initialization

How to test?

1. Run the new unit tests: go test ./internal/orchestrator -v
2. Test the indexer with different configurations:
   • With a higher block in staging storage
   • With a higher block in main storage
   • With force from block enabled
   • With storage errors

Why make this change?

This change prevents the indexer from reprocessing blocks that have already been indexed in either staging or main storage. By selecting the highest available block number, we avoid duplicate processing and improve efficiency, especially after restarts or when switching between different storage systems.

Summary by CodeRabbit

• Bug Fixes

  • Improved the logic for determining the starting block during poller initialization, ensuring the correct block is selected from available storage sources.

• Tests

  • Added comprehensive unit tests to verify poller initialization under various configuration and storage scenarios, including error handling and edge cases.

[coderabbitai]

Walkthrough

The changes update the initialization logic for the poller's starting block in the orchestrator, adding conditional checks and error handling when determining the last polled block from staging and main storage. Additionally, comprehensive unit tests are introduced to verify poller initialization under various scenarios, including configuration options and storage states.

Changes

Cohort / File(s)	Change Summary
Poller Initialization Logic internal/orchestrator/poller.go	Refined the logic for setting lastPolledBlock in the poller constructor: now only updates from staging storage if its block number exceeds the configured start block, and further checks main storage for a higher block number. Added error handling for main storage retrieval and updated debug logs to reflect these changes.
Poller Initialization Unit Tests internal/orchestrator/poller_test.go	Added a new test file with unit tests for poller creation logic. Tests cover scenarios involving configuration flags, staged and main storage values, error handling, and edge cases. Also tests the initialization of boundless pollers and the application of configuration and options. Ensures poller state is correctly set under various conditions.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Note

⚡️ Unit Test Generation is now available in beta!

Learn more here, or try it out under "Finishing Touches" below.

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 451af23 and a98e53c.

📒 Files selected for processing (2)

• internal/orchestrator/poller.go (1 hunks)
• internal/orchestrator/poller_test.go (1 hunks)

🧰 Additional context used

🧬 Code Graph Analysis (1)

internal/orchestrator/poller_test.go (8)

configs/config.go (3)

• Cfg (193-193)
• PollerConfig (18-26)
• Config (179-191)

internal/orchestrator/poller.go (6)

• Poller (22-35)
• NewPoller (76-108)
• NewBoundlessPoller (50-72)
• DEFAULT_BLOCKS_PER_POLL (19-19)
• DEFAULT_TRIGGER_INTERVAL (20-20)
• WithPollerWorkModeChan (44-48)

test/mocks/MockIRPCClient.go (1)

• MockIRPCClient (17-19)

test/mocks/MockIStagingStorage.go (1)

• MockIStagingStorage (17-19)

test/mocks/MockIMainStorage.go (1)

• MockIMainStorage (17-19)

test/mocks/MockIOrchestratorStorage.go (1)

• MockIOrchestratorStorage (17-19)

internal/storage/connector.go (1)

• IStorage (67-71)

internal/orchestrator/work_mode_monitor.go (1)

• WorkMode (16-16)

🔇 Additional comments (16)

internal/orchestrator/poller.go (2)

89-93: LGTM! Improved staging block selection logic.

The conditional check ensures that the staging block is only used when it's actually higher than the configured start block, preventing regression to earlier blocks. The logging provides good visibility into the decision-making process.

---

94-102: LGTM! Main storage integration improves block determination.

The addition of main storage block retrieval provides a more comprehensive approach to determining the starting block. The error handling is appropriate - logging the error but continuing with initialization rather than failing completely. The conditional logic correctly ensures main storage block is only used when higher than the configured start.

internal/orchestrator/poller_test.go (14)

13-27: LGTM! Good test setup utility.

The setupTestConfig function properly initializes the global config with zero values when not already set, ensuring consistent test environment. This prevents test failures due to uninitialized configuration.

---

29-64: LGTM! Comprehensive test for ForceFromBlock scenario.

The test correctly verifies that when ForceFromBlock is enabled, the poller uses the configured start block minus one without querying storage systems. The test properly sets up mocks, saves/restores config, and validates all expected values.

---

66-112: LGTM! Well-structured test for staging block priority.

This test effectively validates that when staging storage returns a block higher than the configured start block, it takes precedence over the main storage block. The mock setup is correct and the assertions verify the expected behavior.

---

114-160: LGTM! Good coverage for main storage priority scenario.

The test correctly validates that main storage block is used when staging returns nil and main storage has a block higher than the configured start. Proper mock expectations and assertions are in place.

---

162-208: LGTM! Excellent test for storage comparison logic.

This test validates the key improvement in the PR - that main storage block takes precedence when it's higher than staging block, even if both are above the configured start. This ensures the poller starts from the highest available block.

---

210-258: LGTM! Good fallback scenario coverage.

The test correctly verifies that when both staging and main storage blocks are lower than the configured start block, the poller falls back to using the configured start block minus one.

---

260-306: LGTM! Proper error handling test for staging storage.

The test validates that staging storage errors are handled gracefully, with the poller continuing to check main storage and using the higher value when available.

---

308-355: LGTM! Good error handling test for main storage.

This test ensures that main storage errors don't break the initialization process, with appropriate fallback to the configured start block when staging block is not suitable.

---

357-403: LGTM! Edge case coverage for zero staging block.

The test properly handles the edge case where staging storage returns a zero block, ensuring main storage block is used instead. This validates the Sign() <= 0 check in the main code.

---

405-451: LGTM! Negative block edge case handled correctly.

This test covers another important edge case where staging returns a negative block number, ensuring the system falls back to main storage appropriately.

---

453-498: LGTM! Default configuration test provides good coverage.

The test validates behavior with zero configuration values, ensuring the system handles default/unset configurations properly. The expected result of -1 for lastPolledBlock when fromBlock is 0 is mathematically correct.

---

500-535: LGTM! Good test coverage for NewBoundlessPoller.

This test validates that NewBoundlessPoller correctly initializes with configuration values and properly sets up the poller structure.

---

537-572: LGTM! Default values test for boundless poller.

The test correctly verifies that default constants are used when configuration values are zero, ensuring robust fallback behavior.

---

574-613: LGTM! Options application test completes the coverage.

This test validates that poller options (like work mode channel) are properly applied during initialization, ensuring the option pattern works correctly.

✨ Finishing Touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch 07-30-improve_start_block_determination_for_poller

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai generate unit tests to generate unit tests for this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[iuwqyir]

• validate group and sort by #254
• improve start block determination for poller #253 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.