cleanup blocks on committer start #260
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WalkthroughA new cleanup mechanism for staging data was introduced. The Changes
Sequence Diagram(s)sequenceDiagram
participant Committer
participant MainStorage
participant StagingStorage
Committer->>MainStorage: Get latest committed block number
alt Blocks committed exist
Committer->>StagingStorage: DeleteOlderThan(chainId, blockNumber)
StagingStorage-->>Committer: Success/Error
end
Committer->>Committer: Enter main loop
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~18 minutes ✨ Finishing Touches
🧪 Generate unit tests
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
jakeloo
approved these changes
Aug 8, 2025
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
internal/orchestrator/committer.go (1)
119-139: Clarify inclusive semantics in comments/logs and prefer structured error logging.
- The info log says “older than or equal to” (inclusive). Ensure both backends implement the same (Postgres should use <=).
- Prefer structured logging with Err(err) for consistency.
Apply:
- latestCommittedBlockNumber, err := c.storage.MainStorage.GetMaxBlockNumber(c.rpc.GetChainID()) + chainID := c.rpc.GetChainID() + latestCommittedBlockNumber, err := c.storage.MainStorage.GetMaxBlockNumber(chainID) if err != nil { - log.Error().Msgf("Error getting latest committed block number: %v", err) + log.Error().Err(err).Msg("Error getting latest committed block number") return } if latestCommittedBlockNumber.Sign() == 0 { log.Debug().Msg("No blocks committed yet, skipping staging data cleanup") return } - // Delete all staging data older than the latest committed block number - if err := c.storage.StagingStorage.DeleteOlderThan(c.rpc.GetChainID(), latestCommittedBlockNumber); err != nil { - log.Error().Msgf("Error deleting staging data older than %v: %v", latestCommittedBlockNumber, err) + // Delete all staging data older than or equal to the latest committed block number + if err := c.storage.StagingStorage.DeleteOlderThan(chainID, latestCommittedBlockNumber); err != nil { + log.Error().Err(err).Msgf("Error deleting staging data older than or equal to %v", latestCommittedBlockNumber) return } - log.Info().Msgf("Deleted staging data older than or equal to %v", latestCommittedBlockNumber) + log.Info().Msgf("Deleted staging data older than or equal to %v", latestCommittedBlockNumber)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
internal/orchestrator/committer.go(2 hunks)internal/storage/clickhouse.go(1 hunks)internal/storage/connector.go(1 hunks)internal/storage/postgres.go(1 hunks)test/mocks/MockIStagingStorage.go(1 hunks)
🔇 Additional comments (2)
internal/orchestrator/committer.go (1)
82-85: Good placement: cleanup before main loop.Running cleanup once on start is a safe, low-overhead way to clear stale staging data.
test/mocks/MockIStagingStorage.go (1)
239-284: Mocks look correct and consistent.Signature matches the interface and helpers provide typed Run/Return ergonomics.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
internal/orchestrator/committer_test.go (2)
407-409: Strengthen expectation: assert the call and match big.Int by valueGood to stub the cleanup call. To actually verify it happens and avoid pointer-equality pitfalls on *big.Int, match by value and assert the invocation.
Apply within the changed lines:
- // Add expectation for DeleteOlderThan call during cleanup - mockStagingStorage.On("DeleteOlderThan", chainID, big.NewInt(100)).Return(nil) + // Expect cleanup to delete anything older than the latest committed block (by value) + mockStagingStorage. + On("DeleteOlderThan", chainID, mock.MatchedBy(func(b *big.Int) bool { return b.Cmp(big.NewInt(100)) == 0 })). + Return(nil). + Once()Additionally, assert after the short wait to ensure it was called:
// After time.Sleep(200 * time.Millisecond) mockStagingStorage.AssertCalled(t, "DeleteOlderThan", chainID, mock.MatchedBy(func(b *big.Int) bool { return b.Cmp(big.NewInt(100)) == 0 }))
444-446: Mirror the same robust expectation and assertion hereMatch big.Int by value and verify the call occurs once when the committer starts, then stops on cancel.
Within the changed lines:
- // Add expectation for DeleteOlderThan call during cleanup - mockStagingStorage.On("DeleteOlderThan", chainID, big.NewInt(100)).Return(nil) + // Expect cleanup to delete anything older than the latest committed block (by value) + mockStagingStorage. + On("DeleteOlderThan", chainID, mock.MatchedBy(func(b *big.Int) bool { return b.Cmp(big.NewInt(100)) == 0 })). + Return(nil). + Once()And after the goroutine finishes (after
<-done):mockStagingStorage.AssertCalled(t, "DeleteOlderThan", chainID, mock.MatchedBy(func(b *big.Int) bool { return b.Cmp(big.NewInt(100)) == 0 }))
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
internal/orchestrator/committer_test.go(2 hunks)internal/storage/clickhouse.go(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- internal/storage/clickhouse.go
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
New Features
Bug Fixes
Tests