lib/storage: adds nextIndexDB to mitigate performance degradation on #4626
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
indexDB rotation Previously, during indexDB dateMetricID cache was reseted and it caused a lot of new records creation. It may saturate memory usage, since lookups for exist entries were made. With new logic, daily index records will be pre-created at the 1 hour before indexDB rotation. There is no need to reset dateMetricID cache, since it belongs to indexDB. It greatly improves perforamnce. It should help to implement next feature #4563
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## master #4626 +/- ##
==========================================
+ Coverage 61.96% 62.03% +0.07%
==========================================
Files 373 373
Lines 69230 69293 +63
==========================================
+ Hits 42899 42988 +89
+ Misses 24042 24015 -27
- Partials 2289 2290 +1
☔ View full report in Codecov by Sentry. |
hagen1778
reviewed
Jul 14, 2023
| @@ -65,6 +65,9 @@ type indexDB struct { | |||
| // The counter for time series which were re-populated from previous indexDB after the rotation. | |||
| timeseriesRepopulated uint64 | |||
|
|
|||
| // The counter for daily index records which were re-populated from current indexDB to the new indexDB before the rotation. | |||
| dailyIndexRecordsRepopulated uint64 | |||
There was a problem hiding this comment.
it is unclear when this counter increases:
- When current indexDB is used to re-populate entires for new indexDB object
- Or when current indexDB gets new entries from previous IndexDB object.
|
Will create separate PR |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
indexDB rotation
Previously, during indexDB dateMetricID cache was reseted and it caused a lot of new records creation. It may saturate memory usage, since lookups for exist entries were made.
With new logic, daily index records will be pre-created at the 1 hour before indexDB rotation.
There is no need to reset dateMetricID cache, since it belongs to indexDB. It greatly improves perforamnce.
It should help to implement next feature #4563