Time based downsampling during compaction #2880
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Awesome work @aleks-p! I'm gong to update block querier to use the downsampled data wherever possible
|
One more thing we may want to add later: compaction should be aware of downsampled data and use them if available. This is only useful, if multiple compaction ranges are configured |
aleks-p
changed the title
* feat: add downsampled tables to block querier * add low res profile source * fix lint issues * better heuristic * time range split with resolutions * read downsampled tables concurrently * add SplitTimeRangeByResolution tests * Fix test (avg is not supported currently) * add basic metric to track table access * reuse WithPartitionSamples * fix store-gateway metrics registration * Update pkg/util/time.go * fix store-gateway metrics multi-tenancy * post-review updates * fix metric registration in ingester * Fix failing test * go mod tidy --------- Co-authored-by: Aleksandar Petrov <8142643+aleks-p@users.noreply.github.com>
# Conflicts: # pkg/phlaredb/compact.go
aleks-p
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
An early and unoptimized implementation of time-based downsampling.
Notes:
profiles.parquetin the same compacted blocks (see pkg/phlaredb/downsample/downsample_test.go for an example)source->5m->1h)Opening this PR to get early feedback about the general direction. See #2118 for more information.