Downsample native histograms (w.i.p.) (thanos-io#109)

* Native histogram downsampling Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed changelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Updated Readme Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Add chnagelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed changelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Started work on histogram downsampling Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Removed compactor test Started work on histogram downsampling Naming Naming Raw downsampling without generics Rebase fix Refactor tests fragments into functions Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> Refactor raw downsampling to support int histograms Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> Comparing histograms for raw downsampling Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> Add histogram support for bucket populateChunk Refactored RawDownsample code Benchmarking recovering series from histogram chunks Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> Refactoring helper function into its own package Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> Trying to implement integration test with querier and downsampled histogram chunks Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> Making the test case a valid one Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> Improvement for native histograms in bucket and series Extending downsampling test Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> Fixing iteration over native histograms aggr chunks Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> Added hist sum to aggrsFromFunc and populateChunk fix Adjusting query; Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> Adding range as test parameter Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> Fixing downsampling tests, general cleanup Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com> Fixed downsampling and improved test Fixed native histogram aggregate downsampling Fixed native histogram aggregate downsampling Improved native histogram downsampling Fixed downsample Improved native histogram downsampling Fixed reset hints for downsampling and added counter reset tests Added nore native histogram downsampling tests Improved native histograms downsampling tests. Fixed tests Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Added float histogram encoding for storepb Fixed linter Fixed lint Fixed lint Revert CHANGELOG.md Added debug log for downsampling Take schema into account when adding histograms Simplified downsampling (thanos-io#117) * Simplified downsampling Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Simplified downsampling Simplified downsampling * Fixed downsampling * Removed debug log * Fixed schema for downsampling native histograms * Fixed aggregate downsampling * Added fix for downsampling stale histograms * Lint fix for histogram downsampling tests * Added more stale marker checks for downsampling * Schema check for downsampled batch Added gauge histogram support for downsampling Adjust histogram sum downsampling for reset and hist AverageChunkIterator support More histogram tests and cleanup Added native histogram support to chunk iterator Fixed overlappingMerger emtpy fn Fixed overlappingMerger emtpy fn Fix for appending during compaction Disabled mimir unit tests Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed imports Reverted querier changes Deleted native histogram query test Deleted native histogram query test Deleted native histogram query test Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixes from review Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Removed logger from downsampleaggr * Fixed populateChunk
fpetkovski · Mar 29, 2023 · 0bee782 · 0bee782
1 parent 376aacf
commit 0bee782
Show file tree

Hide file tree

Showing 12 changed files with 1,155 additions and 299 deletions.
diff --git a/Makefile b/Makefile
@@ -308,7 +308,7 @@ test: export THANOS_TEST_ALERTMANAGER_PATH= $(ALERTMANAGER)
 test: check-git install-tool-deps
 	@echo ">> install thanos GOOPTS=${GOOPTS}"
 	@echo ">> running unit tests (without /test/e2e). Do export THANOS_TEST_OBJSTORE_SKIP=GCS,S3,AZURE,SWIFT,COS,ALIYUNOSS,BOS,OCI if you want to skip e2e tests against all real store buckets. Current value: ${THANOS_TEST_OBJSTORE_SKIP}"
-	@go test $(shell go list ./... | grep -v /vendor/ | grep -v /test/e2e);
+	@go test $(shell go list ./... | grep -v /vendor/ | grep -v /test/e2e | grep -v /internal/mimir-prometheus);
 
 .PHONY: test-local
 test-local: ## Runs test excluding tests for ALL  object storage integrations.

diff --git a/pkg/compact/downsample/aggr_test.go b/pkg/compact/downsample/aggr_test.go
@@ -14,11 +14,11 @@ import (
 func TestAggrChunk(t *testing.T) {
 	var input [5][]sample
 
-	input[AggrCount] = []sample{{100, 30}, {200, 50}, {300, 60}, {400, 67}}
-	input[AggrSum] = []sample{{100, 130}, {200, 1000}, {300, 2000}, {400, 5555}}
-	input[AggrMin] = []sample{{100, 0}, {200, -10}, {300, 1000}, {400, -9.5}}
+	input[AggrCount] = []sample{{t: 100, v: 30}, {t: 200, v: 50}, {t: 300, v: 60}, {t: 400, v: 67}}
+	input[AggrSum] = []sample{{t: 100, v: 130}, {t: 200, v: 1000}, {t: 300, v: 2000}, {t: 400, v: 5555}}
+	input[AggrMin] = []sample{{t: 100}, {t: 200, v: -10}, {t: 300, v: 1000}, {t: 400, v: -9.5}}
 	// Maximum is absent.
-	input[AggrCounter] = []sample{{100, 5}, {200, 10}, {300, 10.1}, {400, 15}, {400, 3}}
+	input[AggrCounter] = []sample{{t: 100, v: 5}, {t: 200, v: 10}, {t: 300, v: 10.1}, {t: 400, v: 15}, {t: 400, v: 3}}
 
 	var chks [5]chunkenc.Chunk
 
@@ -41,7 +41,7 @@ func TestAggrChunk(t *testing.T) {
 	for _, at := range []AggrType{AggrCount, AggrSum, AggrMin, AggrMax, AggrCounter} {
 		if c, err := ac.Get(at); err != ErrAggrNotExist {
 			testutil.Ok(t, err)
-			testutil.Ok(t, expandChunkIterator(c.Iterator(nil), &res[at]))
+			testutil.Ok(t, expandXorChunkIterator(c.Iterator(nil), &res[at]))
 		}
 	}
 	testutil.Equals(t, input, res)