compact: data corruption during downsapmle, test and fix. (thanos-io#…

…6598) * Samples to reproduce data corruption during downsapmle, tests and fix. Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * Samples to reproduce data corruption during downsapmle, tests and fix. Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * added test for chunk with NaN values only Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * CHANGELOG.md Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * added check for math.NaN Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * optimized NaN checking Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> --------- Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com>
coleenquadros · Sep 18, 2023 · a341596 · a341596
1 parent eecb7da
commit a341596
Show file tree

Hide file tree

Showing 3 changed files with 710 additions and 1 deletion.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -139,6 +139,7 @@ We use *breaking :warning:* to mark changes that are not backward compatible (re
 - [#6592](https://github.com/thanos-io/thanos/pull/6592) Query Frontend: fix bugs in vertical sharding `without` and `union` function to allow more queries to be shardable.
 - [#6317](https://github.com/thanos-io/thanos/pull/6317) *: Fix internal label deduplication bug, by resorting store response set.
 - [#6189](https://github.com/thanos-io/thanos/pull/6189) Rule: Fix panic when calling API `/api/v1/rules?type=alert`.
+- [#6598](https://github.com/thanos-io/thanos/pull/6598) compact: fix data corruption with "invalid size" error during downsample
 
 ### Changed
 - [#6049](https://github.com/thanos-io/thanos/pull/6049) Compact: *breaking :warning:* Replace group with resolution in compact metrics to avoid cardinality explosion on compact metrics for large numbers of groups.

diff --git a/pkg/compact/downsample/downsample.go b/pkg/compact/downsample/downsample.go
@@ -372,8 +372,17 @@ func downsampleRawLoop(data []sample, resolution int64, numChunks int) []chunks.
 		for ; j < len(data) && data[j].t <= curW; j++ {
 		}
 
-		batch := data[:j]
+		batch := make([]sample, 0, j)
+		for _, s := range data[:j] {
+			if math.IsNaN(s.v) {
+				continue
+			}
+			batch = append(batch, s)
+		}
 		data = data[j:]
+		if len(batch) == 0 {
+			continue
+		}
 
 		ab := newAggrChunkBuilder()