Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

compact: data corruption during downsapmle, test and fix. #6598

Merged
merged 10 commits into from
Sep 12, 2023
Merged

compact: data corruption during downsapmle, test and fix. #6598

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
7d40d81
Samples to reproduce data corruption during downsapmle, tests and fix.
xBazilio Aug 9, 2023
26cee2c
Samples to reproduce data corruption during downsapmle, tests and fix.
xBazilio Aug 9, 2023
053e5e5
added test for chunk with NaN values only
xBazilio Aug 11, 2023
9666d26
Merge branch 'main' into fix_downsample_data_corruption
xBazilio Aug 11, 2023
33e413f
CHANGELOG.md
xBazilio Aug 11, 2023
c8b6eac
Merge branch 'main' into fix_downsample_data_corruption
xBazilio Aug 18, 2023
088628b
Merge branch 'main' into fix_downsample_data_corruption
xBazilio Aug 21, 2023
58aa26e
added check for math.NaN
xBazilio Aug 21, 2023
e61bac0
optimized NaN checking
xBazilio Aug 24, 2023
6c70731
Merge branch 'main' into fix_downsample_data_corruption
xBazilio Aug 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
11 changes: 10 additions & 1 deletion pkg/compact/downsample/downsample.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -372,8 +372,17 @@ func downsampleRawLoop(data []sample, resolution int64, numChunks int) []chunks.
for ; j < len(data) && data[j].t <= curW; j++ {
}

batch := data[:j]
batch := make([]sample, 0, j)
for _, s := range data[:j] {
if value.IsStaleNaN(s.v) || math.Float64bits(s.v) == value.NormalNaN {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am wondering if we should filter those NaNs beforehand... So that we can calculate batchSize, windowTime, etc better.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/thanos-io/thanos/blob/main/pkg/compact/downsample/downsample.go#L648 the ApplyCounterResetsSeriesIterator ignores NaN values but https://github.com/thanos-io/thanos/blob/main/pkg/compact/downsample/downsample.go#L474 expandChunkIterator only ignores stale NaN.

I am wondering if we should just change this. We have so many places in the code that allows normal NaN and ignores just stale NaN. Not sure what's the correct behavior here. Should we keep the normal NaN? @bwplotka

continue
}
batch = append(batch, s)
}
data = data[j:]
if len(batch) == 0 {
continue
}

ab := newAggrChunkBuilder()

Expand Down