Values in Grafana seem inflated after some time, possibly after Thanos compaction.

[luuktrs]

When requesting metrics in Grafana, data after some days seems quite inflated.

• When the same data is requested another day, the result of the query can be quite inflated. See screenshots where earlier the data seems normal but when requested another day it is suddenly inflated.
• The problematic results always seem to start occurring around 0200 in the morning, but the actual day changes. One day all results younger than 32 hours are correct(consistent with earlier results from the same query), another day all results younger than 6 days are correct
• There aren't any very specific error messages in Thanos' logging
• When the metric in question has few data points, the result graph seems to always be correct, but when there's a lot of data in a metric, the resulting graph shows this problem

You can see on the screenshots provided. Same timeframe, but suddenly the data starts to inflate multiple times over.

The query executed:
sum by (datname,usename) (cnpg_backends_total{namespace="$namespace",release="$release",pod=~"${cluster}-.*",usename!="cnpg_pooler_pgbouncer",datname!="postgres",usename!="postgres"})

It seems to resemble what happens in #922 but this should be fixed years ago.

I dont see any indication in the parameters of the various components. The timeframe doesnt match up to any parameters mentioned.
Store gateway:
--max-time=-4h
--min-time=-1d

Receiver:
--tsdb.retention=8h

Compactor:
--retention.resolution-raw=14d
--retention.resolution-5m=90d
--retention.resolution-1h=2y

Is there anyone who knows what the problem could be?

[dosubot[bot]]

Hey there, @luuktrs! I'm here to help you with bug fixes, questions, and becoming a contributor. Let's tackle this issue together!

I found a similar unsolved discussion titled "[Thanos Query] Query latency rises every 2 hours" which might be related to your issue, although it focuses on query latency rather than data inflation ^[1]. Additionally, there are two open issues that might be relevant:

1. "Problem with receiver and store" ^[2].
2. "Receive: compaction failure causing query returning irrelevant time series" ^[3].

Regarding your issue, it seems like the problem might be related to how Thanos handles downsampling and aggregation. Thanos creates downsampled blocks at different resolution levels, and the query resolution might not align with these downsampling intervals, leading to inflated data in Grafana. The default downsampling intervals are 5 minutes and 1 hour, with downsampling starting at 40 hours and 10 days, respectively ^[4][5].

To troubleshoot, you can try the following:

1. Test the queries directly in the Thanos query UI, selecting different levels of downsampling (raw, 5m, 1h) to see if the issue persists.
2. Disable auto-downsampling to observe if the metrics appear more accurate.
3. Ensure that the query resolution aligns with the downsampling intervals to avoid discrepancies.

These steps might help in identifying if the issue is related to downsampling and aggregation.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[dosubot[bot]]

Yes, that's correct. When using the sum function on data with a 1-hour resolution, it can indeed result in a "sum of sums" scenario. This occurs because the data has already been aggregated into hourly sums during the downsampling process. Applying another sum function on top of these pre-aggregated sums can lead to inflated values, as it effectively adds up these already summed values again. This behavior can be undesirable, especially when accurate totals are needed. To avoid this, consider using raw data or adjusting your query to account for the aggregation level.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[luuktrs]

@dosu Can you reference code where it specifically retrieves the sum incase you execute the sum function in this scenario.

[dosubot[bot]]

The code where the sum function is specifically executed in the scenario of using 1-hour resolution data in Thanos can be found in the TestDownsampleNativeHistograms function. Specifically, the sum aggregation is calculated in the following part of the code:

t.Run("aggr, check sum aggregate", func(t *testing.T) {
    ch, err := ac.Get(AggrSum)
    testutil.Ok(t, err)

    sumIt := ch.Iterator(nil)
    var samples1h []sample
    testutil.Ok(t, expandChunkIterator(sumIt, ch.Encoding(), &samples1h))

    // Since each histogram is the same, sum must also be the same.
    // Not enough data for the last window so adding the last window's data manually.
    // The same number of observations so it's the same as in 5m.
    testutil.Equals(t, true, len(samples1h) > 0)
    for i, s := range samples1h {
        var f = sum1hFh.Copy()
        if i == 10 {
            f.Count = 8
            f.PositiveBuckets = []float64{4, 4}
            f.Sum = 0.002204836
        }

        s.t = 0
        testutil.Equals(t, sample{
            fh: f,
        }, s)
    }
})

This code is part of the test case for downsampling native histograms, where it checks the sum aggregation for 1-hour resolution data ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[luuktrs]

@dosu This seems to be a test. Notice t.Run and t *testing.T. So this can not cause this behavior.

[dosubot[bot]]

The logic for handling the sum function on downsampled data, including the 1-hour resolution, is implemented in the pkg/compact/downsample/downsample.go file. This file contains functions like DownsampleRaw and downsampleAggr, which process raw and aggregated data, respectively. These functions manage the aggregation of data, including the sum operation, by processing samples and creating downsampled chunks ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}