Aggregated buckets do not match eachother.

[dforste]

I am trying to verify my data is correct so I have some aggregations setup like this:

    ^app\.([^.]+)\.dc\.([^.]+)\.env\.([^.]+)\.host\.([^.]+)\.instance\.([^.]+)\.sub1\.([^.]+)\.sub2\.([^.]+)\.sub3\.([^.]+)\..metric\.count\.([^.]+)
        every 10 seconds
        expire after 20 seconds
        compute sum write to
          aggregated_app.\1.dc.\2.env.\3.host.\4.instance.\5.sub1.\6.sub2.\7.sub3.\8.metric.count.\9
        compute sum write to
          aggregated_app.\1.dc.\2.env.\3.sub1.\6.metric.count.\9
        compute sum write to
          aggregated_app.\1.dc.\2.env.\3.sub2.\7.metric.count.\9
        compute sum write to
          aggregated_app.\1.dc.\2.env.\3.sub3.\8.metric.count.\9
        ;

When I try to validate the aggregations they give me graphs like this:

I am already using consolidateBy(sum) and scaleToSeconds(60) to match the data store but this data is still not equal like I would expect from the aggregation. Storage is setup with retention of 10s:1m, 1m:2y

Any ideas why this data may not be correct?

One last thought is that each datapoint comes in with a value of 1 so each aggregation combines that number of points.

[grobian]

has this happened always, or is this since recent version(s)?
I assume the few graphs in your image are the aggregate + sumSeries() variants, or ... ?
Can you elaborate what "each datapoint comes in with a value of 1" means exactly?

[dforste]

I have only tied with carbon-c-relay v0.44 (38fac7)
The datapoint comes in like:
app.01.dc.01.env.prd.host.01.instance.01.sub1.01.sub2.01.sub3.01..metric.count.01 1 $epoch
So the value is to = the number of occurrences of this metric.

So the config would create new metrics as follows:
aggregated_app.01.dc.01.env.01.sub1.01.metric.count.01
aggregated_app.01.dc.01.env.01.sub2.01.metric.count.01
aggregated_app.01.dc.01.env.01.sub3.01.metric.count.01

The graphed dataset is essentially:
scaleToSeconds( consolidateBy( aggregated_app.01.dc.01.env.01.*.01.metric.count.01, sum),60)

[grobian]

consolidateBy(x,sum) => should mean per graph pixel do sum iso average (not sure what it does for a grafana/json graph)
scaleToSeconds(x,60) => should scale the results back to minutely values

The relay config will generate sums for 10-second buckets, so you should get something which is a sum of a sum for 6 buckets.

What do you expect, and what do you see?

[dforste]

I expected the values to be all the same for each of the generated metrics. I am starting to wonder if this is not a bug here but in go-carbon I use to persist and update the whisper files. Like I said earlier the archives are 10s:1m, 1m:2y. I wonder if having values in cache when updating is the problem. I will ask over there.

I just setup another aggregation like this to a whisper store with only one archive 1m:2y and values were identical.

I am going to pursue this question over there. Thanks for your time.

[grobian]

If you need anything, please let me know, I'm interested in getting to the bottom of this.