Aggregated Metric seems to be fed back into Aggregator...

[rewolf]

I have the following aggregation rule:
abc.prod.ALL.<service>.<metric>.count (60) = sum abc.prod.*.<service>.<<metric>>.count
Given metrics like:
abc.prod.host1.aservice.ametric.count
abc.prod.host2.aservice.ametric.count
I would expect them to be aggregated to
abc.prod.ALL.aservice.ametric.count

But that metric is never created. In aggregator logs, I see

Allocating new metric buffer for abc.prod.ALL.aservice.ametric.count

but it's not created. If I add a layer to the generated metric like:
abc.prod.extralayer.ALL.<service>.<metric>.count (60) = sum abc.local.*.<service>.<<metric>>.count
then we seem to get a recursive explosion of created metrics like:

abc.prod.extralayer.ALL.aservice.ametric.count
abc.prod.extralayer.ALL.ALL.aservice.ametric.count
abc.prod.extralayer.ALL.ALL.ALL.aservice.ametric.count
abc.prod.extralayer.ALL.ALL.ALL.ALL.aservice.ametric.count

Which led me to believe that the generated metric is then aggregated again...

I added a logging line to AggregationProcessor.process:

      else:
        log.clients("Found aggregate " + aggregate_metric + " for " + metric)
        aggregate_metrics.add(aggregate_metric)

And then tried with my original, desired rule.. and I eventually started to see, loglines like:

Found aggregate abc.prod.ALL.aservice.ametric.count for abc.prod.ALL.aservice.ametric.count

It matched itself as if it was a new incoming metric... Why is it being fed back into the aggregator?

[iksaif]

I think we've hit the same issue, we've fixed it with criteo-forks@825693c

[deniszh]

@iksaif - nice!
Should we include that fix in carbon or it has some side-effects?

[obfuscurity]

It appears this is a duplicate of #455.

[iksaif]

@deniszh it does have the side affect of not feedback the values in the aggregator (which some people may be depending on). My patch restore the same behavior that 0.9.x had, so feed free to merge it.

[obfuscurity]

Fixed by 69f5d0e.