How to handle graphite multi-valued metrics

[tlk2126]

Graphite allows you to ask for something like system.*.cpu_load' and get back an array of data for everything that matches that pattern --- you get back a json array forsystem.host_a.cpu_load' and `system.host_b.cpu_load', etc.

For our use, we'd like to be able to get alerts for any of those wildcarded metrics going over the limit, e.g. "Any system load over 3" or whatnot. Right now, Tattle accepts that as a metric, but only does notification for the first metric that happens to be reported (Check::getResultValue explicitly looks at $data[0]).

Setting aside the future issue of things like "I want notified for all system's cpu load over 3, except big_host, where I don't care until it reaches 10, or test_host, where I don't care at all", I'd like to figure out the sanest way of handling this. Has anyone else thought of this, or wanted this? I've got a few ideas on how to handle the simple case, but I wanted to check with other folks before going too far down that road.

[ssandler]

This would definitely be a nice feature to add. Right now you can sort of fake it by wrapping your target in graphite's highestAverage() function I think, but that alert wouldn't tell you which or how many hosts actually tripped the check. In terms of code we'd just have to figure out the best way to process each array of results separately and how to convey the information properly in an alert.

[twinforces]

wrapping it in max will work as well.

[mreeves1]

One thing I learned today is there is a boolean OR you can apply. So in @tlk2126 's case you could use
system.{host_a,host_b}.cpu_load so you could leave out test_host...

[g76r]

@tlk2126 I think this is definitely an issue to be solved using graphite target syntax, being them {} instead of * as proposed by @mreeves1 or being them functions like maxSeries() and exclude() (see http://graphite.readthedocs.org/en/latest/functions.html#graphite.render.functions.exclude).
Therefore I close the issue.