Composition Engine #323

Closed
wants to merge 2 commits into
from

Conversation

Projects
None yet
4 participants

Here at Chaordic we use Librato and Graphite. Librato for several reasons. As you all know Librato can't compose metrics using functions like Graphite (ex.: Graph Data > Apply Function > Calculate > Ratio). So we built an engine to compose metrics based on existing ones.
Metrics are sent to the composition engine before flushing it to backends. Compositions are calculate based on rules defined at a source file. For example:

Rule:

{
      name: "#0.ctr.#1",
      regexp:[
        /^([a-z]+)\.click\.([a-z]+)/,
        /^([a-z]+)\.view\.([a-z]+)/
      ],
      compose: function(click, view) {
        return 100 * (click / view);
      }
}

Source Metrics:

{
    counters: {
        "site.click.page": 10,
        "site.view.page": 100
    }
}

The engine will store it matching metric to an array [10, 100] where 10 is the first matching regexp and 100 is the second matching regexp. After all matches in metrics the engine starting applying collected metrics to user-defined compose functions. So it will become:

click = 10;
view = 100;

Applied to:

compose: function(click, view) {
    return 100 * (click / view);
}

Will result: 10

The output compose metric will be named like name attribute defined in the rule object. The sharps are back reference to the matching strings, so:
#0 = site
#1 = page

Then, output metric will be:

site.ctr.page = 10

The final metrics object to be flushed to backends will look like:

{
    counters: {
        "site.click.page": 10,
        "site.view.page": 100,
        "site.ctr.page": 100
    }
}

Cheers!

Owner

mrtazz commented Aug 21, 2013

This looks really interesting, thanks for sending a pull request! From reading the code it looks like the engine works on the processed metrics. I'm wondering if it would make more sense to have it as a backend instead of hooking into the flush event?

Contributor

mheffner commented Aug 22, 2013

@mrtazz I think this would need to be a processing stage between the flush event and the backends. Maybe we could add a plugin API for transforming metrics? The plugin would fire at flush time, but before the backends fire. Composite metrics would appear just as normal metrics to backends, requiring zero change to existing backends.

Owner

mrtazz commented Aug 24, 2013

Yeah I guess you're right. It would be even better if we find a way to have that plugin architecture in the metrics processing module itself. That way we would be able to do it in a single pass over the metrics.

Wow was surprised to see this pop up in my inbox in 2016. Probably worth noting for posterity while the limitations of composing metrics were true in 2013, Librato added support for "composite metrics" about a year later and have been expanding on them in the two years since.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment