Metric Store

Metric Store

The purpose of the metric store is to allow an application to store large volumes of metric data and be able to perform calculations by grouping those metrics into time frames. For example the basic MetricStore simply stores metric data such that the sum for all the metrics can be aggregated over a minute, hour, day, or month. So you can query over a week and get back the sum of all the metrics for every day of the week. The following is an example for saving and querying data from the metric store. The query method simply retrieves a day's worth of data, and getting the sums for the hour.

public class Example {
    
    MetricStore metricStore;

    public Example(Connector connector) throws Exception{
        metricStore = new AccumuloMetricStore(connector);
    }
    
    public void saveMetrics(Collection<Metric> metrics) {
        metricStore.save(metrics);
    }
    
    public Iterable<Metric> getHourlyMetricsforDay(String group, String type, String name, Auths authorizations) {
        long now = System.currentTimeMillis();
        long dayAgo = now - TimeUnit.DAYS.toMillis(1);
        
        return metricStore.query(
                new Date(dayAgo),                        //give a query range
                new Date(now), 
                group,                                   //group type and name uniquely identify a metric
                type,                                    //if type or name are null then all values for those will be retrieved.
                name, 
                MetricTimeUnit.HOURS,                    //specify an aggregation window
                authorizations
        );
    }
}

StatsMetricStore

Understanding that simply adding up all metric data is enough, the StatsMetricStore implementation attempts to provide even more information about the data in the store. This store will additionally provide information such as the min, max, count, and sum for the data over a given aggregation timeframe. This store also provides the same API as the generic metric store, so getting simple sums is still as simple as the example above. The following is an example of retrieving the hourly stats for the day.

public Iterable<Stats> getHourlyStatsforDay(String group, String type, String name, Auths authorizations) {
    long now = System.currentTimeMillis();
    long dayAgo = now - TimeUnit.DAYS.toMillis(1);

    return metricStore.queryStats(
            new Date(dayAgo),                        //give a query range
            new Date(now),
            group,                                   //group type and name uniquely identify a metric
            type,                                    //if type or name are null then all values for those will be retrieved.
            name,
            MetricTimeUnit.HOURS,                    //specify an aggregation window
            authorizations
    );
}

CustomMetricStore

This implementation of a metric store, allows the caller to specify a custom metric function to use at query time. This is powerful if you need to run different calculations across the same metric data and the full functionality isn't provided by one of the other implementations. The major downside to using this store is that the data in the store will not be compacted, as there is no way to precalculate the type of information that will be queried later. For this reason, this store should only be used if you are looking over small time frames, the metrics are sparse, or there is no other implementation that meets your needs. Like the basic metric store, it is possible to do a simple query for summation metric data using the MetricStore API. The CustomMetricStore interface however adds the the queryCustom method to allow you to use your own custom function implementation. The following shows an example similar to the ones above.

public Iterable<CustomMetric<Long>> getHourlyMinsforDay(String group, String type, String name, Auths authorizations) throws InstantiationException, IllegalAccessException {
    long now = System.currentTimeMillis();
    long dayAgo = now - TimeUnit.DAYS.toMillis(1);

    return metricStore.queryCustom(
            new Date(dayAgo),                        //give a query range
            new Date(now),
            group,                                   //group type an name uniquely identify a metric
            type,                                    //if type or name are null then all values for those will be retrieved.
            name,
            MinFunction.class,                       //custom function definition
            MetricTimeUnit.HOURS,                    //specify an aggregation window
            authorizations
    );
}

Examples of methods to extract additional information from your metrics.

Each of these methods takes in a result from the query function to the metric service.

Find the average metric value from a query result

public static double average(Iterable<Metric> metrics) {
    long sum = 0;
    long count = 0;

    for (Metric metric : metrics) {
        //Note as this may overflow for large numbers use a checked add where applicable.
        sum += metric.getValue();
        count++;
    }
    return sum / (double)count;
}

Find the normal distribution of your metric values.

import org.apache.commons.math3.distribution.NormalDistribution;
import org.apache.commons.math3.stat.descriptive.SummaryStatistics;

public static SummaryStatistics summaryStatistics(Iterable<Metric> metrics) {
    SummaryStatistics stats = new SummaryStatistics();

    for (Metric metric : metrics)
        stats.addValue(metric.getValue());

    return stats;
}

public static NormalDistribution normalDistribution(Iterable<Metric> metrics) {
    SummaryStatistics stats = summaryStatistics(metrics);
    return new NormalDistribution(stats.getMean(), stats.getStandardDeviation());
}

Notes

These various metrics stores are not interchangable. By default they will all try to write to different tables for this reason. You cannot start using the StatsMetricStore, for example, then necessarily switch over to the CustomMetricStore on the same table. This type of behaviour is undefined.