Skip to content
This repository

Raw and averages #105

Open
wants to merge 11 commits into from

7 participants

Jeff Minard Don't Add Me To Your Organization a.k.a The Travis Bot Daniel Schauenberg Peter N Dan Rowe Damien Nozay Jeff Minard
Jeff Minard

See #84

Basically, the same thing, but I've also updated the documentation and simplified the format for raw data.

Don't Add Me To Your Organization a.k.a The Travis Bot

This pull request passes (merged dc24ba1 into 4e4ed79).

Daniel Schauenberg
Owner

Does this make etsy/statsd#84 obsolete then?

Peter N
pcn commented March 06, 2013

I'd like to believe it does (I'm not using it yet, though) but what needs to be done to actually merge raw and averages?

Dan Rowe

Hey @pcn, now that we removed the last blocker, we're going to work on bumping version number and releasing the current version.

Definitely appologize for the delay on this.

I'd say updating this to be merge-able would be the next steps, but we are contemplating a metric abstraction, so hold off on that just yet.

Damien Nozay

any update on this pull request?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
57  README.md
Source Rendered
@@ -23,43 +23,54 @@ general values should be integer.
23 23
   After the flush interval timeout (default 10 seconds), stats are
24 24
   aggregated and sent to an upstream backend service.
25 25
 
26  
-Counting
27  
---------
  26
+Collection Methods
  27
+------------------
28 28
 
29  
-    gorets:1|c
  29
+ *  **Counting**
30 30
 
31  
-This is a simple counter. Add 1 to the "gorets" bucket. It stays in memory until the flush interval `config.flushInterval`.
  31
+        gorets:1|c
  32
+        regals:-2|c
32 33
 
  34
+    *Add 1 to the gorest bucket and subtract 2 from the regals bucket.* Counters add the reported value to the provided bucket. Counters stay in memory until the flush interval `config.flushInterval` and are zeroed after each flush. This also means that counters, once used, will always report a 0 value to the backend.
33 35
 
34  
-Timing
35  
-------
36 36
 
37  
-    glork:320|ms
  37
+ * **Timing**
38 38
 
39  
-The glork took 320ms to complete this time. StatsD figures out 90th percentile,
40  
-average (mean), lower and upper bounds for the flush interval.  The percentile
41  
-threshold can be tweaked with `config.percentThreshold`.
  39
+        glork:320|ms
42 40
 
43  
-The percentile threshold can be a single value, or a list of values, and will
44  
-generate the following list of stats for each threshold:
  41
+    *The glork took 320ms to complete this time.* StatsD figures out 90th percentile, average (mean), lower and upper bounds for the flush interval.  The percentile threshold can be tweaked with `config.percentThreshold`.
45 42
 
46  
-    stats.timers.$KEY.mean_$PCT stats.timers.$KEY.upper_$PCT
  43
+    The percentile threshold can be a single value, or a list of values, and will generate the following list of stats for each threshold:
47 44
 
48  
-Where `$KEY` is the key you stats key you specify when sending to statsd, and
49  
-`$PCT` is the percentile threshold.
  45
+        stats.timers.$KEY.mean_$PCT
  46
+        stats.timers.$KEY.upper_$PCT
50 47
 
51  
-Sampling
52  
---------
  48
+    Where `$KEY` is the key you stats key you specify when sending to statsd, and `$PCT` is the percentile threshold.
  49
+
  50
+ * **Sampling**
  51
+
  52
+        gorets:1|c|@0.1
  53
+
  54
+    Tells StatsD that this counter is being sent sampled every 1/10th of the time.
  55
+
  56
+ * **Gauges**
  57
+
  58
+        gaugor:333|g
  59
+
  60
+    *Set the gaugor bucket to 333.* Gauges store a single value per bucket and always pass that the most recently reported value to the backend at flush. For example, if you send a Guage to Statsd 12 times within your flush window, Statsd will only pass the most recent value to the backend and will continue to report that value until another is sent to Statsd.
  61
+
  62
+ * **Averaging**
  63
+
  64
+        load_5min:2.3|a
53 65
 
54  
-    gorets:1|c|@0.1
  66
+    Average are collected individually during the flush interval. At flush, the individual values will be averaged and sent to the backend as a single value.
55 67
 
56  
-Tells StatsD that this counter is being sent sampled every 1/10th of the time.
  68
+ * **Raw Data**
57 69
 
58  
-Gauges
59  
-------
60  
-StatsD now also supports gauges, arbitrary values, which can be recorded.
  70
+        population:45344|r
  71
+        population:45344|r|123456789
61 72
 
62  
-    gaugor:333|g
  73
+    Will buffer a set of metrics to send to the backend without any preprocessing. You can also specify the timestamp for the metric, or it will be added accordingly to when the metric is submitted.
63 74
 
64 75
 Debugging
65 76
 ---------
23  backends/graphite.js
@@ -51,8 +51,10 @@ var flush_stats = function graphite_flush(ts, metrics) {
51 51
   var key;
52 52
 
53 53
   var counters = metrics.counters;
54  
-  var gauges = metrics.gauges;
55 54
   var timers = metrics.timers;
  55
+  var gauges = metrics.gauges;
  56
+  var raws = metrics.raws;
  57
+  var averages = metrics.averages;
56 58
   var pctThreshold = metrics.pctThreshold;
57 59
 
58 60
   for (key in counters) {
@@ -116,6 +118,25 @@ var flush_stats = function graphite_flush(ts, metrics) {
116 118
     numStats += 1;
117 119
   }
118 120
 
  121
+  for (key in raws) {
  122
+    statString += 'stats.' + raws[key][0] + ' ' + raws[key][1] + ' ' + raws[key][2] + "\n";
  123
+    numStats += 1;
  124
+  }
  125
+
  126
+  for (key in averages) {
  127
+    var vals = averages[key],
  128
+        valCount = averages[key].length,
  129
+        valTotal = 0;
  130
+    if (vals.length >= 1) {
  131
+      for (idx in vals) {
  132
+        valTotal += vals[idx];
  133
+      }
  134
+      var averageVal = valTotal / valCount;
  135
+      statString += 'stats.' + key + ' ' + averageVal + ' ' + ts + "\n";
  136
+      numStats += 1;
  137
+    }
  138
+  }
  139
+
119 140
   statString += 'statsd.numStats ' + numStats + ' ' + ts + "\n";
120 141
   post_stats(statString);
121 142
 };
59  stats.js
@@ -8,6 +8,8 @@ var dgram  = require('dgram')
8 8
 var keyCounter = {};
9 9
 var counters = {};
10 10
 var timers = {};
  11
+var raws = [];
  12
+var averages = {};
11 13
 var gauges = {};
12 14
 var pctThreshold = null;
13 15
 var debugInt, flushInterval, keyFlushInt, server, mgmtServer;
@@ -35,6 +37,8 @@ function flushMetrics() {
35 37
 
36 38
   var metrics_hash = {
37 39
     counters: counters,
  40
+    raws: raws,
  41
+    averages: averages,
38 42
     gauges: gauges,
39 43
     timers: timers,
40 44
     pctThreshold: pctThreshold
@@ -42,7 +46,7 @@ function flushMetrics() {
42 46
 
43 47
   // After all listeners, reset the stats
44 48
   backendEvents.once('flush', function clear_metrics(ts, metrics) {
45  
-    // Clear the counters
  49
+    // Zero the counters
46 50
     for (key in metrics.counters) {
47 51
       metrics.counters[key] = 0;
48 52
     }
@@ -51,6 +55,14 @@ function flushMetrics() {
51 55
     for (key in metrics.timers) {
52 56
       metrics.timers[key] = [];
53 57
     }
  58
+
  59
+    // Clear the raws
  60
+    metrics.raws.length = 0;
  61
+
  62
+    // Clear the averages
  63
+    for (key in averages) {
  64
+      metrics.averages[key] = [];
  65
+    }
54 66
   });
55 67
 
56 68
   // Flush metrics to each backend.
@@ -72,11 +84,13 @@ config.configFile(process.argv[2], function (config, oldConfig) {
72 84
 
73 85
   if (config.debug) {
74 86
     if (debugInt !== undefined) { clearInterval(debugInt); }
75  
-    debugInt = setInterval(function () {
76  
-      util.log("Counters:\n" + util.inspect(counters) +
77  
-               "\nTimers:\n" + util.inspect(timers) +
78  
-               "\nGauges:\n" + util.inspect(gauges));
79  
-    }, config.debugInterval || 10000);
  87
+      debugInt = setInterval(function () {
  88
+        util.log("Counters:\n" + util.inspect(counters) +
  89
+                 "\nTimers:\n" + util.inspect(timers) +
  90
+                 "\nRaws:\n" + util.inspect(raws) +
  91
+                 "\nAverages:\n" + util.inspect(averages) +
  92
+                 "\nGauges:\n" + util.inspect(gauges));
  93
+      }, config.debugInterval || 10000);
80 94
   }
81 95
 
82 96
   if (server === undefined) {
@@ -118,6 +132,13 @@ config.configFile(process.argv[2], function (config, oldConfig) {
118 132
           timers[key].push(Number(fields[0] || 0));
119 133
         } else if (fields[1].trim() == "g") {
120 134
           gauges[key] = Number(fields[0] || 0);
  135
+        } else if (fields[1].trim() == "r") {
  136
+          raws.push([key, Number(fields[0] || 0), Number(fields[2] || Math.round(new Date().getTime()/1000))]);
  137
+        } else if (fields[1].trim() == "a") {
  138
+          if (! averages[key]) {
  139
+            averages[key] = [];
  140
+          }
  141
+          averages[key].push(Number(fields[0] || 0));
121 142
         } else {
122 143
           if (fields[2] && fields[2].match(/^@([\d\.]+)/)) {
123 144
             sampleRate = Number(fields[2].match(/^@([\d\.]+)/)[1]);
@@ -196,6 +217,16 @@ config.configFile(process.argv[2], function (config, oldConfig) {
196 217
             stream.write("END\n\n");
197 218
             break;
198 219
 
  220
+          case "raws":
  221
+            stream.write(util.inspect(raws) + "\n");
  222
+            stream.write("END\n\n");
  223
+            break;
  224
+
  225
+          case "averages":
  226
+            stream.write(util.inspect(averages) + "\n");
  227
+            stream.write("END\n\n");
  228
+            break;
  229
+
199 230
           case "gauges":
200 231
             stream.write(util.inspect(gauges) + "\n");
201 232
             stream.write("END\n\n");
@@ -217,6 +248,22 @@ config.configFile(process.argv[2], function (config, oldConfig) {
217 248
             stream.write("END\n\n");
218 249
             break;
219 250
 
  251
+          case "delraws":
  252
+            for (index in cmdline) {
  253
+              delete raws[cmdline[index]];
  254
+              stream.write("deleted: " + cmdline[index] + "\n");
  255
+            }
  256
+            stream.write("END\n\n");
  257
+            break;
  258
+
  259
+          case "delaverages":
  260
+            for (index in cmdline) {
  261
+              delete averages[cmdline[index]];
  262
+              stream.write("deleted: " + cmdline[index] + "\n");
  263
+            }
  264
+            stream.write("END\n\n");
  265
+            break;
  266
+
220 267
           case "delgauges":
221 268
             for (index in cmdline) {
222 269
               delete gauges[cmdline[index]];
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.