Skip to content
This repository

Allow multiple metrics per packet #93

Merged
merged 2 commits into from almost 2 years ago

5 participants

Phil Sorber Don't Add Me To Your Organization a.k.a The Travis Bot Mike Heffner Daniel Schauenberg Dieter Plaetinck
Phil Sorber

This just adds a new loop that splits first on "\n". The patch looks bigger than it is because of white space chagnes.

Don't Add Me To Your Organization a.k.a The Travis Bot

This pull request passes (merged 04e397f5 into 087fde9).

Mike Heffner

The whitespace changes should really be done separately, it's hard to tell what the functional changes are.

Mike Heffner

Forgot to add: I do like the idea of supporting multiple metrics per payload and have considered submitting a similar change myself in the past.

Phil Sorber

I've reverted then separated the patch into a functional piece and a white space piece.

Don't Add Me To Your Organization a.k.a The Travis Bot

This pull request passes (merged 01bfd03 into 087fde9).

Daniel Schauenberg
Owner
mrtazz commented June 06, 2012

This is really great! I will test it some more and merge it then. It would be awesome if you could also take a stab at writing some unit tests for this.

Dieter Plaetinck

people that want to implement this in clients need to be careful to never construct udp packets larger than 512B. see http://stackoverflow.com/questions/1098897/what-is-the-largest-safe-udp-packet-size-on-the-internet

seems like a good idea to add a README change too..

Phil Sorber

I looked at writing some tests for this, but the only existing test script that I could find was for graphite. I didn't feel that I could adequately write a completely new test script for stats.js proper. If I am misunderstanding what you are requesting let me know. I am happy to do whatever you need to get this accepted.

Also, RE the README change, is that a general comment or a request for me to supply a patch for that as well?

Daniel Schauenberg
Owner
mrtazz commented June 12, 2012

The tests are pretty graphite centric, which we should improve at some point. Thanks for looking into it, it's not a blocker for getting this merged. If you can update the README, that would be great. If not, I'll add the notice into the README when merging the PR.

Daniel Schauenberg mrtazz merged commit 01bfd03 into from June 25, 2012
Daniel Schauenberg mrtazz closed this June 25, 2012
Daniel Schauenberg
Owner
mrtazz commented June 25, 2012

Finally found time to test it, sorry for the long wait.

Dieter Plaetinck

seems like there's still nothing in the readme about the udp packet descriptions, which affects this feature.

Phil Sorber

Dieterbe,

I was reluctant to add something because I didn't feel knowledgeable enough on the subject. Since then I have researched this some more and it's a rather complicated topic. It seems to me that what you are concerned about is the MTU. Even the article you linked goes into how clouded this issue is. It might be nice to mention that there are limits to the size, but what they are can vary greatly. Most internal networks where I assume statsd would be implemented are going to have an MTU of 1500 which gives you a minimum of 1432 bytes of payload (68 byte max ip + udp headers). If you are on a gige network with jumbo frame support (9000 MTU), you can do 8932 bytes. Going out over the routed internet adds tighter limits, and I think there something in the 512 byte area might be more prudent.

Given all that, I think a warning might be a good idea, maybe with suggestions for the three scenario's I mentioned. But I don't think claiming that they should never be over 512 bytes is a good idea. That would limit the benefit of this feature.

Thoughts?

Dieter Plaetinck

that makes perfect sense indeed.

Phil Sorber

Sent a new pull request with my proposed change #225.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.

Showing 1 changed file with 40 additions and 36 deletions. Show diff stats Hide diff stats

  1. 76  stats.js
76  stats.js
@@ -85,47 +85,51 @@ config.configFile(process.argv[2], function (config, oldConfig) {
85 85
     var keyFlushInterval = Number((config.keyFlush && config.keyFlush.interval) || 0);
86 86
 
87 87
     server = dgram.createSocket('udp4', function (msg, rinfo) {
88  
-      if (config.dumpMessages) { util.log(msg.toString()); }
89  
-      var bits = msg.toString().split(':');
90  
-      var key = bits.shift()
91  
-                    .replace(/\s+/g, '_')
92  
-                    .replace(/\//g, '-')
93  
-                    .replace(/[^a-zA-Z_\-0-9\.]/g, '');
94  
-
95  
-      if (keyFlushInterval > 0) {
96  
-        if (! keyCounter[key]) {
97  
-          keyCounter[key] = 0;
  88
+      var metrics = msg.toString().split("\n");
  89
+
  90
+      for (midx in metrics) {
  91
+        if (config.dumpMessages) { util.log(metrics[midx].toString()); }
  92
+        var bits = metrics[midx].toString().split(':');
  93
+        var key = bits.shift()
  94
+                      .replace(/\s+/g, '_')
  95
+                      .replace(/\//g, '-')
  96
+                      .replace(/[^a-zA-Z_\-0-9\.]/g, '');
  97
+
  98
+        if (keyFlushInterval > 0) {
  99
+          if (! keyCounter[key]) {
  100
+            keyCounter[key] = 0;
  101
+          }
  102
+          keyCounter[key] += 1;
98 103
         }
99  
-        keyCounter[key] += 1;
100  
-      }
101 104
 
102  
-      if (bits.length == 0) {
103  
-        bits.push("1");
104  
-      }
105  
-
106  
-      for (var i = 0; i < bits.length; i++) {
107  
-        var sampleRate = 1;
108  
-        var fields = bits[i].split("|");
109  
-        if (fields[1] === undefined) {
110  
-            util.log('Bad line: ' + fields);
111  
-            stats['messages']['bad_lines_seen']++;
112  
-            continue;
  105
+        if (bits.length == 0) {
  106
+          bits.push("1");
113 107
         }
114  
-        if (fields[1].trim() == "ms") {
115  
-          if (! timers[key]) {
116  
-            timers[key] = [];
117  
-          }
118  
-          timers[key].push(Number(fields[0] || 0));
119  
-        } else if (fields[1].trim() == "g") {
120  
-          gauges[key] = Number(fields[0] || 0);
121  
-        } else {
122  
-          if (fields[2] && fields[2].match(/^@([\d\.]+)/)) {
123  
-            sampleRate = Number(fields[2].match(/^@([\d\.]+)/)[1]);
  108
+
  109
+        for (var i = 0; i < bits.length; i++) {
  110
+          var sampleRate = 1;
  111
+          var fields = bits[i].split("|");
  112
+          if (fields[1] === undefined) {
  113
+              util.log('Bad line: ' + fields);
  114
+              stats['messages']['bad_lines_seen']++;
  115
+              continue;
124 116
           }
125  
-          if (! counters[key]) {
126  
-            counters[key] = 0;
  117
+          if (fields[1].trim() == "ms") {
  118
+            if (! timers[key]) {
  119
+              timers[key] = [];
  120
+            }
  121
+            timers[key].push(Number(fields[0] || 0));
  122
+          } else if (fields[1].trim() == "g") {
  123
+            gauges[key] = Number(fields[0] || 0);
  124
+          } else {
  125
+            if (fields[2] && fields[2].match(/^@([\d\.]+)/)) {
  126
+              sampleRate = Number(fields[2].match(/^@([\d\.]+)/)[1]);
  127
+            }
  128
+            if (! counters[key]) {
  129
+              counters[key] = 0;
  130
+            }
  131
+            counters[key] += Number(fields[0] || 1) * (1 / sampleRate);
127 132
           }
128  
-          counters[key] += Number(fields[0] || 1) * (1 / sampleRate);
129 133
         }
130 134
       }
131 135
 
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.