Skip to content
This repository has been archived by the owner. It is now read-only.

Handle Fragmentation, small fixes #5

Closed
wants to merge 3 commits into from
Closed
Changes from 1 commit
Commits
File filter...
Filter file types
Jump to…
Jump to file
Failed to load files.

Always

Just for now

Prev

Add logic to handle message fragmentation

Collectd had a bad habit of fragmenting messages. In short, it will
send two POSTS that are meant to be handled together. In some cases,
these POSTS will each contain part of a single line of data.

This logic stores fragments for the next POST to put back together.
It discards the fragment from the first POST because it sometimes is
complete enough to parse and really mess up the DB.

Fragmentation seems to be discoverable by looking for a lack of blank
line at the end of the first POST.

Example fragments:
First POST, last line: PUTVAL host.example.com/disk-d
Second POST, first line: m-0/disk_octets interval=10 1314724861:654568
  • Loading branch information
Corry Haines
Corry Haines committed Aug 30, 2011
commit c6d3d8f528ac694ccb73f899db9ac345128a8b0e
@@ -56,11 +56,31 @@ graphite_connection.on("error", function() {
throw new Error("Connection error");
});

var fragment = ""

var request_handler = function(request, response) {
var putval_re = /^PUTVAL ([^ ]+)(?: ([^ ]+=[^ ]+)?) ([0-9.]+)(:.*)/;
request.addListener("data", function(chunk) {
metrics = chunk.toString().split("\n");
for (var i in metrics) {
metrics = chunk.toString().split("\r\n");

if ( fragment.length > 0 ) {
// Discard the fragment if the new line is well formed
// I have never seen this, but it may be possible
if ( String(metrics[0]).indexOf("PUTVAL") < 0 ) {
metrics[0] = fragment.concat(metrics[0])
}
fragment = ""
}
// If the last line is not empty, then this chunk is likely split
// in two. So, we will store the fragment for the next round
// The fragment is not used in this run, as it is corrupted and may
// actaully pass the regex (and mess up the DB)
if ( String(metrics[metrics.length-1]).length != 0 ) {
fragment = metrics[metrics.length-1]
metrics[metrics.length-1] = ""
}

for (var i=0; i<metrics.length; i++) {
var m = putval_re.exec(metrics[i]);
if (!m) {
continue;
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.