Error when deserializing response from other nodes

[gmucha]

During certain queries the intra-cluster transport is seemingly corrupting data.

Example query in my case:

{
    "range" : { "type": "relative", "unit": "DAYS",  "value" : "365"},
    "filter" : ["key", "my-metric-2-PT3M"],
    "aggregation" : {"type" : "average", "sampling" : {"unit": "HOURS",  "value" : "2"}}
}

If the query runs locally (ie. within current instance) it completes successfully:

{
  "range": {
    "start": 1416830400000,
    "end": 1448373600000
  },
  "result": [
    {
      "type": "points",
      "hash": "7fafe0fd",
      "shard": {},
      "cadence": 7200000,
      "values": [
        [
          1416844800000,
          2.768182805494971
        ],
        [
          1416852000000,
          2.734896426939305
        ],
       ],
....

  "statistics": {
    "counters": {}
  },
  "errors": [],
  "latencies": [],
  "trace": {
    "what": {
      "name": "com.spotify.heroic.CoreQueryManager#query"
    },
    "elapsed": 331830371,
    "children": [
      {
        "what": {
          "name": "com.spotify.heroic.cluster.LocalClusterNode#query"
        },
        "elapsed": 331439004,
        "children": [
          {
            "what": {
              "name": "com.spotify.heroic.metric.LocalMetricManager#query"
            },
            "elapsed": 328125094,
            "children": []
          }
        ]
      }
    ]
  }
}

Response from cluster when querying against remote node:

{
  "range": {
    "start": 1416657600000,
    "end": 1448373600000
  },
  "result": [
    {
      "type": "points",
      "hash": "ae1a6628",
      "shard": {},
      "cadence": 7200000,
      "values": [],
      "tags": {},
      "tagCounts": {}
    }
  ],
  "statistics": {
    "counters": {}
  },
  "errors": [
    {
      "type": "node",
      "nodeId": "80aa41b3-7c79-4fa3-a7f5-864600ff0b62",
      "tags": {
        "site": "bos"
      },
      "error": "Failed to handle response, caused by Illegal character ((CTRL-CHAR, code 0)): only regular white space (\r, \n, \t) is allowed between tokens\n at [Source: [B@4a0d15; line: 1, column: 1000] (through reference chain: com.spotify.heroic.metric.ResultGroups[\"groups\"]->java.util.ArrayList[0]->com.spotify.heroic.metric.ResultGroup[\"group\"]), caused by Illegal character ((CTRL-CHAR, code 0)): only regular white space (\r, \n, \t) is allowed between tokens\n at [Source: [B@4a0d15; line: 1, column: 1000]",
      "internal": true,
      "node": "nativerpc://198.18.157.86:1394"
    }
  ],
  "latencies": [],
  "trace": {
    "what": {
      "name": "com.spotify.heroic.CoreQueryManager#query"
    },
    "elapsed": 325662602,
    "children": [
      {
        "what": {
          "name": "com.spotify.heroic.CoreQueryManager#query_node"
        },
        "elapsed": 0,
        "children": []
      }
    ]
  }
}

[gmucha]

OK, so trying to look what's happening - on the client, when deserializing the response from cluster looks truncated - see
https://gist.github.com/gmucha/68ac9f6dbe6baf7d3f31

Ok - so the problem seems to be caused by the fact that only a part of the file is being read - after the 1st read available() on GzipInputStream returns 1 -so there's more to be read. More proper version would be something like this patch (basic version,

IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- rpc/nativerpc/src/main/java/com/spotify/heroic/rpc/nativerpc/NativeUtils.java   (revision a5fb75376677b7cfcbd918557b78736a3e1af782)
+++ rpc/nativerpc/src/main/java/com/spotify/heroic/rpc/nativerpc/NativeUtils.java   (revision )
@@ -46,9 +46,16 @@
         final byte[] bytes = new byte[bodySize];

         try (final GZIPInputStream in = new GZIPInputStream(new ByteArrayInputStream(body))) {
-            in.read(bytes);
+            byte[] buf = new byte[1024];
+            int idx = 0;
+            while (in.available() > 0) {
+                int read = in.read(buf);
+                if (read > 0) {
+                    System.arraycopy(buf, 0, bytes, idx, read);
+                    idx += read;
-        }
+                }
-
+            }
+        }
         return bytes;
     }

[udoprog]

This issue should now be fixed, thanks for the report!