Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

2.4 Error when avg downsampling rollup data using api/query endpoint #1501

Open
rudolphkurt opened this issue Jan 23, 2019 · 5 comments
Open

2.4 Error when avg downsampling rollup data using api/query endpoint #1501

rudolphkurt opened this issue Jan 23, 2019 · 5 comments
Labels
bug

Comments

@rudolphkurt
Copy link

I am attempting to test out the rollup functionality in 2.4. I followed the documentation and inserted some rollup data using the api/rollup endpoint. As per the document, I did not add any AVG aggregations to the rollup table and instead used only SUM and COUNT.

When I attempt to perform an avg downsample using the api/query endpoint, I receive and error stating that I don't have AVG in my rollup_config.json, however, if I use the command line query tool, I get the expected result.

My query and error

curl localhost:4242/api/query -d '
{
  "start":1546344000,
   "end":1546347600,
   "msResolution":false,
   "queries":[{
     "aggregator":"none",
     "metric":"client.current_conns",
     "rate":false,
     "downsample":"1h-avg",
     "rollupUsage": "ROLLUP_NOFALLBACK"
  }]
 }'

{"error":{"code":400,"message":"No ID found mapping to aggregator: avg","details":"No ID found mapping to aggregator: avg","trace":"net.opentsdb.tsd.BadRequestException: No ID found mapping to aggregator: avg\n\tat net.opentsdb.tsd.QueryRpc$1ErrorCB.call(QueryRpc.java:220) [tsdb-2.4.0.jar:]\n\tat net.opentsdb.tsd.QueryRpc$1ErrorCB.call(QueryRpc.java:180) [tsdb-2.4.0.jar:]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.DeferredGroup.done(DeferredGroup.java:169) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.DeferredGroup.recordCompletion(DeferredGroup.java:158) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.DeferredGroup.access$200(DeferredGroup.java:36) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.DeferredGroup$1NotifyOrdered.call(DeferredGroup.java:97) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]\n\tat net.opentsdb.core.SaltScanner.validateAndTriggerCallback(SaltScanner.java:949) [tsdb-2.4.0.jar:]\n\tat net.opentsdb.core.SaltScanner.access$2300(SaltScanner.java:68) [tsdb-2.4.0.jar:]\n\tat net.opentsdb.core.SaltScanner$ScannerCB.close(SaltScanner.java:910) [tsdb-2.4.0.jar:]\n\tat net.opentsdb.core.SaltScanner$ScannerCB.call(SaltScanner.java:539) [tsdb-2.4.0.jar:]\n\tat net.opentsdb.core.SaltScanner$ScannerCB.call(SaltScanner.java:461) [tsdb-2.4.0.jar:]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]\n\tat org.hbase.async.HBaseRpc.callback(HBaseRpc.java:720) [asynchbase-1.8.2.jar:na]\n\tat org.hbase.async.RegionClient.decode(RegionClient.java:1575) [asynchbase-1.8.2.jar:na]\n\tat org.hbase.async.RegionClient.decode(RegionClient.java:88) [asynchbase-1.8.2.jar:na]\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:na]\n\tat org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1230) [asynchbase-1.8.2.jar:na]\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.10.6.Final.jar:na]\n\tat org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:3857) [asynchbase-1.8.2.jar:na]\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.10.6.Final.jar:na]\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]\n\tat java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]\nCaused by: java.lang.IllegalArgumentException: No ID found mapping to aggregator: avg\n\tat net.opentsdb.rollup.RollupConfig.getIdForAggregator(RollupConfig.java:285) ~[tsdb-2.4.0.jar:]\n\tat net.opentsdb.rollup.RollupSeq.<init>(RollupSeq.java:115) ~[tsdb-2.4.0.jar:]\n\tat net.opentsdb.rollup.RollupSpan.createRowSequence(RollupSpan.java:82) ~[tsdb-2.4.0.jar:]\n\tat net.opentsdb.rollup.RollupSpan.addRow(RollupSpan.java:69) ~[tsdb-2.4.0.jar:]\n\tat net.opentsdb.core.SaltScanner.mergeDataPoints(SaltScanner.java:429) [tsdb-2.4.0.jar:]\n\tat net.opentsdb.core.SaltScanner.mergeAndReturnResults(SaltScanner.java:307) [tsdb-2.4.0.jar:]\n\tat net.opentsdb.core.SaltScanner.validateAndTriggerCallback(SaltScanner.java:947) [tsdb-2.4.0.jar:]\n\t... 42 common frames omitted\n"}}

The output from the cli tool

tsdb query 1546344000 1546347600 none downsample 3600000 avg client.current_conns
client.current_conns 1546344000000 33.250000 {host=host2, port=80, datacenter=dc1}
client.current_conns 1546344000000 40.500000 {host=host1, port=80, datacenter=dc1}

tsdb config file

tsd.storage.hbase.data_table = tsdb
tsd.storage.hbase.uid_table = tsdb-uid
tsd.storage.hbase.meta_table = tsdb-meta
tsd.storage.hbase.tree_table = tsdb-tree
tsd.rollups.enable = true
tsd.rollups.config = /etc/opentsdb/rollup_config.json

tsd.core.meta.enable_tracking       = true
tsd.core.meta.enable_realtime_ts    = true
tsd.core.meta.enable_realtime_uid   = true
tsd.core.meta.enable_tsuid_tracking = true
tsd.http.query.allow_delete         = true
tsd.http.request.enable_chunked     = true
tsd.http.request.max_chunk          = 33554432
tsd.query.enable_fuzzy_filter       = false
tsd.storage.enable_appends          = false
tsd.storage.fix_duplicates          = true
tsd.storage.max_tags                = 16
tsd.storage.repair_appends          = false
tsd.storage.salt.buckets            = 6
tsd.storage.salt.width              = 1

rollup_config.json

{
  "aggregationIds": {
    "sum": 0,
    "count": 1,
    "min": 2,
    "max": 3
  },
  "intervals": [
    {
      "table": "tsdb",
      "preAggregationTable": "tsdb-preagg",
      "interval": "1m",
      "rowSpan": "1h",
      "defaultInterval": true
    },
    {
      "table": "tsdb-rollup-1h",
      "preAggregationTable": "tsdb-rollup-preagg-1h",
      "interval": "1h",
      "rowSpan": "1d"
    }
  ]
}

I used these commands to insert the rollup data

curl -XPOST http://localhost:4242/api/rollup?details
[
  {
    "metric": "client.current_conns",
    "timestamp": 1546344000,
    "value": 133,
    "tags": {
      "host": "host2",
      "datacenter": "dc1",
      "port": "80"
    },
    "interval": "1h",
    "aggregator": "SUM"
  },
  {
    "metric": "client.current_conns",
    "timestamp": 1546344000,
    "value": 162,
    "tags": {
      "host": "host1",
      "datacenter": "dc1",
      "port": "80"
    },
    "interval": "1h",
    "aggregator": "SUM"
  },
  {
    "metric": "client.current_conns",
    "timestamp": 1546344000,
    "value": 4,
    "tags": {
      "host": "host2",
      "datacenter": "dc1",
      "port": "80"
    },
    "interval": "1h",
    "aggregator": "COUNT"
  },
  {
    "metric": "client.current_conns",
    "timestamp": 1546344000,
    "value": 4,
    "tags": {
      "host": "host1",
      "datacenter": "dc1",
      "port": "80"
    },
    "interval": "1h",
    "aggregator": "COUNT"
  }
]'
@manolama
Copy link
Member

Hi, do you have any stack traces from the logs where the AVG exception occurred?

@manolama manolama added the bug label Jan 27, 2019
@rudolphkurt
Copy link
Author 

13:03:01.295 ERROR [SaltScanner.mergeDataPoints] - Exception adding row to span
java.lang.IllegalArgumentException: No ID found mapping to aggregator: avg
	at net.opentsdb.rollup.RollupConfig.getIdForAggregator(RollupConfig.java:285) ~[tsdb-2.4.0.jar:]
	at net.opentsdb.rollup.RollupSeq.<init>(RollupSeq.java:115) ~[tsdb-2.4.0.jar:]
	at net.opentsdb.rollup.RollupSpan.createRowSequence(RollupSpan.java:82) ~[tsdb-2.4.0.jar:]
	at net.opentsdb.rollup.RollupSpan.addRow(RollupSpan.java:69) ~[tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner.mergeDataPoints(SaltScanner.java:429) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner.mergeAndReturnResults(SaltScanner.java:307) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner.validateAndTriggerCallback(SaltScanner.java:947) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner.access$2300(SaltScanner.java:68) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner$ScannerCB.close(SaltScanner.java:910) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner$ScannerCB.call(SaltScanner.java:539) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner$ScannerCB.call(SaltScanner.java:461) [tsdb-2.4.0.jar:]
	at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]
	at org.hbase.async.HBaseRpc.callback(HBaseRpc.java:720) [asynchbase-1.8.2.jar:na]
	at org.hbase.async.RegionClient.decode(RegionClient.java:1575) [asynchbase-1.8.2.jar:na]
	at org.hbase.async.RegionClient.decode(RegionClient.java:88) [asynchbase-1.8.2.jar:na]
	at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:na]
	at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1230) [asynchbase-1.8.2.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.10.6.Final.jar:na]
	at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:3857) [asynchbase-1.8.2.jar:na]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.10.6.Final.jar:na]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
13:03:01.297 ERROR [QueryRpc.call] - Query exception:
com.stumbleupon.async.DeferredGroupException: At least one of the Deferreds failed, first exception:
	at com.stumbleupon.async.DeferredGroup.done(DeferredGroup.java:169) [async-1.4.0.jar:na]
	at com.stumbleupon.async.DeferredGroup.recordCompletion(DeferredGroup.java:158) [async-1.4.0.jar:na]
	at com.stumbleupon.async.DeferredGroup.access$200(DeferredGroup.java:36) [async-1.4.0.jar:na]
	at com.stumbleupon.async.DeferredGroup$1NotifyOrdered.call(DeferredGroup.java:97) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]
	at net.opentsdb.core.SaltScanner.validateAndTriggerCallback(SaltScanner.java:949) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner.access$2300(SaltScanner.java:68) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner$ScannerCB.close(SaltScanner.java:910) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner$ScannerCB.call(SaltScanner.java:539) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner$ScannerCB.call(SaltScanner.java:461) [tsdb-2.4.0.jar:]
	at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]
	at org.hbase.async.HBaseRpc.callback(HBaseRpc.java:720) [asynchbase-1.8.2.jar:na]
	at org.hbase.async.RegionClient.decode(RegionClient.java:1575) [asynchbase-1.8.2.jar:na]
	at org.hbase.async.RegionClient.decode(RegionClient.java:88) [asynchbase-1.8.2.jar:na]
	at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:na]
	at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1230) [asynchbase-1.8.2.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.10.6.Final.jar:na]
	at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:3857) [asynchbase-1.8.2.jar:na]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.10.6.Final.jar:na]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: java.lang.IllegalArgumentException: No ID found mapping to aggregator: avg
	at net.opentsdb.rollup.RollupConfig.getIdForAggregator(RollupConfig.java:285) ~[tsdb-2.4.0.jar:]
	at net.opentsdb.rollup.RollupSeq.<init>(RollupSeq.java:115) ~[tsdb-2.4.0.jar:]
	at net.opentsdb.rollup.RollupSpan.createRowSequence(RollupSpan.java:82) ~[tsdb-2.4.0.jar:]
	at net.opentsdb.rollup.RollupSpan.addRow(RollupSpan.java:69) ~[tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner.mergeDataPoints(SaltScanner.java:429) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner.mergeAndReturnResults(SaltScanner.java:307) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner.validateAndTriggerCallback(SaltScanner.java:947) [tsdb-2.4.0.jar:]
	... 42 common frames omitted
13:03:01.301 WARN  [HttpQuery.logWarn] - [id: 0x410a8c11, /0:0:0:0:0:0:0:1:60026 => /0:0:0:0:0:0:0:1:4246] Bad Request on /api/query: No ID found mapping to aggregator: avg

If I try to add the "avg" aggregator to the rollup_config.json file, then I get the following error when running the query

from STDOUT
{"error":{"code":400,"message":"Attempt to add a different aggrregate cell =KeyValue(key=[2, 0, 3, 40, 92, 42, -83, -128, 0, 0, 1, 0, 2, -6, 0, 0, 14, 0, 2, -7, 0, 0, 23, 0, 2, -5], family=\"t\", qualifier=[0, 0, -63], value=[0, -94], timestamp=1548183838441), expected aggregator \"raw:\"","details":"Attempt to add a different aggrregate cell =KeyValue(key=[2, 0, 3, 40, 92, 42, -83, -128, 0, 0, 1, 0, 2, -6, 0, 0, 14, 0, 2, -7, 0, 0, 23, 0, 2, -5], family=\"t\", qualifier=[0, 0, -63], value=[0, -94], timestamp=1548183838441), expected aggregator \"raw:\"","trace":"net.opentsdb.tsd.BadRequestException: Attempt to add a different aggrregate cell =KeyValue(key=[2, 0, 3, 40, 92, 42, -83, -128, 0, 0, 1, 0, 2, -6, 0, 0, 14, 0, 2, -7, 0, 0, 23, 0, 2, -5], family=\"t\", qualifier=[0, 0, -63], value=[0, -94], timestamp=1548183838441), expected aggregator \"raw:\"\n\tat net.opentsdb.tsd.QueryRpc$1ErrorCB.call(QueryRpc.java:220) [tsdb-2.4.0.jar:]\n\tat net.opentsdb.tsd.QueryRpc$1ErrorCB.call(QueryRpc.java:180) [tsdb-2.4.0.jar:]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.DeferredGroup.done(DeferredGroup.java:169) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.DeferredGroup.recordCompletion(DeferredGroup.java:158) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.DeferredGroup.access$200(DeferredGroup.java:36) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.DeferredGroup$1NotifyOrdered.call(DeferredGroup.java:97) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]\n\tat net.opentsdb.core.SaltScanner.validateAndTriggerCallback(SaltScanner.java:949) [tsdb-2.4.0.jar:]\n\tat net.opentsdb.core.SaltScanner.access$2300(SaltScanner.java:68) [tsdb-2.4.0.jar:]\n\tat net.opentsdb.core.SaltScanner$ScannerCB.close(SaltScanner.java:910) [tsdb-2.4.0.jar:]\n\tat net.opentsdb.core.SaltScanner$ScannerCB.call(SaltScanner.java:539) [tsdb-2.4.0.jar:]\n\tat net.opentsdb.core.SaltScanner$ScannerCB.call(SaltScanner.java:461) [tsdb-2.4.0.jar:]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]\n\tat com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]\n\tat org.hbase.async.HBaseRpc.callback(HBaseRpc.java:720) [asynchbase-1.8.2.jar:na]\n\tat org.hbase.async.RegionClient.decode(RegionClient.java:1575) [asynchbase-1.8.2.jar:na]\n\tat org.hbase.async.RegionClient.decode(RegionClient.java:88) [asynchbase-1.8.2.jar:na]\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:na]\n\tat org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1230) [asynchbase-1.8.2.jar:na]\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.10.6.Final.jar:na]\n\tat org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:3857) [asynchbase-1.8.2.jar:na]\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.10.6.Final.jar:na]\n\tat org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.10.6.Final.jar:na]\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]\n\tat java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]\nCaused by: net.opentsdb.core.IllegalDataException: Attempt to add a different aggrregate cell =KeyValue(key=[2, 0, 3, 40, 92, 42, -83, -128, 0, 0, 1, 0, 2, -6, 0, 0, 14, 0, 2, -7, 0, 0, 23, 0, 2, -5], family=\"t\", qualifier=[0, 0, -63], value=[0, -94], timestamp=1548183838441), expected aggregator \"raw:\"\n\tat net.opentsdb.rollup.RollupSeq.setRow(RollupSeq.java:158) ~[tsdb-2.4.0.jar:]\n\tat net.opentsdb.rollup.RollupSpan.addRow(RollupSpan.java:70) ~[tsdb-2.4.0.jar:]\n\tat net.opentsdb.core.SaltScanner.mergeDataPoints(SaltScanner.java:429) [tsdb-2.4.0.jar:]\n\tat net.opentsdb.core.SaltScanner.mergeAndReturnResults(SaltScanner.java:307) [tsdb-2.4.0.jar:]\n\tat net.opentsdb.core.SaltScanner.validateAndTriggerCallback(SaltScanner.java:947) [tsdb-2.4.0.jar:]\n\t... 42 common frames omitted\n"}}

and from the log

13:06:10.889 ERROR [SaltScanner.mergeDataPoints] - Exception adding row to span
net.opentsdb.core.IllegalDataException: Attempt to add a different aggrregate cell =KeyValue(key=[2, 0, 3, 40, 92, 42, -83, -128, 0, 0, 1, 0, 2, -6, 0, 0, 14, 0, 2, -7, 0, 0, 23, 0, 2, -5], family="t", qualifier=[0, 0, -63], value=[0, -94], timestamp=1548183838441), expected aggregator "raw:"
	at net.opentsdb.rollup.RollupSeq.setRow(RollupSeq.java:158) ~[tsdb-2.4.0.jar:]
	at net.opentsdb.rollup.RollupSpan.addRow(RollupSpan.java:70) ~[tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner.mergeDataPoints(SaltScanner.java:429) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner.mergeAndReturnResults(SaltScanner.java:307) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner.validateAndTriggerCallback(SaltScanner.java:947) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner.access$2300(SaltScanner.java:68) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner$ScannerCB.close(SaltScanner.java:910) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner$ScannerCB.call(SaltScanner.java:539) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner$ScannerCB.call(SaltScanner.java:461) [tsdb-2.4.0.jar:]
	at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]
	at org.hbase.async.HBaseRpc.callback(HBaseRpc.java:720) [asynchbase-1.8.2.jar:na]
	at org.hbase.async.RegionClient.decode(RegionClient.java:1575) [asynchbase-1.8.2.jar:na]
	at org.hbase.async.RegionClient.decode(RegionClient.java:88) [asynchbase-1.8.2.jar:na]
	at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:na]
	at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1230) [asynchbase-1.8.2.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.10.6.Final.jar:na]
	at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:3857) [asynchbase-1.8.2.jar:na]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.10.6.Final.jar:na]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
13:06:10.891 ERROR [QueryRpc.call] - Query exception:
com.stumbleupon.async.DeferredGroupException: At least one of the Deferreds failed, first exception:
	at com.stumbleupon.async.DeferredGroup.done(DeferredGroup.java:169) [async-1.4.0.jar:na]
	at com.stumbleupon.async.DeferredGroup.recordCompletion(DeferredGroup.java:158) [async-1.4.0.jar:na]
	at com.stumbleupon.async.DeferredGroup.access$200(DeferredGroup.java:36) [async-1.4.0.jar:na]
	at com.stumbleupon.async.DeferredGroup$1NotifyOrdered.call(DeferredGroup.java:97) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]
	at net.opentsdb.core.SaltScanner.validateAndTriggerCallback(SaltScanner.java:949) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner.access$2300(SaltScanner.java:68) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner$ScannerCB.close(SaltScanner.java:910) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner$ScannerCB.call(SaltScanner.java:539) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner$ScannerCB.call(SaltScanner.java:461) [tsdb-2.4.0.jar:]
	at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.access$300(Deferred.java:430) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]
	at org.hbase.async.HBaseRpc.callback(HBaseRpc.java:720) [asynchbase-1.8.2.jar:na]
	at org.hbase.async.RegionClient.decode(RegionClient.java:1575) [asynchbase-1.8.2.jar:na]
	at org.hbase.async.RegionClient.decode(RegionClient.java:88) [asynchbase-1.8.2.jar:na]
	at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:na]
	at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1230) [asynchbase-1.8.2.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.10.6.Final.jar:na]
	at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:3857) [asynchbase-1.8.2.jar:na]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.10.6.Final.jar:na]
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.10.6.Final.jar:na]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: net.opentsdb.core.IllegalDataException: Attempt to add a different aggrregate cell =KeyValue(key=[2, 0, 3, 40, 92, 42, -83, -128, 0, 0, 1, 0, 2, -6, 0, 0, 14, 0, 2, -7, 0, 0, 23, 0, 2, -5], family="t", qualifier=[0, 0, -63], value=[0, -94], timestamp=1548183838441), expected aggregator "raw:"
	at net.opentsdb.rollup.RollupSeq.setRow(RollupSeq.java:158) ~[tsdb-2.4.0.jar:]
	at net.opentsdb.rollup.RollupSpan.addRow(RollupSpan.java:70) ~[tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner.mergeDataPoints(SaltScanner.java:429) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner.mergeAndReturnResults(SaltScanner.java:307) [tsdb-2.4.0.jar:]
	at net.opentsdb.core.SaltScanner.validateAndTriggerCallback(SaltScanner.java:947) [tsdb-2.4.0.jar:]
	... 42 common frames omitted
13:06:10.895 WARN  [HttpQuery.logWarn] - [id: 0xa8dba13e, /0:0:0:0:0:0:0:1:60170 => /0:0:0:0:0:0:0:1:4246] Bad Request on /api/query: Attempt to add a different aggrregate cell =KeyValue(key=[2, 0, 3, 40, 92, 42, -83, -128, 0, 0, 1, 0, 2, -6, 0, 0, 14, 0, 2, -7, 0, 0, 23, 0, 2, -5], family="t", qualifier=[0, 0, -63], value=[0, -94], timestamp=1548183838441), expected aggregator "raw:"

@maduxi
Copy link

maduxi commented Jun 18, 2019

Im having the same issue with a very similar setup. @rudolphkurt where you able to make this work?

@rudolphkurt
Copy link
Author

@maduxi , unfortunately no.

@neilfordyce
Copy link
Contributor

The query works if you add avg to aggreagtionIds in the json config and change the aggregator from none to match the downsample aggreagtor, e.g. with reference to the original query:

curl localhost:4242/api/query -d '
{
  "start":1546344000,
   "end":1546347600,
   "msResolution":false,
   "queries":[{
     "aggregator":"avg",
     "metric":"client.current_conns",
     "rate":false,
     "downsample":"1h-avg",
     "rollupUsage": "ROLLUP_NOFALLBACK"
  }]
 }'

I think there is an issue with some aggregation logic here

opentsdb/src/rollup/RollupSeq.java

Line 99 in b5133bd

need_count = rollup_query.getGroupBy() == Aggregators.AVG ||

I think that should point to rollup_query.getRollupAgg() instead. The idea is that tsdb should only retrieve the count rollup rows from storage if the downsample aggregator is avg. It should not matter what the series aggregator is set to.

I want to do a bit more testing, but hopefully that fixes it. Happy to open a PR soon.

@neilfordyce neilfordyce mentioned this issue Jun 20, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@manolama @neilfordyce @maduxi @rudolphkurt