OpenTSDB not starting up - losing connection to HBase .META

[karanalang]

opentsdb.log 2.zip

opentsdb.conf.txt
hbase-site.xml.txt

Hi All - I've kerberized HDP 2.4 (HBase version - 1.1.2.2.4.0.0-169, openTSDB version - 2.2.1) On startiong OpenTSDB, i'm getting error shown below :

What i understand (from the posts on the internet) is that this is because of connection issues between HBase & OpenTSDB

In connecting to HBase .META, the connection is failing. Any ideas on how to fix this ?

Here is the command used to start OpenTSDB

/build/tsdb tsd --zkbasedir=/hbase-secure --port=9999 --zkquorum=sandbox.hortonworks.com:2181 --cachedir=/tmp/tsd --staticroot=build/staticroot --auto-metric

Also attaching the hbase-site.xml, (the complete) opentsdb.log, opentsdb.conf

Appreciate your inputs, since i'm stuck because of this issue.

---

2016-12-14 18:56:00,391 INFO [AsyncHBase I/O Worker #4] HBaseClient: Need to find the .META. region
2016-12-14 18:56:00,392 WARN [AsyncHBase I/O Worker #4] HBaseClient: Failed to schedule timer. Ignore this if we're shutting down.
java.lang.IllegalStateException: cannot be started once stopped
at org.jboss.netty.util.HashedWheelTimer.start(HashedWheelTimer.java:279) ~[netty-3.9.4.Final.jar:na]
at org.jboss.netty.util.HashedWheelTimer.newTimeout(HashedWheelTimer.java:337) ~[netty-3.9.4.Final.jar:na]
at org.hbase.async.HBaseClient.newTimeout(HBaseClient.java:924) [asynchbase-1.7.2.jar:na]
at org.hbase.async.HBaseClient.handleNSRE(HBaseClient.java:2952) [asynchbase-1.7.2.jar:na]
at org.hbase.async.RegionClient.failOrRetryRpcs(RegionClient.java:1212) [asynchbase-1.7.2.jar:na]
at org.hbase.async.RegionClient.cleanup(RegionClient.java:1174) [asynchbase-1.7.2.jar:na]
at org.hbase.async.RegionClient.channelDisconnected(RegionClient.java:1151) [asynchbase-1.7.2.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) [netty-3.9.4.Final.jar:na]
at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1223) [asynchbase-1.7.2.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelHandler.channelDisconnected(SimpleChannelHandler.java:199) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:120) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:208) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.9.4.Final.jar:na]
at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:3121) [asynchbase-1.7.2.jar:na]
at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.4.Final.jar:na]
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.9.4.Final.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_95]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_95]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_95]
2016-12-14 18:56:00,414 INFO [AsyncHBase Timer HBaseClient #1-EventThread] TSDB: Completed shutting down the TSDB
Exception in thread "main" java.lang.RuntimeException: Initialization failed
at net.opentsdb.tools.TSDMain.main(TSDMain.java:196)
Caused by: com.stumbleupon.async.DeferredGroupException: At least one of the Deferreds failed, first exception:
at com.stumbleupon.async.DeferredGroup.done(DeferredGroup.java:169)
at com.stumbleupon.async.DeferredGroup.recordCompletion(DeferredGroup.java:142)
at com.stumbleupon.async.DeferredGroup.access$000(DeferredGroup.java:36)
at com.stumbleupon.async.DeferredGroup$1Notify.call(DeferredGroup.java:82)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)
at com.stumbleupon.async.Deferred.access$300(Deferred.java:430)
at com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1313)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1284)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1005)
at org.hbase.async.HBaseRpc.callback(HBaseRpc.java:712)
2016-12-14 18:56:00,423 INFO [AsyncHBase I/O Worker #4-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Session establishment complete on server sandbox.hortonworks.com/10.0.2.15:2181, sessionid = 0x158fa6cce650234, negotiated timeout = 5000
at org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient.java:2058)
at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1920)
at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944)
at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1005)
at org.hbase.async.HBaseClient$ZKClient$ZKCallback.processResult(HBaseClient.java:3645)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
Caused by: org.hbase.async.NonRecoverableException: Too many attempts: HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key=[116, 115, 100, 98, 45, 117, 105, 100, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58], region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=11, timeout=0, hasTimedout=false)
at org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient.java:2056)
... 9 more

[IDerr]

Hi, your TSDB can't connect to your regionservers .
if your question has been answered can you close this issue please ?

Thanks @karanalang