Opentsdb on Mapr error: Deferred.handleContinuation] - callback=shutdown HBase client@1898325501 returned Deferred@2075809815

[shivam-dwivedi]

Hi All,
I am trying to implement Opensource OpenTSDB on MaprDB after all the configuration changes.. tsd daemon just shutsdown .. i dont see any major error message. Below are the logs. Appreciate if anyone can help me.

/usr/share/opentsdb/bin/tsdb tsd --port=4242 --staticroot="/usr/share/opentsdb/static/" --cachedir="/tmp/opentsdb" --config=/etc/opentsdb/opentsdb.conf
Exception in thread "main" java.lang.RuntimeException: Initialization failed
at net.opentsdb.tools.TSDMain.main(TSDMain.java:289)
Caused by: java.lang.NullPointerException
at net.opentsdb.utils.Config.getBoolean(Config.java:413)
at net.opentsdb.tools.TSDMain.main(TSDMain.java:248)

===
Logs

15:57:36.186 INFO [TSDMain.main] - Starting.
15:57:36.189 INFO [TSDMain.main] - net.opentsdb.tools 2.4.0 built at revision 8d13bd5 (MODIFIED)
15:57:36.189 INFO [TSDMain.main] - Built on 2019/10/31 19:09:07 +0000 by root@3d8f599bb923:/root/opensource/mapr-opentsdb-2.4.0/dl/mapr-opentsdb-2.4.0/build
15:57:36.192 INFO [Config.loadConfig] - Successfully loaded configuration file: /etc/opentsdb/opentsdb.conf
15:57:36.199 DEBUG [SelectorUtil.debug] - Using select timeout of 500
15:57:36.199 DEBUG [SelectorUtil.debug] - Epoll-bug workaround enabled = false
15:57:36.258 INFO [Config.loadConfig] - Successfully loaded configuration file: /etc/opentsdb/opentsdb.conf
15:57:36.423 DEBUG [CLDBRpcCommonUtils.init] - init
15:57:36.427 DEBUG [JVMProperties.setPropIfUnset] - Setting JVM property javax.net.ssl.trustStore to /opt/mapr/conf/ssl_truststore
15:57:36.427 DEBUG [JVMProperties.setPropIfUnset] - Setting JVM property http.auth.preference to basic
15:57:36.427 DEBUG [JVMProperties.setPropIfUnset] - Setting JVM property zookeeper.saslprovider to com.mapr.security.maprsasl.MaprSaslProvider
15:57:36.427 DEBUG [JVMProperties.setPropIfUnset] - Setting JVM property java.security.auth.login.config to /opt/mapr/conf/mapr.login.conf
15:57:36.429 DEBUG [Configuration.getProperties] - Loaded org.apache.hadoop.conf.CoreDefaultProperties
15:57:36.432 INFO [MapRThreadPool.] - Creating ThreadPoolExecutor with 128 workerThreads and 5 callbackThreads.
15:57:36.460 DEBUG [HBaseClient.] - Connection Type: NONE
15:57:36.478 DEBUG [Configuration.getProperties] - Loaded org.apache.hadoop.conf.CoreDefaultProperties
15:57:36.532 DEBUG [Shell.isSetsidSupported] - setsid exited with exit code 0
15:57:36.538 DEBUG [Configuration.getProperties] - Loaded org.apache.hadoop.conf.CoreDefaultProperties
15:57:36.572 DEBUG [MutableMetricsFactory.newForField] - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
15:57:36.576 DEBUG [MutableMetricsFactory.newForField] - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
15:57:36.576 DEBUG [MutableMetricsFactory.newForField] - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
15:57:36.577 DEBUG [MetricsSystemImpl.register] - UgiMetrics, User and group related metrics
15:57:36.584 DEBUG [Configuration.getProperties] - Loaded org.apache.hadoop.conf.CoreDefaultProperties
15:57:36.585 DEBUG [UserGroupInformation.initialize] - HADOOP_SECURITY_AUTHENTICATION is set to: CUSTOM
15:57:36.591 DEBUG [Configuration.getProperties] - Loaded org.apache.hadoop.conf.CoreDefaultProperties
15:57:36.615 DEBUG [RpcAuthRegistry.addRpcAuthMethod] - Added SIMPLE to registry.
15:57:36.615 DEBUG [RpcAuthRegistry.addRpcAuthMethod] - Added KERBEROS to registry.
15:57:36.615 DEBUG [RpcAuthRegistry.addRpcAuthMethod] - Added TOKEN to registry.
15:57:36.615 DEBUG [RpcAuthRegistry.addRpcAuthMethod] - Added FAKE to registry.
15:57:36.615 DEBUG [RpcAuthRegistry.addRpcAuthMethod] - Added MAPRSASL to registry.
15:57:36.616 DEBUG [UserGroupInformation.initialize] - Security is enabled.
15:57:36.616 DEBUG [UserGroupInformation.initialize] - Login configuration entry is hadoop_default
15:57:36.616 DEBUG [UserGroupInformation.setUGIAuthenticationMethodFromJAASConfiguration] - authenticationMethod from JAAS configuration:CUSTOM
15:57:36.619 DEBUG [Groups.getUserToGroupsMappingService] - Creating new Groups object
15:57:36.621 DEBUG [NativeCodeLoader.] - Trying to load the custom-built native-hadoop library...
15:57:36.621 DEBUG [NativeCodeLoader.] - Loaded the native-hadoop library
15:57:36.621 DEBUG [JniBasedUnixGroupsMapping.] - Using JniBasedUnixGroupsMapping for Group resolution
15:57:36.622 DEBUG [JniBasedUnixGroupsMappingWithFallback.] - Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping
15:57:36.624 DEBUG [Groups.] - Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
15:57:36.630 DEBUG [MaprSecurityLoginModule.login] - Need to addplainticket: false
15:57:36.633 DEBUG [MapRLoginHttpsClient.authenticateIfNeeded] - Entering authenticate if needed.
15:57:36.634 DEBUG [MapRLoginHttpsClient.isKerberosAnOption] - Kerberos not configured for this cluster.
15:57:36.645 DEBUG [MapRLoginHttpsClient.doesSecurityHaveGoodKey] - found existing MapR ticket
15:57:36.646 DEBUG [MapRLoginHttpsClient.authenticateIfNeeded] - Already have good ticket, done
15:57:36.647 DEBUG [HadoopLoginModule.initialize] - Priority principal search list is [class javax.security.auth.kerberos.KerberosPrincipal, class com.mapr.security.MapRPrincipal]
15:57:36.647 DEBUG [HadoopLoginModule.initialize] - Additional principal search list is [class com.sun.security.auth.UnixPrincipal]
15:57:36.647 DEBUG [HadoopLoginModule.login] - hadoop login
15:57:36.647 DEBUG [HadoopLoginModule.commit] - hadoop login commit
15:57:36.648 DEBUG [UserGroupInformation.] - found custom auth principal com.mapr.security.MapRPrincipal in subject. marking authentication method as CUSTOM
15:57:36.649 DEBUG [UserGroupInformation.] - found custom auth principal com.mapr.security.MapRPrincipal in subject. marking authentication method as CUSTOM
15:57:36.649 DEBUG [UserGroupInformation.loginUserFromSubject] - UGI loginUser:mapr (auth:CUSTOM)
15:57:36.686 DEBUG [TSDB.] - TSD Configuration:
File [/etc/opentsdb/opentsdb.conf]
Key [tsd.core.meta.enable_realtime_uid] Value [false]
Key [tsd.storage.hbase.tree_table] Value [/user/mapr/tsdb-tree]
Key [tsd.core.meta.cache.enable] Value [false]
Key [tsd.core.stats_with_port] Value [false]
Key [tsd.http.cachedir] Value [/tmp/opentsdb]
Key [tsd.uid.lru.id.size] Value [5000000]
Key [tsd.core.tree.enable_processing] Value [false]
Key [tsd.query.multi_get.concurrent] Value [20]
Key [tsd.storage.use_otsdb_timestamp] Value [false]
Key [tsd.query.enable_fuzzy_filter] Value [true]
Key [tsd.storage.get_date_tiered_compaction_start] Value [0]
Key [tsd.query.skip_unresolved_tagvs] Value [false]
Key [tsd.storage.compaction.flush_speed] Value [2]
Key [tsd.rollups.enable] Value [false]
Key [tsd.timeseriesfilter.enable] Value [false]
Key [tsd.no_diediedie] Value [false]
Key [tsd.query.limits.overrides.interval] Value [60000]
Key [tsd.query.multi_get.limit] Value [131072]
Key [tsd.storage.hbase.prefetch_meta] Value [false]
Key [tsd.core.meta.enable_tsuid_incrementing] Value [false]
Key [tsd.query.filter.expansion_limit] Value [4096]
Key [tsd.core.auto_create_metrics] Value [true]
Key [tsd.core.plugin_path] Value [/usr/share/opentsdb/plugins]
Key [tsd.query.limits.bytes.default] Value [0]
Key [tsd.startup.enable] Value [false]
Key [tsd.http.request.cors_domains] Value []
Key [tsd.search.plugin] Value []
Key [tsd.query.multi_get.batch_size] Value [1024]
Key [tsd.uid.use_mode] Value [false]
Key [tsd.storage.fix_duplicates] Value [false]
Key [tsd.network.keep_alive] Value [true]
Key [tsd.storage.use_max_value] Value [true]
Key [tsd.network.port] Value [14242]
Key [tsd.core.uid.random_metrics] Value [false]
Key [HADOOP_HOME] Value [/opt/mapr/hadoop/hadoop-2.7.0]
Key [tsd.query.limits.bytes.allow_override] Value [false]
Key [tsd.storage.flush_interval] Value [1000]
Key [tsd.storage.enable_compaction] Value [true]
Key [tsd.core.bulk.allow_out_of_order_timestamps] Value [false]
Key [tsd.storage.hbase.data_table] Value [/tmp/mapr.opentsdb/tsdb]
Key [tsd.core.meta.enable_realtime_ts] Value [false]
Key [tsd.startup.plugin] Value []
Key [tsd.uid.lru.name.size] Value [5000000]
Key [tsd.core.auto_create_tagks] Value [true]
Key [tsd.query.limits.data_points.allow_override] Value [false]
Key [tsd.rtpublisher.enable] Value [false]
Key [tsd.network.worker_threads] Value []
Key [tsd.core.preload_uid_cache.max_entries] Value [300000]
Key [tsd.core.authentication.enable] Value [false]
Key [tsd.rtpublisher.plugin] Value []
Key [tsd.rollups.block_derived] Value [true]
Key [tsd.uidfilter.enable] Value [false]
Key [tsd.rollups.agg_tag_key] Value [_aggregate]
Key [tsd.network.async_io] Value [true]
Key [tsd.core.authentication.plugin] Value []
Key [tsd.core.enable_ui] Value [true]
Key [tsd.query.timeout] Value [0]
Key [tsd.http.request.max_chunk] Value [4096]
Key [tsd.storage.hbase.zk_quorum] Value [localhost:5181]
Key [tsd.core.enable_api] Value [true]
Key [tsd.http.staticroot] Value [/usr/share/opentsdb/static/]
Key [tsd.rpc.telnet.return_errors] Value [true]
Key [tsd.query.limits.data_points.default] Value [0]
Key [tsd.storage.enable_appends] Value [false]
Key [tsd.storage.compaction.min_flush_threshold] Value [100]
Key [tsd.network.reuse_address] Value [true]
Key [tsd.query.multi_get.get_all_salts] Value [false]
Key [tsd.rollups.raw_agg_tag_value] Value [RAW]
Key [tsd.http.query.allow_delete] Value [false]
Key [tsd.search.enable] Value [false]
Key [tsd.http.header_tag] Value []
Key [tsd.storage.hbase.uid_table] Value [/tmp/mapr.opentsdb/tsdb-uid]
Key [tsd.storage.hbase.zk_basedir] Value [/hbase]
Key [tsd.storage.compaction.max_concurrent_flushes] Value [10000]
Key [MAPR_TICKETFILE_LOCATION] Value [/opt/mapr/conf/mapruserticket]
Key [tsd.core.auto_create_tagvs] Value [true]
Key [tsd.http.show_stack_trace] Value [true]
Key [tsd.http.request.enable_chunked] Value [false]
Key [tsd.storage.repair_appends] Value [false]
Key [tsd.http.request.cors_headers] Value [Authorization, Content-Type, Accept, Origin, User-Agent, DNT, Cache-Control, X-Mx-ReqToken, Keep-Alive, X-Requested-With, If-Modified-Since]
Key [tsd.core.socket.timeout] Value [0]
Key [tsd.query.allow_simultaneous_duplicates] Value [true]
Key [tsd.stats.canonical] Value [false]
Key [tsd.uid.lru.enable] Value [false]
Key [tsd.network.bind] Value [0.0.0.0]
Key [tsd.rollups.tag_raw] Value [false]
Key [tsd.storage.hbase.scanner.maxNumRows] Value [128]
Key [tsd.network.tcp_no_delay] Value [true]
Key [tsd.mode] Value [rw]
Key [tsd.query.multi_get.enable] Value [false]
Key [tsd.storage.hbase.meta_table] Value [/tmp/mapr.opentsdb/tsdb-meta]
Key [tsd.core.connections.limit] Value [0]
Key [tsd.core.storage_exception_handler.enable] Value [false]
Key [tsd.core.meta.enable_tsuid_tracking] Value [false]
Key [tsd.core.preload_uid_cache] Value [false]
Key [tsd.storage.compaction.flush_interval] Value [10]
15:57:36.687 DEBUG [PluginLoader.loadJARs] - No JAR files found in path: /usr/share/opentsdb/plugins
15:57:36.689 WARN [PluginLoader.loadPlugins] - Unable to locate any plugins of the type: net.opentsdb.query.filter.TagVFilter
15:57:36.695 DEBUG [Configuration.getProperties] - Loaded org.apache.hadoop.conf.CoreDefaultProperties
15:57:36.702 DEBUG [MapRHTable.init] - Init table /tmp/mapr.opentsdb/tsdb
15:57:36.703 DEBUG [MapRLoginHttpsClient.authenticateIfNeeded] - Entering authenticate if needed.
15:57:36.703 DEBUG [MapRLoginHttpsClient.isKerberosAnOption] - Kerberos not configured for this cluster.
15:57:36.703 DEBUG [MapRLoginHttpsClient.doesSecurityHaveGoodKey] - found existing MapR ticket
15:57:36.703 DEBUG [MapRLoginHttpsClient.authenticateIfNeeded] - Already have good ticket, done
15:57:36.723 DEBUG [MapRFileSystem.lookupClient] - User Info object initialized for user mapr with user ID 5000
15:57:36.729 DEBUG [Inode.debug] - >Inode GetAttr: table: /tmp/mapr.opentsdb/tsdb, size: 3, chunksize: 0, fid: 5445.71.131416
15:57:36.740 DEBUG [MapRHTable.refreshSchema] - Updating schema #families 1
15:57:36.740 DEBUG [MapRHTable.refreshSchema] - Caching family name=t, id=1
15:57:36.741 DEBUG [MapRHTable.init] - opened table /tmp/mapr.opentsdb/tsdb clustername maprclsdevisJson false
15:57:36.743 DEBUG [MapRHTable.init] - Init table /tmp/mapr.opentsdb/tsdb-uid
15:57:36.743 DEBUG [MapRFileSystem.lookupClient] - User Info object initialized for user mapr with user ID 5000
15:57:36.744 DEBUG [Inode.debug] - >Inode GetAttr: table: /tmp/mapr.opentsdb/tsdb-uid, size: 3, chunksize: 0, fid: 5445.74.131422
15:57:36.745 DEBUG [MapRHTable.refreshSchema] - Updating schema #families 2
15:57:36.745 DEBUG [MapRHTable.refreshSchema] - Caching family name=id, id=1
15:57:36.745 DEBUG [MapRHTable.refreshSchema] - Caching family name=name, id=2
15:57:36.745 DEBUG [MapRHTable.init] - opened table /tmp/mapr.opentsdb/tsdb-uid clustername maprclsdevisJson false
15:57:36.746 DEBUG [MapRThreadPool.run] - Calling callback for ThreadId: 286, request: Exists(table="/tmp/mapr.opentsdb/tsdb", key=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], families=null, attempt=0, region=null)
15:57:36.746 DEBUG [MapRThreadPool.run] - Calling callback for ThreadId: 287, request: Exists(table="/tmp/mapr.opentsdb/tsdb-uid", key=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], families=null, attempt=0, region=null)
15:57:36.760 INFO [RpcManager.initializeBuiltinRpcs] - Mode: READWRITE, HTTP UI Enabled: true, HTTP API Enabled: true
15:57:36.770 DEBUG [GnuplotInstaller.] - Inspecting PATH for Gnuplot Exe: [/usr/local/bin]
15:57:36.770 DEBUG [GnuplotInstaller.] - Inspecting PATH for Gnuplot Exe: [/bin]
15:57:36.770 INFO [GnuplotInstaller.] - Found gnuplot at [/bin/gnuplot]
15:57:36.770 INFO [GraphHandler.findGnuplotHelperScript] - Auto Installed Gnuplot Invoker at [/tmp/.tsdb/tsdb-gnuplot/mygnuplot.sh]
15:57:36.780 INFO [RpcHandler.] - TSD is in READWRITE mode
15:57:36.780 INFO [RpcHandler.] - CORS domain list was empty, CORS will not be enabled
15:57:36.781 INFO [RpcHandler.] - Loaded CORS headers (Authorization, Content-Type, Accept, Origin, User-Agent, DNT, Cache-Control, X-Mx-ReqToken, Keep-Alive, X-Requested-With, If-Modified-Since)
15:57:36.783 INFO [ConnectionManager.] - TSD concurrent connection limit set to: 0
15:57:36.785 WARN [PluginLoader.loadPlugins] - Unable to locate any plugins of the type: net.opentsdb.tsd.HttpSerializer
15:57:36.790 INFO [TSDMain.main] - Binding on /0.0.0.0:14242
15:57:36.801 INFO [TSDMain.main] - Ready to serve on /0.0.0.0:14242
15:57:36.812 INFO [TSDB.shutdown] - Flushing compaction queue
15:57:36.815 DEBUG [MapRThreadPool.run] - Calling mTable.asyncFlush(), ThreadId: {}
15:57:36.815 DEBUG [MapRThreadPool.run] - Calling mTable.asyncFlush(), ThreadId: {}
15:57:36.815 DEBUG [MapRThreadPool.run] - Calling callback for ThreadId: 289, request: org.hbase.async.MapRThreadPool$AsyncHBaseRpc@7b5916a8
15:57:36.815 DEBUG [MapRThreadPool.run] - Calling callback for ThreadId: 288, request: org.hbase.async.MapRThreadPool$AsyncHBaseRpc@36cbe7aa
15:57:36.816 DEBUG [Deferred.handleContinuation] - callback=shutdown HBase client@1898325501 returned Deferred@2075809815(state=RUNNING, result=[null, null], callback=net.opentsdb.core.TSDB$1FinalShutdown@742ff096 -> (continuation of Deferred@1967355409 after shutdown HBase client@1898325501), errback=net.opentsdb.core.TSDB$1FinalShutdown@742ff096 -> (continuation of Deferred@1967355409 after shutdown HBase client@1898325501)), so the following Deferred is getting paused: Deferred@1967355409(state=PAUSED, result=Deferred@2075809815, callback=, errback=)
15:57:36.816 DEBUG [HBaseClient.call] - Releasing all remaining resources
15:57:36.817 INFO [TSDB.call] - Completed shutting down the TSDB

[manolama]

Which tsd version/commit are you using? L248 points to a null config object in the loadStartupPlugins() method but I doubt thats it. (and tsd.startup.enable is false in your config).