OpenTSDB support for remote storage

[olegeech-me]

Looks like OpenTSDB support in fact is broken for Prometheus 1.4.1.
I'm trying to export metrics to OpenTSDB 2.3.0 but receiving a lot of warnings from both sides.
Prometheus:

./prometheus -config.file=prometheus.yml -storage.remote.opentsdb-url=http://172.20.1.29:4242/api/put/

WARN[0679] error sending 100 samples to remote storage: read tcp 172.20.1.27:44502->172.20.1.29:4242: read: connection reset by peer  source=queue_manager.go:246
WARN[0681] error sending 100 samples to remote storage: read tcp 172.20.1.27:44510->172.20.1.29:4242: read: connection reset by peer  source=queue_manager.go:246
WARN[0681] error sending 100 samples to remote storage: read tcp 172.20.1.27:44512->172.20.1.29:4242: read: connection reset by peer  source=queue_manager.go:246

OpenTSDB:

13:24:31.608 WARN  [HttpQuery.logWarn] - [id: 0xe5b4ea8a, /172.20.1.27:41453 => /172.20.1.29:4242] Bad Request on /api/put: Chunked request not supported.
13:24:31.608 WARN  [HttpQuery.logWarn] - [id: 0xd31743c8, /172.20.1.27:41452 => /172.20.1.29:4242] Bad Request on /api/put: Chunked request not supported.
13:24:31.608 INFO  [ConnectionManager.handleUpstream] - [id: 0xe5b4ea8a, /172.20.1.27:41453 :> /172.20.1.29:4242] DISCONNECTED
13:24:31.608 INFO  [ConnectionManager.handleUpstream] - [id: 0xd31743c8, /172.20.1.27:41452 :> /172.20.1.29:4242] DISCONNECTED
13:24:31.608 ERROR [RpcHandler.logError] - [id: 0xe5b4ea8a, /172.20.1.27:41453 :> /172.20.1.29:4242] Unexpected message type class org.jboss.netty.handler.codec.http.DefaultHttpChunk: org.jboss.netty.handler.codec.http.DefaultHttpChunk@5bd45d9c
13:24:31.608 ERROR [RpcHandler.logError] - [id: 0xd31743c8, /172.20.1.27:41452 :> /172.20.1.29:4242] Unexpected message type class org.jboss.netty.handler.codec.http.DefaultHttpChunk: org.jboss.netty.handler.codec.http.DefaultHttpChunk@42ab5c93
13:24:31.608 INFO  [ConnectionManager.handleUpstream] - [id: 0xe5b4ea8a, /172.20.1.27:41453 :> /172.20.1.29:4242] UNBOUND
13:24:31.609 INFO  [ConnectionManager.handleUpstream] - [id: 0xd31743c8, /172.20.1.27:41452 :> /172.20.1.29:4242] UNBOUND
13:24:31.609 INFO  [ConnectionManager.handleUpstream] - [id: 0xe5b4ea8a, /172.20.1.27:41453 :> /172.20.1.29:4242] CLOSED
13:24:31.609 INFO  [ConnectionManager.handleUpstream] - [id: 0xd31743c8, /172.20.1.27:41452 :> /172.20.1.29:4242] CLOSED
13:24:31.609 INFO  [HttpQuery.logInfo] - [id: 0xe5b4ea8a, /172.20.1.27:41453 :> /172.20.1.29:4242] HTTP /api/put done in 1ms
13:24:31.609 INFO  [HttpQuery.logInfo] - [id: 0xd31743c8, /172.20.1.27:41452 :> /172.20.1.29:4242] HTTP /api/put done in 1ms

I can also see some metrics are partially appearing in OpenTSDB with tsdb uid grep etcd but not the whole list.

As I understand there is going to be no further support for this kind of exporter and I should instead expect the new generic exporter for influx/opentsdb/graphite altogether in the forthcoming release.
Is my understanding right? :)

[olegeech-me]

Prometheus config:

global:
  scrape_interval:     15s 
  evaluation_interval: 15s 
  external_labels:
      monitor: 'codelab-monitor'
rule_files:
  # - "first.rules"
  # - "second.rules"

scrape_configs:

  - job_name: 'etcd-env-4'
    scrape_interval: 5s
    scrape_timeout: 5s
    scheme: https
    tls_config:
       insecure_skip_verify: true
    file_sd_configs:
       - files:
          - etcd-env-4.yml

etcd-env-4.yml:

- targets:
  - 172.20.9.20:2379
  - 172.20.9.15:2379
  - 172.20.9.16:2379
  labels:
    env: env-4

OpenTSDB backended with HBASE:

tsd.network.port = 4242
tsd.network.bind = 0.0.0.0
tsd.network.tcp_no_delay = true
tsd.network.keep_alive = true
tsd.network.reuse_address = true
tsd.network.worker_threads = 8
tsd.network.async_io = true
tsd.http.staticroot = /usr/share/opentsdb/static/
tsd.http.cachedir = /tmp/opentsdb
tsd.core.auto_create_metrics = true
tsd.core.plugin_path = /usr/share/opentsdb/plugins
tsd.storage.hbase.data_table = tsdb
tsd.storage.hbase.zk_basedir = /hbase
tsd.storage.hbase.zk_quorum = localhost

[brian-brazil]

I should instead expect the new generic exporter for influx/opentsdb/graphite altogether in the forthcoming release.

Yes, but the code will still hang around as an example.

This smells like a network issue to me, so I'd suggest discussing on https://groups.google.com/forum/#!forum/prometheus-users

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.