Prometheus failed sending data to openTSDB

[shenshouer]

I'm using prometheus with remote storage of openTSDB, but prometheus failed to send metrics.

prometheus version of docker image : prom/prometheus:v1.5.2
openTSDB version: opentsdb-2.3.0_all

remote-storage-adapter logs:

...
time="2017-04-19T06:09:29Z" level=warning msg="cannot send value NaN to OpenTSDB, skipping sample &model.Sample{Metric:model.Metric{\"resource\":\"rolebindings\", \"verb\":\"POST\", \"quantile\":\"0.9\", \"__name__\":\"apiserver_request_latencies_summary\", \"job\":\"kubernetes-apiservers\", \"instance\":\"172.21.9.101:6443\"}, Value:NaN, Timestamp:1492582166600}" source="client.go:78"
time="2017-04-19T06:09:29Z" level=warning msg="cannot send value NaN to OpenTSDB, skipping sample &model.Sample{Metric:model.Metric{\"verb\":\"POST\", \"quantile\":\"0.99\", \"__name__\":\"apiserver_request_latencies_summary\", \"job\":\"kubernetes-apiservers\", \"instance\":\"172.21.9.101:6443\", \"resource\":\"roles\"}, Value:NaN, Timestamp:1492582166600}" source="client.go:78"
time="2017-04-19T06:09:29Z" level=warning msg="cannot send value NaN to OpenTSDB, skipping sample &model.Sample{Metric:model.Metric{\"instance\":\"172.21.9.101:6443\", \"resource\":\"serviceaccounts\", \"verb\":\"POST\", \"quantile\":\"0.99\", \"__name__\":\"apiserver_request_latencies_summary\", \"job\":\"kubernetes-apiservers\"}, Value:NaN, Timestamp:1492582166600}" source="client.go:78"
time="2017-04-19T06:09:29Z" level=warning msg="cannot send value NaN to OpenTSDB, skipping sample &model.Sample{Metric:model.Metric{\"quantile\":\"0.5\", \"__name__\":\"apiserver_request_latencies_summary\", \"instance\":\"172.21.9.101:6443\", \"job\":\"kubernetes-apiservers\", \"resource\":\"serviceaccounts\", \"verb\":\"PUT\"}, Value:NaN, Timestamp:1492582166600}" source="client.go:78"
time="2017-04-19T06:09:29Z" level=warning msg="cannot send value NaN to OpenTSDB, skipping sample &model.Sample{Metric:model.Metric{\"job\":\"kubernetes-apiservers\", \"instance\":\"172.21.9.101:6443\", \"quantile\":\"0.5\", \"__name__\":\"etcd_request_cache_get_latencies_summary\"}, Value:NaN, Timestamp:1492582166600}" source="client.go:78"
time="2017-04-19T06:09:29Z" level=warning msg="Error sending samples to remote storage" err="invalid character '<' looking for beginning of value" num_samples=100 source="main.go:282" storage=opentsdb
time="2017-04-19T06:09:29Z" level=warning msg="Error sending samples to remote storage" err="invalid character '<' looking for beginning of value" num_samples=100 source="main.go:282" storage=opentsdb
time="2017-04-19T06:09:29Z" level=warning msg="Error sending samples to remote storage" err="invalid character '<' looking for beginning of value" num_samples=100 source="main.go:282" storage=opentsdb
time="2017-04-19T06:09:29Z" level=warning msg="Error sending samples to remote storage" err="invalid character '<' looking for beginning of value" num_samples=100 source="main.go:282" storage=opentsdb
time="2017-04-19T06:09:29Z" level=warning msg="Error sending samples to remote storage" err="invalid character '<' looking for beginning of value" num_samples=100 source="main.go:282" storage=opentsdb
time="2017-04-19T06:09:29Z" level=warning msg="Error sending samples to remote storage" err="invalid character '<' looking for beginning of value" num_samples=100 source="main.go:282" storage=opentsdb
time="2017-04-19T06:09:29Z" level=warning msg="Error sending samples to remote storage" err="invalid character '<' looking for beginning of value" num_samples=100 source="main.go:282" storage=opentsdb
time="2017-04-19T06:09:29Z" level=warning msg="Error sending samples to remote storage" err="invalid character '<' looking for beginning of value" num_samples=100 source="main.go:282" storage=opentsdb
time="2017-04-19T06:09:29Z" level=warning msg="Error sending samples to remote storage" err="invalid character '<' looking for beginning of value" num_samples=100 source="main.go:282" storage=opentsdb
time="2017-04-19T06:09:29Z" level=warning msg="Error sending samples to remote storage" err="invalid character '<' looking for beginning of value" num_samples=100 source="main.go:282" storage=opentsdb
time="2017-04-19T06:09:29Z" level=warning msg="Error sending samples to remote storage" err="invalid character '<' looking for beginning of value" num_samples=100 source="main.go:282" storage=opentsdb
time="2017-04-19T06:09:29Z" level=warning msg="cannot send value NaN to OpenTSDB, skipping sample &model.Sample{Metric:model.Metric{\"quantile\":\"0.5\", \"beta_kubernetes_io_os\":\"linux\", \"job\":\"kubernetes-nodes\", \"beta_kubernetes_io_arch\":\"amd64\", \"kubernetes_io_hostname\":\"k8s-02\", \"operation_type\":\"pull_image\", \"__name__\":\"kubelet_docker_operations_latency_microseconds\", \"instance\":\"k8s-02\"}, Value:NaN, Timestamp:1492582169379}" source="client.go:78"
time="2017-04-19T06:09:29Z" level=warning msg="cannot send value NaN to OpenTSDB, skipping sample &model.Sample{Metric:model.Metric{\"beta_kubernetes_io_arch\":\"amd64\", \"instance\":\"k8s-02\", \"beta_kubernetes_io_os\":\"linux\", \"job\":\"kubernetes-nodes\", \"__name__\":\"kubelet_docker_operations_latency_microseconds\", \"kubernetes_io_hostname\":\"k8s-02\", \"operation_type\":\"remove_container\", \"quantile\":\"0.99\"}, Value:NaN, Timestamp:1492582169379}" source="client.go:78"
time="2017-04-19T06:09:29Z" level=warning msg="cannot send value NaN to OpenTSDB, skipping sample &model.Sample{Metric:model.Metric{\"__name__\":\"kubelet_docker_operations_latency_microseconds\", \"beta_kubernetes_io_os\":\"linux\", \"beta_kubernetes_io_arch\":\"amd64\", \"instance\":\"k8s-02\", \"kubernetes_io_hostname\":\"k8s-02\", \"operation_type\":\"start_container\", \"quantile\":\"0.99\", \"job\":\"kubernetes-nodes\"}, Value:NaN, Timestamp:1492582169379}" source="client.go:78"
...

prometheus logs:

...
time="2017-04-19T06:02:14Z" level=warning msg="Remote storage queue full, discarding sample." source="queue_manager.go:169"
time="2017-04-19T06:02:14Z" level=warning msg="Remote storage queue full, discarding sample." source="queue_manager.go:169"
time="2017-04-19T06:02:14Z" level=warning msg="Remote storage queue full, discarding sample." source="queue_manager.go:169"
time="2017-04-19T06:02:14Z" level=warning msg="Remote storage queue full, discarding sample." source="queue_manager.go:169"
time="2017-04-19T06:02:14Z" level=warning msg="Remote storage queue full, discarding sample." source="queue_manager.go:169"
time="2017-04-19T06:02:14Z" level=warning msg="Remote storage queue full, discarding sample." source="queue_manager.go:169"
time="2017-04-19T06:02:14Z" level=warning msg="Remote storage queue full, discarding sample." source="queue_manager.go:169"
time="2017-04-19T06:02:14Z" level=warning msg="Remote storage queue full, discarding sample." source="queue_manager.go:169"
time="2017-04-19T06:02:14Z" level=warning msg="Remote storage queue full, discarding sample." source="queue_manager.go:169"
time="2017-04-19T06:02:14Z" level=warning msg="Remote storage queue full, discarding sample." source="queue_manager.go:169"
time="2017-04-19T06:02:14Z" level=warning msg="Remote storage queue full, discarding sample." source="queue_manager.go:169"
time="2017-04-19T06:02:14Z" level=warning msg="Remote storage queue full, discarding sample." source="queue_manager.go:169"
...

openTSDB log:

...
06:15:29.617 WARN  [HttpQuery.logWarn] - [id: 0xcd3945b6, /172.21.9.102:54414 => /172.21.9.101:4242] Bad Request on /api/put: Chunked request not supported.
06:15:29.618 INFO  [HttpQuery.logInfo] - [id: 0xcd3945b6, /172.21.9.102:54414 => /172.21.9.101:4242] HTTP /api/put done in 0ms
06:15:29.618 ERROR [RpcHandler.logError] - [id: 0xcd3945b6, /172.21.9.102:54414 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.DefaultHttpChunk: org.jboss.netty.handler.codec.http.DefaultHttpChunk@6391760c
06:15:29.619 ERROR [RpcHandler.logError] - [id: 0xcd3945b6, /172.21.9.102:54414 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.DefaultHttpChunk: org.jboss.netty.handler.codec.http.DefaultHttpChunk@66848bb2
06:15:29.619 ERROR [RpcHandler.logError] - [id: 0xee1a3c88, /172.21.9.102:54404 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.DefaultHttpChunk: org.jboss.netty.handler.codec.http.DefaultHttpChunk@9f6b866
06:15:29.619 ERROR [RpcHandler.logError] - [id: 0xee1a3c88, /172.21.9.102:54404 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.DefaultHttpChunk: org.jboss.netty.handler.codec.http.DefaultHttpChunk@27c0ac2f
06:15:29.620 ERROR [RpcHandler.logError] - [id: 0xca9271c8, /172.21.9.102:54380 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.DefaultHttpChunk: org.jboss.netty.handler.codec.http.DefaultHttpChunk@4f099fcd
06:15:29.620 ERROR [RpcHandler.logError] - [id: 0xca9271c8, /172.21.9.102:54380 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.DefaultHttpChunk: org.jboss.netty.handler.codec.http.DefaultHttpChunk@d924485
06:15:29.620 ERROR [RpcHandler.logError] - [id: 0xca9271c8, /172.21.9.102:54380 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.DefaultHttpChunk: org.jboss.netty.handler.codec.http.DefaultHttpChunk@311be8d9
06:15:29.621 ERROR [RpcHandler.logError] - [id: 0xca9271c8, /172.21.9.102:54380 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.HttpChunk$1: org.jboss.netty.handler.codec.http.HttpChunk$1@1893411c
06:15:29.621 ERROR [RpcHandler.logError] - [id: 0xcd3945b6, /172.21.9.102:54414 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.DefaultHttpChunk: org.jboss.netty.handler.codec.http.DefaultHttpChunk@13989b01
06:15:29.621 ERROR [RpcHandler.logError] - [id: 0xcd3945b6, /172.21.9.102:54414 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.DefaultHttpChunk: org.jboss.netty.handler.codec.http.DefaultHttpChunk@10ec9948
06:15:29.622 ERROR [RpcHandler.logError] - [id: 0xcd3945b6, /172.21.9.102:54414 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.DefaultHttpChunk: org.jboss.netty.handler.codec.http.DefaultHttpChunk@f4551bc
06:15:29.622 ERROR [RpcHandler.logError] - [id: 0xcd3945b6, /172.21.9.102:54414 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.HttpChunk$1: org.jboss.netty.handler.codec.http.HttpChunk$1@1893411c
06:15:29.622 INFO  [ConnectionManager.handleUpstream] - [id: 0xcd3945b6, /172.21.9.102:54414 :> /172.21.9.101:4242] DISCONNECTED
06:15:29.623 INFO  [ConnectionManager.handleUpstream] - [id: 0xcd3945b6, /172.21.9.102:54414 :> /172.21.9.101:4242] UNBOUND
06:15:29.623 INFO  [ConnectionManager.handleUpstream] - [id: 0xcd3945b6, /172.21.9.102:54414 :> /172.21.9.101:4242] CLOSED
06:15:29.624 ERROR [RpcHandler.logError] - [id: 0xee1a3c88, /172.21.9.102:54404 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.DefaultHttpChunk: org.jboss.netty.handler.codec.http.DefaultHttpChunk@61f63dad
06:15:29.624 ERROR [RpcHandler.logError] - [id: 0xee1a3c88, /172.21.9.102:54404 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.DefaultHttpChunk: org.jboss.netty.handler.codec.http.DefaultHttpChunk@3661e201
06:15:29.624 ERROR [RpcHandler.logError] - [id: 0xee1a3c88, /172.21.9.102:54404 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.HttpChunk$1: org.jboss.netty.handler.codec.http.HttpChunk$1@1893411c
06:15:29.625 ERROR [RpcHandler.logError] - [id: 0xee1a3c88, /172.21.9.102:54404 => /172.21.9.101:4242] Received an unsupported chunked request: DefaultHttpRequest(chunked: true)
POST /api/put HTTP/1.1
Host: 172.21.9.101:4242
User-Agent: Go-http-client/1.1
Content-Length: 31777
Content-Type: application/json
Accept-Encoding: gzip
06:15:29.625 WARN  [HttpQuery.logWarn] - [id: 0xee1a3c88, /172.21.9.102:54404 => /172.21.9.101:4242] Bad Request on /api/put: Chunked request not supported.
06:15:29.626 INFO  [HttpQuery.logInfo] - [id: 0xee1a3c88, /172.21.9.102:54404 => /172.21.9.101:4242] HTTP /api/put done in 1ms
06:15:29.626 ERROR [RpcHandler.logError] - [id: 0xee1a3c88, /172.21.9.102:54404 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.DefaultHttpChunk: org.jboss.netty.handler.codec.http.DefaultHttpChunk@42b30b5c
06:15:29.628 ERROR [RpcHandler.logError] - [id: 0xee1a3c88, /172.21.9.102:54404 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.DefaultHttpChunk: org.jboss.netty.handler.codec.http.DefaultHttpChunk@67de7a07
06:15:29.629 ERROR [RpcHandler.logError] - [id: 0xee1a3c88, /172.21.9.102:54404 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.DefaultHttpChunk: org.jboss.netty.handler.codec.http.DefaultHttpChunk@37152804
06:15:29.630 ERROR [RpcHandler.logError] - [id: 0xee1a3c88, /172.21.9.102:54404 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.DefaultHttpChunk: org.jboss.netty.handler.codec.http.DefaultHttpChunk@eddb50d
06:15:29.631 ERROR [RpcHandler.logError] - [id: 0xee1a3c88, /172.21.9.102:54404 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.DefaultHttpChunk: org.jboss.netty.handler.codec.http.DefaultHttpChunk@54eea661
06:15:29.631 ERROR [RpcHandler.logError] - [id: 0xee1a3c88, /172.21.9.102:54404 => /172.21.9.101:4242] Unexpected message type class org.jboss.netty.handler.codec.http.HttpChunk$1: org.jboss.netty.handler.codec.http.HttpChunk$1@1893411c
06:15:29.633 INFO  [ConnectionManager.handleUpstream] - [id: 0xee1a3c88, /172.21.9.102:54404 :> /172.21.9.101:4242] DISCONNECTED
06:15:29.633 INFO  [ConnectionManager.handleUpstream] - [id: 0xee1a3c88, /172.21.9.102:54404 :> /172.21.9.101:4242] UNBOUND
06:15:29.633 INFO  [ConnectionManager.handleUpstream] - [id: 0xee1a3c88, /172.21.9.102:54404 :> /172.21.9.101:4242] CLOSED
...

prometheus and remote-storage-adapter running in k8s, and openTSDB running in standalone server.

[ghost]

Is this potentially a similar issue i asked about here?

i.e., ultimately issue with how opentsdb handles NaN:
OpenTSDB/opentsdb#183

[shenshouer]

I know the NaN value is come from type SampleValue float64, But why is NaN not the 0 ?

[shenshouer]

And No data in hbase, because I lose the tsd.core.meta.enable_realtime_ts = true in OpenTSDB config file. I think this issue can be close

[NikitaGl]

I've encountered the same issue. Enabling tsd.core.meta.enable_realtime_ts is not helping.
My env is:

prometheus, version 2.2.1 (branch: HEAD, revision: bc6058c81272a8d938c05e75607371284236aadc)
  build user:       root@149e5b3f0829
  build date:       20180314-14:15:45
  go version:       go1.10

OpenTSDB version [2.3.1] built from revision in a MODIFIED state
Built on Tue May 22 20:08:21 GMT+300 2018 by root@centos.localhost:/root/rpmbuild/BUILD/opentsdb-2.3.1

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.