Prometheus failed sending data to openTSDB

[reAsOn2010]

I'm using prometheus with remote storage of openTSDB, but prometheus failed to send metrics.

prometheus version 1.2.0
openTSDB version 2.2.0

prometheus log:

time="2016-10-11T07:43:46Z" level=warning msg="error sending 100 samples to remote storage: Post http://opentsdb-1.default.svc:4242/api/put: read tcp 192.168.75.6:32806->192.168.71.7:4242: i/o timeout" source="queue_manager.go:246" 
time="2016-10-11T07:43:46Z" level=warning msg="error sending 100 samples to remote storage: Post http://opentsdb-1.default.svc:4242/api/put: read tcp 192.168.75.6:32807->192.168.71.7:4242: i/o timeout" source="queue_manager.go:246" 
time="2016-10-11T07:43:46Z" level=warning msg="error sending 100 samples to remote storage: Post http://opentsdb-1.default.svc:4242/api/put: read tcp 192.168.75.6:32808->192.168.71.7:4242: i/o timeout" source="queue_manager.go:246" 
time="2016-10-11T07:43:46Z" level=warning msg="error sending 100 samples to remote storage: Post http://opentsdb-1.default.svc:4242/api/put: read tcp 192.168.75.6:32809->192.168.71.7:4242: i/o timeout" source="queue_manager.go:246"
time="2016-10-11T07:43:48Z" level=warning msg="cannot send value NaN to OpenTSDB, skipping sample &model.Sample{Metric:model.Metric{\"operation_type\":\"runContainerInPod\", \"quantile\":\"0.5\", \"__name__\":\"kubelet_container_manager_latency_microseconds\", \"job\":\"kubernetes-nodes\", \"beta_kubernetes_io_os\":\"linux\", \"kubernetes_io_hostname\":\"kube-node-1\", \"beta_kubernetes_io_arch\":\"amd64\", \"instance\":\"kube-node-1\"}, Value:NaN, Timestamp:1476171826102}" source="client.go:79"

openTSDB log:

2016-10-11 07:43:46,756 INFO  [OpenTSDB I/O Worker #3] ConnectionManager: [id: 0x8512e244, /192.168.75.6:32803 :> /192.168.71.7:4242] CLOSED
2016-10-11 07:43:46,915 INFO  [OpenTSDB I/O Boss #1] ConnectionManager: [id: 0x17ba8b25, /192.168.75.6:32806 => /192.168.71.7:4242] OPEN
2016-10-11 07:43:46,915 INFO  [OpenTSDB I/O Boss #1] ConnectionManager: [id: 0x2c9fbab5, /192.168.75.6:32807 => /192.168.71.7:4242] OPEN
2016-10-11 07:43:46,915 INFO  [OpenTSDB I/O Worker #5] ConnectionManager: [id: 0x17ba8b25, /192.168.75.6:32806 => /192.168.71.7:4242] BOUND: /192.168.71.7:4242
2016-10-11 07:43:46,916 INFO  [OpenTSDB I/O Worker #5] ConnectionManager: [id: 0x17ba8b25, /192.168.75.6:32806 => /192.168.71.7:4242] CONNECTED: /192.168.75.6:32806
2016-10-11 07:43:46,917 INFO  [OpenTSDB I/O Worker #7] ConnectionManager: [id: 0x2c9fbab5, /192.168.75.6:32807 => /192.168.71.7:4242] BOUND: /192.168.71.7:4242
2016-10-11 07:43:46,920 INFO  [OpenTSDB I/O Worker #5] ConnectionManager: [id: 0x17ba8b25, /192.168.75.6:32806 :> /192.168.71.7:4242] DISCONNECTED
2016-10-11 07:43:46,929 INFO  [OpenTSDB I/O Worker #7] ConnectionManager: [id: 0x2c9fbab5, /192.168.75.6:32807 => /192.168.71.7:4242] CONNECTED: /192.168.75.6:32807
2016-10-11 07:43:46,932 INFO  [OpenTSDB I/O Worker #7] ConnectionManager: [id: 0x2c9fbab5, /192.168.75.6:32807 :> /192.168.71.7:4242] DISCONNECTED
2016-10-11 07:43:46,933 INFO  [OpenTSDB I/O Worker #7] ConnectionManager: [id: 0x2c9fbab5, /192.168.75.6:32807 :> /192.168.71.7:4242] UNBOUND
2016-10-11 07:43:46,935 INFO  [OpenTSDB I/O Worker #7] ConnectionManager: [id: 0x2c9fbab5, /192.168.75.6:32807 :> /192.168.71.7:4242] CLOSED
2016-10-11 07:43:46,936 INFO  [OpenTSDB I/O Worker #5] ConnectionManager: [id: 0x17ba8b25, /192.168.75.6:32806 :> /192.168.71.7:4242] UNBOUND
2016-10-11 07:43:46,937 INFO  [OpenTSDB I/O Worker #5] ConnectionManager: [id: 0x17ba8b25, /192.168.75.6:32806 :> /192.168.71.7:4242] CLOSED
2016-10-11 07:43:46,956 INFO  [OpenTSDB I/O Boss #1] ConnectionManager: [id: 0x4175e3a6, /192.168.75.6:32808 => /192.168.71.7:4242] OPEN
2016-10-11 07:43:46,962 INFO  [OpenTSDB I/O Boss #1] ConnectionManager: [id: 0xe9fc750b, /192.168.75.6:32809 => /192.168.71.7:4242] OPEN
2016-10-11 07:43:46,964 INFO  [OpenTSDB I/O Worker #11] ConnectionManager: [id: 0xe9fc750b, /192.168.75.6:32809 => /192.168.71.7:4242] BOUND: /192.168.71.7:4242
2016-10-11 07:43:46,964 INFO  [OpenTSDB I/O Worker #11] ConnectionManager: [id: 0xe9fc750b, /192.168.75.6:32809 => /192.168.71.7:4242] CONNECTED: /192.168.75.6:32809
2016-10-11 07:43:46,962 INFO  [OpenTSDB I/O Worker #6] ConnectionManager: [id: 0x4175e3a6, /192.168.75.6:32808 => /192.168.71.7:4242] BOUND: /192.168.71.7:4242
2016-10-11 07:43:46,968 INFO  [OpenTSDB I/O Worker #6] ConnectionManager: [id: 0x4175e3a6, /192.168.75.6:32808 => /192.168.71.7:4242] CONNECTED: /192.168.75.6:32808
2016-10-11 07:43:46,973 INFO  [OpenTSDB I/O Worker #11] ConnectionManager: [id: 0xe9fc750b, /192.168.75.6:32809 :> /192.168.71.7:4242] DISCONNECTED
2016-10-11 07:43:46,974 INFO  [OpenTSDB I/O Worker #11] ConnectionManager: [id: 0xe9fc750b, /192.168.75.6:32809 :> /192.168.71.7:4242] UNBOUND
2016-10-11 07:43:46,980 INFO  [OpenTSDB I/O Worker #11] ConnectionManager: [id: 0xe9fc750b, /192.168.75.6:32809 :> /192.168.71.7:4242] CLOSED
2016-10-11 07:43:46,980 INFO  [OpenTSDB I/O Worker #6] ConnectionManager: [id: 0x4175e3a6, /192.168.75.6:32808 :> /192.168.71.7:4242] DISCONNECTED
2016-10-11 07:43:46,998 INFO  [OpenTSDB I/O Worker #6] ConnectionManager: [id: 0x4175e3a6, /192.168.75.6:32808 :> /192.168.71.7:4242] UNBOUND
2016-10-11 07:43:46,998 INFO  [OpenTSDB I/O Worker #6] ConnectionManager: [id: 0x4175e3a6, /192.168.75.6:32808 :> /192.168.71.7:4242] CLOSED
2016-10-11 07:43:49,000 INFO  [OpenTSDB I/O Boss #1] ConnectionManager: [id: 0xd43b2356, /192.168.75.6:32812 => /192.168.71.7:4242] OPEN
2016-10-11 07:43:49,002 INFO  [OpenTSDB I/O Boss #1] ConnectionManager: [id: 0xa163f990, /192.168.75.6:32813 => /192.168.71.7:4242] OPEN

Comparing the date time of the log, the two parts of log is corresponding. openTSDB said the connection was closed shortly after connection open, but prometheus said it was timed out...

They all running in k8s. But my partner @harryge00 reproduced this problem locally on his PC.

[harryge00]

prometheus version 1.1.0 can work locally.

[brian-brazil]

This smells like a network issue on your end. Does sending data by hand work?

[reAsOn2010]

No network issue, we just downgraded prometheus's version to 1.1.x, everything went well~

But I noticed that in 1.2.0 the command line config storage.remote.timeout was removed, and the log said that prometheus closed the connection self throwing timeout exceptions. Is the timeout configuration not properly applied?

[brian-brazil]

That would be a regression from #1957 alright. Not that this code is going to kicked out to separate binaries soon anyway.

[reAsOn2010]

@brian-brazil
so the command line config seems to be deprecated in the future, where could I find some examples configuring remote storage using yaml config file? I found no docs about this...

[brian-brazil]

https://www.robustperception.io/using-the-remote-write-path/

The existing 3 remote storage integrations will have their code moved out somewhere so you can use them with this.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.