Querying returns HTTP 400 Bad Request

[simonszu]

What did you do?
Installed prometheus with node_exporter scraper. Tried to fetch the node_cpu metrics first, any other metrics after that. I have selected node_cpu on the dropdown menu on the Graph tab, and clicked "Execute".

What did you expect to see?
Some numbers representing actual values for this metrics.

What did you see instead? Under which circumstances?
A red error message, Error executing query: server returned HTTP status 400 Bad Request. So i looked into the access.log of the nginx instance i am reverse proxying prometheus with, and discovered that the API call looked like this: GET /api/v1/query_range?query=node_memory_MemFree%7Binstance%3D~%27calad%27%7D&start=1495440400&end=1496045200&step=600 HTTP/1.1

So i tried to access this API endpoint manually and all i got was a result {"status":"error","errorType":"execution","error":"server returned HTTP status 400 Bad Request"}

Maybe Prometheus creates faulty request strings for its own API? I don't know.

Environment

• System information:

  Linux 4.4.50-v7+ armv7l
  (Raspberry Pi 3, with Raspbian installed)

• Prometheus version:

  prometheus, version 1.6.3 (branch: master, revision: c580b60)
  build user: root@a6410e65f5c7
  build date: 20170522-09:41:06
  go version: go1.8.1

• Prometheus configuration file:
  https://pastebin.com/uHWYiZ6E

• Logs:
  I am starting prometheus via systemd. Loglevel is set to "debug". The output of journalctl does not show any log entries relevant to this issue.

[simonszu]

So i have tracked it down to the remote_storage_adapter i am trying to use to connect prometheus with influxDB. Seems that this application does not work yet. I have compiled it, installed it, and added it to my prometheus config as you can see.
However, i have zero hits on the public facing InfluxDB endpoint the adapter is pointing to, so it does not deliver any data, so prometheus does interpret it as a invalid request.
However, it behaves like it does accept data prometheus wants to write into the InfluxDB - stopping it causes prometheus to spawn lots of error messages, if it has the adapter configured as remote_write. The public facing InfluxDB web endpoint does not receive any requests, though.

So with InfluxDB basically removed from prometheus core and the adapter not really working, i currently have no real possibility to connect prometheus and influxdb, right?

[juliusv]

However, i have zero hits on the public facing InfluxDB endpoint the adapter is pointing to, so it does not deliver any data, so prometheus does interpret it as a invalid request.

Hm, I'm not sure I understand where you mean the error exactly occurs. If a given expression selects no series on the InfluxDB side and the InfluxDB response is empty, this should not result in an error.

Using the same remote_read and remote_write sections as you have in your config and running both Prometheus and the adapter built from current master works for me when I select a non-existent metric name.

Is InfluxDB throwing an error? Which version of InfluxDB are you using? I'm on 1.2.2.

[brian-brazil]

Could this be a protocol mismatch due to the compression changes?

[juliusv]

Oh. Yeah, this is likely if the adapter was built from HEAD. Prometheus 1.6.3 does not include the changed Snappy compression yet. @simonszu Could you try either building the InfluxDB adapter from the v1.6.3 git tag again, or build both Prometheus and the adapter from the current master?

[simonszu]

Ah, okay. I'll try. Yes, i compiled the adapter from HEAD this morning.

[simonszu]

Okay, after using the binary i have compiled from v1.6.3 and completely dropping the old prometheus InfluxDB and recreating it it works like a charm now. Thanks guys.

(I got lots of HTTP 500s first, but i vaguely remembered that InfluxDB had changed something with their automatic retention from default to autogen, and since the old data wasn't important for me anymore, it was no big deal).

However, is it planned to release pre-compiled binaries for the adapter in the future? I hope so.

[juliusv]

However, is it planned to release pre-compiled binaries for the adapter in the future? I hope so.

Yes, once the adapter is more than just an example in the documentation directory. Probably it should be a separate repo at some point.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.