influxdb & grafana, absence of data on the graphs

[oshamin]

System info:
OS: CentOS Linux release 7.3.1611 (Core)
influxdb: influxdb-1.2.0-1.x86_64 (https acccess)
grafana: grafana-4.1.2-1486989747.x86_64

Steps to reproduce:
I faced the problem, that grafana shows always "InfluxDB Error: undefined" after influxdb works for awhile.
Here is how it is presented, you can see exclamation mark on the upper left side with error (InfluxDB Error: undefined).

Dashboard loads

Waited ~2 seconds and clicked on refresh

So after clicking refresh some graphs show data and some not.
The InfluxQL queries are registered in logs without errors.
But in grafana logs some queries have response code 200 and some have 502, when queries are proxied via grafana.

Additional info:
If I only open one graph, then response is always successful.
Moreover the problem disappears after influxdb restarted and then appears after 5-10 minutes again.

my config is here

What can be a cause of such inconsistent behavior?

[jwlighting]

I had the same problem discribed above with OS Ubuntu 16.10 after the update from InfluxDB version 1.1 to 1.2. As a solution I downgraded to 1.1 again, which is working without problems
Let me know if I can provide you any information or support to solve this.

[jwilder]

A 502 might indicate that the proxy timed out during the query. Have you tried switch to querying influxdb directly? Also, are there any errors in the influxdb logs associated with those queries? What are the actual queries you are running that are returning 502 from grafana?

[oshamin]

@jwlighting my running influxdb instance was also updated from 1.1 to 1.2 version.

@jwilder yes, I have tested both direct and proxy access to influxdb.
The difference is that with proxy access 502 errors are recorded in grafana logs and in browser console.

And with direct access they are only in browser console.

[jwilder]

@oshamin Are there errors in the influxdb logs? Can you include the logs for those queries?

[oshamin]

@jwilder It looks like not all queries come to influxdb. I made video.
The dashboard had 6 graphs which were build by 9 queries, but influxdb only logged 1-3 query per refresh.

Then there are two possibilities, either queries are not logged properly or it is grafana bug.

How do you think should I create issue in grafana tracker?

[jwilder]

@oshamin Can you enable HTTP logging on influxdb?

[http]
  log-enabled = true

[oshamin]

here is log.

There is only one http request per refresh.

[jwilder]

@oshamin Try w/o the proxy config so that grafana is not in the middle of the browser and the databases. If there is no HTTP log for the failed queries, then the query is not reaching influxdb for some reason.

[oshamin]

@jwilder the result is the same either via proxy or direct access. And not all http queries are logged.
But after influxdb restart everything works smoothly about 1-5 minutes and then the problem appears again.

I also checked connections with tcpdump. After refreshing page, tcpdump registers more then one new TCP SYNC. And here is connection view from browser side.

[graphex]

I'm seeing the same sorts of things happening with 1.2 and grafana. In one graph I can see up to 7 lines (each one has a different tag value), but no more. When I look at the raw json being returned, I see partial: true where it stops. Is this a new 'feature' that grafana doesn't know how to deal with? Is it configurable?
Edit: my issue appears to be the max-row-limit configuration. Not a fan of the low 10000 default...

[oshamin]

@jwilder console client also does not work.

$ influx -precision rfc3339
Failed to connect to http://localhost:8086: Get http://localhost:8086/ping: EOF
Please check your connection settings and ensure 'influxd' is running.

$curl -v http://localhost:8086/ping
* About to connect() to localhost port 8086 (#0)
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8086 (#0)
> GET /ping HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:8086
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer


$ ps aux|grep influxdb
influxdb 30356  6.5  2.6 497440 157016 ?       Ssl  01:05   0:40 /usr/bin/influxd -config /etc/influxdb/influxdb.conf

and in log file only write requests are logged (POST).
After restart it works for awhile and the problem repeats again.

How can I help to find a cause of the problem, what information do you need?

[gerrickw]

Same problem. Before I stopped using 1.2.0 I noticed the same issue, but didn't research too much as had to revert due to '_' issue and 1.1.4 started working again. Would like to move to 1.2.1 so here are some repro steps.

Steps to reproduce:

1. Install Grafana
2. Install influxdb 1.1.4
3. Setup graph in grafana with the following query:
   SELECT max(value) FROM test_db.autogen./^f$/ WHERE pool =~ /^hhh_iii$/ AND colo =~ /^colob$/ AND time > now() - 1h GROUP BY time(10s), y, pool
4. Download python gist I made for another influxdb issue a while back. https://gist.github.com/gerrickw/f83fb4d4d69aef2dfd37/6975818e9d6d3ef4ce248f57891ecb112f33edd0
5. pip install influxdb
6. Execute influx

7. CREATE DATABASE "test_db"

8. CREATE USER "test_db" WITH PASSWORD 'test_db'

9. grant read on test_db to test_db

10. grant write on test_db to test_db

11. python simple_influx_writer.py --hhh_iii -x 50 -y 80 -s 20000
    11a. python simple_influx_writer.py --help for options.
12. Probably doesn't need to be that much data, but this is what I did to reproduce.
13. Run for an hour.
14. In graph in grafana notice for an hour period you have 80 Ys.
15. Stop script and upgrade influxdb to 1.2.1
16. Execute script again.
17. Look at graph in Grafana for an hour and notice you only see 20ish Ys. Reduce time to 30 mins you see 40-50 Ys. Reduce to 5 mins you will see all 80 Ys.

[oshamin]

Hi @gerrickw, what do you mean by "Ys."? Can you show image?

[gerrickw]

Grouping by Ys. In this case from the script there are 50 Xs by 80 Ys. So each Y is an aggregated view of max() of all of the Xs. So a large grid or shards that is showing the max value of the largest X of the Y. Images showing the same query with only changing the amount of time queried in the graph. Notice the legend showing suddenly more data, even though the same query. With influxdb 1.1.4, the legend always shows 80 Ys no matter the time range.

[jwilder]

@gerrickw Can you see if setting:

[http]
max-row-limit = 0

resolves that?

[gerrickw]

@jwilder Setting to 0 no longer causes the issue. Hmm, seems odd to suddenly make a default. This makes it appear it is a bug with influx not giving all of the data or possibly make the user believe there are no other values and drawing incorrect conclusions. But this fixed my issue, so possibly this isn't related to main issue here.

[elvarb]

Here is the Grafana ticket on this issue, grafana/grafana#7380

[elvarb]

It seems to be how InfluxDB now returns the json results in chunks, which is good but Grafana does not support it.

Is there some way to disable the chunked feature?

[elvarb]

[http]
max-row-limit = 0

does indeed solve this

[jwilder]

Fixed via #8133