After upgrading to 1.2.0: series missing for queries, depending on time interval supplied in group by time

[FrederikP]

Bug report

After upgrading InfluxDB from 1.1.1 to 1.2.0 I noticed that some series where missing in my grafana graphs. For some reason these series reappeared when decreasing the browser window width (which leads to a bigger time interval supplied in the Group By time() statement that grafana generates.

I then looked deeper into the queries and executed the exact same query on the same data using InfluxDB 1.1.1 and 1.2.0 and compared the results. The data I operated on where application based cpu measurements (written every two minutes) which I grouped by app tag.

The query grafana (and I) performed was:

curl -G http://localhost:8086/query --data-urlencode "db=mydb" --data-urlencode "q=SELECT max("value") FROM "cpu_app" WHERE time > 1485323700000ms - 7d GROUP BY time(5m), "app" fill(null)" --data-urlencode "epoch=ms"

For an interval time of 5 minutes the result included 15 series (apps) for influxdb 1.1.1 and only 5 series for influxdb 1.2.0.

When using an interval time of 1 hour the results included 15 series for both versions, which is the expected behavior.
I don't understand why there is a difference between 1.1.1 and 1.2.0 for the 5 minute interval version and I really don't know what's happening here. It's hard to give any more useful information, because I don't know what other info I could supply.

I'll attach the results of the 4 queries, so you can see the difference between the versions.
influx-check.tar.gz

Please let me know what other info I can supply to help you solve this problem.

System info:

InfluxDB 1.2.0
Debian Jessie
.deb install

Steps to reproduce:

1. write data to influx
2. perform queries with different group by time intervals on 1.1.1 and 1.2.0 version of influx

Expected behavior:

The number of series is equal for 1.1.1. and 1.2.0 for all queries

Actual behavior:

The number of series differs depending on the interval size when compared to the complete queried time frame. I guess the relation between interval size and queried time frame is important here, but I don't know how exactly.

[FrederikP]

I took two screenshots in grafana to visualize the issue. I set a start- and end time and took a screen shot for 1.1.1 and then I only upgraded the db to 1.2.0 and refreshed the grafana graph (not the complete page only the data using grafanas update button), so it displays the exact same timeframe and thus uses the same query.

Screenshot for v1.1.1:

Screenshot for v1.2.0:

As you can see the 1.2.0 result is missing many series (for example the sshd one) just like described in the issue and shown in the raw results that I attached in the issue.

What I noticed is: The smaller I make the browser window (horizontally), the more series come up, meaning: the bigger the group by interval, the more series show up.

[jsternberg]

I think this may be caused by a bug I fixed at the same time as when I implemented subqueries. If you notice, the responses in the 1.2 version have "partial": true attached to them. There is one inside of the series which means that the series was split into a separate response and there is one on the result object itself meaning that the query response is telling the client that there will be another JSON object to follow.

Can you see if Grafana is using chunked=true when issuing a query? We have lagged behind in terms of making sure there's a Javascript client that can handle very large query returns. You can also just disable this feature by setting max-row-limit = 0 in the configuration file. Be aware that the reason why it is set to a default is to prevent the server from blowing up when the query is too large which is the reason why it's trying to send a partial response.

[jsternberg]

Also, removing chunked=true won't stop the problem. If chunked=false and the query exceeds max-row-limit, it just truncates the rest and the server response tells you the data is partial. This is to protect from people issuing queries that could OOM the server.

[FrederikP]

As far as I can tell grafana doesn't use chunked=true in queries. Also the response includes "partial": true, but no following request is performed.

After setting max-row-limit to 0 in my influx config, the graph behaves as expected and all data is shown. So this is not an influx issue, but a grafana issue that their team is maybe not aware exists.

Thanks for your response.

[petrslavotinek]

I've also noticed this behavior. First in grafana but then also in InfluxDB admin interface - so it seems like InfluxDB issue.

I have queries, that I execute in admin interface all at once:
A = SELECT mean("value") FROM "sauter" WHERE "id" = '1880' AND time > now() - 60d GROUP BY time(1h);
B = SELECT mean("value") FROM "sauter" WHERE "id" = '1894' AND time > now() - 60d GROUP BY time(1h);
C =SELECT mean("value") FROM "sauter" WHERE "id" = '3640' AND time > now() - 60d GROUP BY time(500s);
D = SELECT mean("value") FROM "sauter" WHERE "id" = '1922' AND time > now() - 60d GROUP BY time(1000s);

The number of results returned by the server depends on their order.
When executed in order A;B;C;D (1h,1h,500s,1000s) the server returns results only for A, B, C => D is missing.
Order: A;B;D;C (1h,1h,1000s,500s) => returns A, B, D, C => ok.
Order: A;C;B;D (1h,500s,1h,1000s) => returns A, C => B, D missing.
Order: A;C;D;B (1h,500s,1000s,1h) => return A, C => B, D missing.
Order: A;D;B;C (1h,1000s,1h,500s) => returns A, D, B, C => ok.
Order A;D;C;B (1h,1000s,500s,1h) => returns A, D, C => B missing.
Order C;A;B;D (500s,1h,1h,1000s) => returns C => A, B, D missing.
Order C;A;D;B (500s,1h,1000s,1h) => returns C => A, B, D missing.
Order C;D;A;B (500s,1000s,1h,1h) => returns C => A, B, D missing.
Order D;A;B;C (1000s,1h,1h,500s) => returns D, A, B, C => ok.
Order D;A;C;B (1000s,1h,500s,1h) => returns D, A, C => B missing.
Order D;C;A;B (1000s,500s,1h,1h) => returns D, C => A, B missing.

C always seems to break it.

I've also tried some other combinations:
1h;1h;1000s;1000s => returns all.
1000s;1000s;1h;1h => returns first two.
1h;1000s;1h;1000s => returns all.
1000s;1h;1000s;1h => returns first three.

But
1s;1h;1s;1h => returns only first one.

everything for x = from 1s to 518s:
x;1h;x;1h => returns only first one.
everything for x = from 519s to 605s:
x;1h;x;1h => returns first two.
everything for x = from 606s to 1211s:
x;1h;x;1h => returns first three.
everything for x = from 1212s:
x;1h;x;1h => returns all.

I hope it's understandable :)