New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elasticsearch table queries limited to 500 entries #3426

Closed
NickG123 opened this Issue Dec 4, 2015 · 13 comments

Comments

Projects
None yet
8 participants
@NickG123
Contributor

NickG123 commented Dec 4, 2015

Hi there,
I have been using the new Table panel in the 2.6.0 beta1.
First of all, this is an awesome new feature, great work!
One thing I noticed is that queries from ElasticSearch seem to have a hard-coded max size of 500, so (for example) when I make a query for log entries from the past 30 days, I only get 500 of them back.
It would be nice if this was a customizable option.

@torkelo

This comment has been minimized.

Show comment
Hide comment
@torkelo

torkelo Dec 6, 2015

Member

500 seems like a reasonably number, you can always narrow the time range?

Member

torkelo commented Dec 6, 2015

500 seems like a reasonably number, you can always narrow the time range?

@1tft

This comment has been minimized.

Show comment
Hide comment
@1tft

1tft Dec 6, 2015

Only for information, the 500 limit does not affect the normal time series histogramm, correct? Also for you information elastic 2.1 has placed a soft limit on the maximum number of results (default 10,000) that can be requested in any single search request (elastic/elasticsearch#13188). But I think grafana does not make a normal "search" when creating the histogramm?!

1tft commented Dec 6, 2015

Only for information, the 500 limit does not affect the normal time series histogramm, correct? Also for you information elastic 2.1 has placed a soft limit on the maximum number of results (default 10,000) that can be requested in any single search request (elastic/elasticsearch#13188). But I think grafana does not make a normal "search" when creating the histogramm?!

@torkelo

This comment has been minimized.

Show comment
Hide comment
@torkelo

torkelo Dec 6, 2015

Member

no it only affects raw log queries

Member

torkelo commented Dec 6, 2015

no it only affects raw log queries

@NickG123

This comment has been minimized.

Show comment
Hide comment
@NickG123

NickG123 Dec 6, 2015

Contributor

500 does seem like a reasonable amount (I can't imagine wanting to scroll through more than that) I just thought it might be desirable to make it configurable. If you disagree, feel free to close the ticket.

Contributor

NickG123 commented Dec 6, 2015

500 does seem like a reasonable amount (I can't imagine wanting to scroll through more than that) I just thought it might be desirable to make it configurable. If you disagree, feel free to close the ticket.

@torkelo torkelo closed this Dec 7, 2015

@Chris7

This comment has been minimized.

Show comment
Hide comment
@Chris7

Chris7 May 9, 2017

Sorry to resurrect this, but I've been using grafana to aggregate logs across multiple docker services, so 500 is actually a small number when aggregating multiple docker containers/services. I noticed this is still non-configurable, would it be possible to have this option definable in the data source or in the query panel itself?

Chris7 commented May 9, 2017

Sorry to resurrect this, but I've been using grafana to aggregate logs across multiple docker services, so 500 is actually a small number when aggregating multiple docker containers/services. I noticed this is still non-configurable, would it be possible to have this option definable in the data source or in the query panel itself?

@daniellee

This comment has been minimized.

Show comment
Hide comment
@daniellee

daniellee May 9, 2017

Member

@Chris7 is this in the table panel?

Member

daniellee commented May 9, 2017

@Chris7 is this in the table panel?

@Chris7

This comment has been minimized.

Show comment
Hide comment
@Chris7

Chris7 May 9, 2017

It isn't, I created my own logstreamer panel (since tables can't really tail a log and the time sorting doesn't really work as well), though it does draw from the table panel for column setup:

ginkgo_logstream

Chris7 commented May 9, 2017

It isn't, I created my own logstreamer panel (since tables can't really tail a log and the time sorting doesn't really work as well), though it does draw from the table panel for column setup:

ginkgo_logstream

@daniellee

This comment has been minimized.

Show comment
Hide comment
@daniellee

daniellee May 10, 2017

Member

@Chris7 That sounds . Any chance you could send submit that as a plugin to https://github.com/grafana/grafana-plugin-repository? And with this use case, we would definitely consider adding a variable for the max limit.

Member

daniellee commented May 10, 2017

@Chris7 That sounds . Any chance you could send submit that as a plugin to https://github.com/grafana/grafana-plugin-repository? And with this use case, we would definitely consider adding a variable for the max limit.

@Chris7

This comment has been minimized.

Show comment
Hide comment
@Chris7

Chris7 May 10, 2017

Certainly @daniellee. I'll spend some time cleaning it up this week so it's not that horrific.

Chris7 commented May 10, 2017

Certainly @daniellee. I'll spend some time cleaning it up this week so it's not that horrific.

@cr42

This comment has been minimized.

Show comment
Hide comment
@cr42

cr42 May 31, 2017

+1 to make the 500 limit configurable for elasticsearch raw documents. I often use grafana as a GUI for ES queries that often exceed this limit and then export the data as csv.

cr42 commented May 31, 2017

+1 to make the 500 limit configurable for elasticsearch raw documents. I often use grafana as a GUI for ES queries that often exceed this limit and then export the data as csv.

@jaune-rouge

This comment has been minimized.

Show comment
Hide comment
@jaune-rouge

jaune-rouge Jun 22, 2017

+1 to that.

Additionally I believe this is the same issue for elastic search template queries. I'm trying to get everything and then use the regex feature to extract file extensions. However, since the query limits itself to 500 I'm not getting all of the file extensions as they don't all appear in the 500 documents returned by the query.

jaune-rouge commented Jun 22, 2017

+1 to that.

Additionally I believe this is the same issue for elastic search template queries. I'm trying to get everything and then use the regex feature to extract file extensions. However, since the query limits itself to 500 I'm not getting all of the file extensions as they don't all appear in the 500 documents returned by the query.

@cr42

This comment has been minimized.

Show comment
Hide comment
@cr42

cr42 Jun 22, 2017

I think this is now covered by #8527

cr42 commented Jun 22, 2017

I think this is now covered by #8527

@aurimasplu

This comment has been minimized.

Show comment
Hide comment
@aurimasplu

aurimasplu commented Feb 27, 2018

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment