Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query limits - log the rejected queries #1637

Open
HariSekhon opened this issue May 15, 2019 · 1 comment
Open

Query limits - log the rejected queries #1637

HariSekhon opened this issue May 15, 2019 · 1 comment

Comments

@HariSekhon
Copy link
Contributor

HariSekhon commented May 15, 2019

Improvement Request to log queries rejected for breaching query limits.

When limiting the number of data points, you get an error message like this:

t=2019-05-15T08:56:20+0000 lvl=info msg="Request failed" logger=tsdb.opentsdb status="413 Request Entity Too Large" body="{\"error\":{\"code\":413,\"message\":\"Sorry, you have attempted to fetch more than our limit of 1000000 data points. Please try filtering using more tags or decrease your time range.\", ...

It would be more useful if when query limits are exceeded, the query was logged in this error message so that we could track them down and fix / improve the queries. The whole company uses Grafana on OpenTSDB so I can't guess which queries from which out of 1000+ dashboards these are, so this logging is quite essential to improving the situation.

This should also apply to queries that exceed max size, and if it starts working, queries that exceed the timeouts (which doesn't work at time of writing, see #1635).

@GreatSnoopy
Copy link
Contributor

@HariSekhon You may be interested in this - we are building for reasons and needs in the same ballpark

https://github.com/adobe/opentsdb-protector

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants