LogQL: Vector and Range Vector Aggregation. #654
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cyriltovena
commented
Jun 18, 2019
tomwilkie
reviewed
Jun 18, 2019
tomwilkie
reviewed
Jun 19, 2019
cyriltovena
force-pushed
the
aggregation
branch
4 times, most recently
from
020c665
to
adba79b
Compare
|
@tomwilkie This is ready to review a final time. I've added a bunch of tests around LogQL evaluation and everything looks good. There are scalar and more complex arithmetic operations still not done, or even the extract function but I'd like to start with this first. Overall I think the current structure will support those two easily. 🙏 |
cyriltovena
force-pushed
the
aggregation
branch
2 times, most recently
from
10cfa85
to
e7d307b
Compare
cyriltovena
commented
Aug 29, 2019
| Examples: | ||
|
|
||
| ```bash | ||
| $ curl -G -s "http://localhost:3100/api/v1/query_range" --data-urlencode 'query=sum(rate({job="varlogs"}[10m])) by (level)' --data-urlencode 'step=300' | jq |
There was a problem hiding this comment.
can we use /api/loki/api/v1/query_range ? @tomwilkie @slim-bean
Requirement :
- we need a unique prefix to the path
- we can point golang client at this endpoint.
- should be similar to cortex api
|
|
- adds avg,min,max,top,bottomk,stddev,stdvar,count - updates api documentation - adds tests Improve yacc & go lexer to understand duration Remove support for regexp in all queries Clean up querier and logselector
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
cyriltovena
force-pushed
the
aggregation
branch
from
2a08a4c
to
9e2835d
Compare
|
I'm trying to retrofit regexp query string. |
- keep supporting regex for old endpoint. - use reserved keyword for removed property in grpc
cyriltovena
force-pushed
the
aggregation
branch
from
2826f30
to
5faefdd
Compare
This was referenced Sep 5, 2019
4 tasks
rfratto
added a commit
that referenced
this pull request
Sep 6, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds vector and range vector aggregation. I've exposed two new endpoint
/api/v1/queryfor instant query and/api/v1/query_rangefor query over a range of time.I've kept the old endpoint for range logs at
/api/prom/querybecause new endpoints have different response structure and this would break Grafana integration. /cc @davkalExamples of new query possible:
count_over_time({job="mysql"}[5m])total count of log line recorded over the last 5mrate(({job="mysql"} |= "error")[1m])rate of log line containingerrorsover the last minute.avg(rate(({app="nginx"} |= "GET")[1m])) by (cluster)average rate of http GET requests by cluster.For now the step evaluation is dictated by the range vector iterator as this is the only one we required and the root AST node is responsible for giving back the result. This might need to change for
extractor unary operator.Performance wise, the biggest bottleneck is retrieving all sample and deduping them. I've also notice a lot of GC while doing large queries but this is due to the fact that we're throwing away all log line very quickly.
Documentation is up to date, next steps will be to work on the
extractfunction allowing to extract sample from log line.