Subquery #4831

codesome · 2018-11-06T16:33:42Z

Based on this design doc: https://docs.google.com/document/d/1P_G87zN88YvmMr4iwLWygChMTZhai1L7S_c0awu1CAE/edit

Checklist

* Introduced a new item called 'itemColon' => ':'. The colon present in the metric identifier wont be considered as itemColon. Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

promql/parse.go

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

codesome · 2018-11-08T08:37:00Z

@brian-brazil I see 2 potential places to execute subqueries

In Engine.populateSeries(...) we execute subquery and populate the series.
In evaluator.eval(...) we execute the expression of subquery using another evaluator and return it.

Am I missing something here? Which one is preferred? (My choice is 1).

brian-brazil · 2018-11-08T08:45:02Z

It should be in eval I think.

codesome · 2018-11-10T14:11:28Z

While trying to device a solution in eval, I see 1 complication.

Consider this expression

rate(baz[1m]) +  rate( ( rate(foo[5m]) + rate(bar[10m]) )[15m:5s] )

As we are populating all the vector and matrix selectors beforehand, we would ideally like to load up to 25m back in time (15m of subquery and 10m of matrix selector inside it).

But as the maxOffset will be taken same for the entire expression, loading up to 25m back in time for baz is unnecessary, as it needs only up to 1m.

As there can be more nesting inside subqueries, this this added overhead fine?

codesome · 2018-11-10T14:30:44Z

Or I can selectively load for subqueries.

brian-brazil · 2018-11-10T14:38:01Z

As there can be more nesting inside subqueries, this this added overhead fine?

This is not a problem for local storage.

* The step of subquery is still not optional, need to be given. * You cannot pass subquery as an argument to a function yet. Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

codesome · 2018-11-12T10:50:09Z

@brian-brazil I am stuck at choosing default step for expressions of the form expr[5m:], where step is optional.

For range queries I am able to take the interval given with it. But I am not sure what is to be done in the case of an instant query.

brian-brazil · 2018-11-12T11:04:20Z

For range queries I am able to take the interval given with it. But I am not sure what is to be done in the case of an instant query.

In both cases, use the global evaluation interval.

codesome · 2018-11-12T12:57:45Z

I was unable to find out where the global evaluation interval is

I believe it is not this

prometheus/config/config.go

Line 268 in b1e779d

EvaluationInterval model.Duration `yaml:"evaluation_interval,omitempty"`

brian-brazil · 2018-11-12T13:19:21Z

That is it.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

codesome · 2018-11-12T17:39:44Z

@brian-brazil I think this is good for a review now. I will be adding tests in the meanwhile.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

These tests are taken from selectors.test and functions.test. The range selector were just converted to subquery to generate these tests. All the converted tests from selectors.test passed, but many from functions.test were failing due to small drift in the result, only the passing ones are in this commit. Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

brian-brazil

How will this interact with sample counting and sample slice pooling?

cmd/prometheus/main.go

promql/engine.go

promql/lex_test.go

promql/parse.go

promql/testdata/subquery.test

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

codesome · 2018-12-15T20:44:46Z

@brian-brazil I have added tests for edge data and offsets. The tests for queries that would return a matrix are in Go, rest all are in testdata directory.

What more do I need to add?

promql/testdata/subquery.test

promql/parse.go

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

brian-brazil · 2018-12-18T09:30:08Z

You've got a test failure.

brian-brazil · 2018-12-18T09:30:51Z

The docs also need updating.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

codesome · 2018-12-18T10:28:39Z

Would the docs go inside docs/querying/basics.md?

brian-brazil · 2018-12-18T10:43:03Z

Yeah, I think that makes most sense. Just before Operators.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

promql/value.go

docs/querying/basics.md

brian-brazil · 2018-12-18T15:32:38Z

docs/querying/basics.md

@@ -169,6 +169,20 @@ The same works for range vectors. This returns the 5-minutes rate that

    rate(http_requests_total[5m] offset 1w)

+## Subquery
+
+Subquery allows you to run an instant query for a given range and resolution. Like how range vector selector is for vector selector, subquery is for an instant query. The result of a subquery is a matrix. 


This comparison to range vector selectors is confusing. It'd be instant vector expressions.

docs/querying/basics.md

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

codesome · 2018-12-20T13:27:07Z

I have updated the docs and tests (#1227 (comment), #1227 (comment)).

I haven't verified the below test - the value is got when rate(metric[5m]) is run at t=1m.

eval instant at 20m min_over_time(rate(metric[5m])[20m:1m])
  {} 0.12119047619047618

brian-brazil · 2018-12-20T15:56:26Z

docs/querying/examples.md

+
+This is an example of nested subquery. The subquery for the rate uses default resolution. Note that using subqueries unnecessarily is unwise.
+
+    max_over_time(rate(http_requests_total[30s:5s])[10m:])


This isn't a good example, you should use a range vector selector with rate.

I thought so, will update it.

max_over_time(rate(rate(distance_covered_total[5s])[30s:5s])[10m:]) - does this look sane? Looks like max acceleration over time.

A rate of a rate doesn't make sense, as rate can only be applied to a counter and the output of rate is a gauge. A deriv of a rate would make sense though.

So does max_over_time(deriv(rate(distance_covered_total[5s])[30s:5s])[10m:]) look fine for the doc?

Yes, that's fine.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

codesome · 2018-12-22T05:25:11Z

I feel that it is pretty much done. Anything missing @brian-brazil?

brian-brazil · 2018-12-22T13:47:18Z

Thanks!

juliusv · 2018-12-22T14:05:44Z

Wow, congrats on getting this done!

roidelapluie · 2018-12-22T15:32:07Z

this is huge!! congratz!!

codesome · 2018-12-22T17:37:55Z

Thanks @brian-brazil for all the patient reviews!

codesome added 3 commits November 3, 2018 22:18

Lexer for subquery

b4a0f85

* Introduced a new item called 'itemColon' => ':'. The colon present in the metric identifier wont be considered as itemColon. Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Parser for subquery based on unary expression

ed23b8c

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Unit tests for unary expression based subquery

687ff22

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

codesome force-pushed the subqueries branch from deebef1 to e6ed4b4 Compare November 6, 2018 16:37

brian-brazil reviewed Nov 6, 2018

View reviewed changes

promql/parse.go Outdated Show resolved Hide resolved

Parser for subquery for all expressions

b30086f

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

codesome force-pushed the subqueries branch from e6ed4b4 to 7f19645 Compare November 7, 2018 12:34

Merge remote-tracking branch 'upstream/master' into subqueries

4c74315

codesome force-pushed the subqueries branch from 7f19645 to 4c74315 Compare November 7, 2018 12:54

Partially working query engine for subqueries

0a015fc

* The step of subquery is still not optional, need to be given. * You cannot pass subquery as an argument to a function yet. Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

codesome force-pushed the subqueries branch from 163823c to f8f7633 Compare November 12, 2018 10:42

Subquery as function argument

7d07e42

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

codesome force-pushed the subqueries branch from f8f7633 to 7d07e42 Compare November 12, 2018 10:43

Optional subquery step

68d77cf

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

codesome changed the title ~~[WIP] Subquery~~ Subquery Nov 12, 2018

codesome added 3 commits November 13, 2018 14:32

Fix duplicate label set in vector

d803fe8

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Extra check for start time of subquery evaluation

7bae5d9

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

brian-brazil reviewed Nov 13, 2018

View reviewed changes

Small nits

1ea2e3d

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Fix review comments

7dc326d

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Merge remote-tracking branch 'upstream/master' into subqueries

6f6cd06

brian-brazil reviewed Dec 17, 2018

View reviewed changes

promql/testdata/subquery.test Outdated Show resolved Hide resolved

promql/parse.go Outdated Show resolved Hide resolved

Update error message

a29b240

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Fix test

af74e46

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Docs

300fdc6

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

free reviewed Dec 18, 2018

View reviewed changes

promql/value.go Show resolved Hide resolved

free reviewed Dec 18, 2018

View reviewed changes

docs/querying/basics.md Outdated Show resolved Hide resolved

docs/querying/basics.md Outdated Show resolved Hide resolved

brian-brazil reviewed Dec 18, 2018

View reviewed changes

docs/querying/basics.md Outdated Show resolved Hide resolved

codesome added 2 commits December 20, 2018 04:32

Fix Seek and added more tests

fd1b008

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Fix doc

5ff3f1a

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Merge remote-tracking branch 'upstream/master' into subqueries

6dd9868

brian-brazil reviewed Dec 20, 2018

View reviewed changes

Update example in the doc

2acacd2

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

brian-brazil merged commit dbe55c1 into prometheus:master Dec 22, 2018

codesome deleted the subqueries branch December 28, 2018 05:23

codesome mentioned this pull request Dec 28, 2018

Subquery blog post prometheus/docs#1248

Merged

4 tasks

liguozhong mentioned this pull request Sep 13, 2020

sum_over_time should use semi-closed range interval #7933

Closed

Subquery #4831

Subquery #4831

codesome commented Nov 6, 2018 •

edited

codesome commented Nov 8, 2018

brian-brazil commented Nov 8, 2018

codesome commented Nov 10, 2018

codesome commented Nov 10, 2018

brian-brazil commented Nov 10, 2018

codesome commented Nov 12, 2018

brian-brazil commented Nov 12, 2018

codesome commented Nov 12, 2018

brian-brazil commented Nov 12, 2018

codesome commented Nov 12, 2018

brian-brazil left a comment

codesome commented Dec 15, 2018

brian-brazil commented Dec 18, 2018

brian-brazil commented Dec 18, 2018

codesome commented Dec 18, 2018

brian-brazil commented Dec 18, 2018

brian-brazil Dec 18, 2018

codesome commented Dec 20, 2018

brian-brazil Dec 20, 2018

codesome Dec 21, 2018

codesome Dec 21, 2018

brian-brazil Dec 21, 2018

codesome Dec 21, 2018

brian-brazil Dec 21, 2018

codesome commented Dec 22, 2018

brian-brazil commented Dec 22, 2018

juliusv commented Dec 22, 2018

roidelapluie commented Dec 22, 2018

codesome commented Dec 22, 2018


		This is an example of nested subquery. The subquery for the rate uses default resolution. Note that using subqueries unnecessarily is unwise.

		max_over_time(rate(http_requests_total[30s:5s])[10m:])

Subquery #4831

Subquery #4831

Conversation

codesome commented Nov 6, 2018 • edited

codesome commented Nov 8, 2018

brian-brazil commented Nov 8, 2018

codesome commented Nov 10, 2018

codesome commented Nov 10, 2018

brian-brazil commented Nov 10, 2018

codesome commented Nov 12, 2018

brian-brazil commented Nov 12, 2018

codesome commented Nov 12, 2018

brian-brazil commented Nov 12, 2018

codesome commented Nov 12, 2018

brian-brazil left a comment

Choose a reason for hiding this comment

codesome commented Dec 15, 2018

brian-brazil commented Dec 18, 2018

brian-brazil commented Dec 18, 2018

codesome commented Dec 18, 2018

brian-brazil commented Dec 18, 2018

brian-brazil Dec 18, 2018

Choose a reason for hiding this comment

codesome commented Dec 20, 2018

brian-brazil Dec 20, 2018

Choose a reason for hiding this comment

codesome Dec 21, 2018

Choose a reason for hiding this comment

codesome Dec 21, 2018

Choose a reason for hiding this comment

brian-brazil Dec 21, 2018

Choose a reason for hiding this comment

codesome Dec 21, 2018

Choose a reason for hiding this comment

brian-brazil Dec 21, 2018

Choose a reason for hiding this comment

codesome commented Dec 22, 2018

brian-brazil commented Dec 22, 2018

juliusv commented Dec 22, 2018

roidelapluie commented Dec 22, 2018

codesome commented Dec 22, 2018

codesome commented Nov 6, 2018 •

edited