feat: improve performance of `first_over_time` and `last_over_time` queries by sharding them #11605

jeschkies · 2024-01-08T11:52:11Z

What this PR does / why we need it:
With the introduction of sending query plans with custom nodes to the querier we can now shard first_over_time queries.

Checklist

Reviewed the CONTRIBUTING.md guide (required)
Documentation added
Tests updated
CHANGELOG.md updated
- If the change is worth mentioning in the release notes, add add-to-release-notes label
Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

github-actions · 2024-01-08T11:55:27Z

Trivy scan found the following vulnerabilities:

HIGH, Target: docker.io/grafana/loki:main-61a4205 (alpine 3.18.4), Type: alpine openssl: Incorrect cipher key and IV length processing in libcrypto3 v3.1.3-r0. Fixed in v3.1.4-r0
HIGH, Target: docker.io/grafana/loki:main-61a4205 (alpine 3.18.4), Type: alpine openssl: Incorrect cipher key and IV length processing in libssl3 v3.1.3-r0. Fixed in v3.1.4-r0
\nTo see more details on these vulnerabilities, and how/where to fix them, please run docker build -t grafana/loki:main-61a4205 -f cmd/loki/Dockerfile .
trivy i grafana/loki:main-61a4205 on your branch. If these were not introduced by your PR, please considering fixing them in via a subsequent PR. Thanks!

Signed-off-by: Callum Styan <callumstyan@gmail.com>

in each vector, not the first Signed-off-by: Callum Styan <callumstyan@gmail.com>

cstyan

I'm extended this to include implementing last_over_time here.

My only comments would be:

that renaming first_over_time.go to something more generic, or first_last_over_time.go like I did in my branch, probably makes sense
some comments would be useful (such as where I commented on the PR) for first time readers of the code

pkg/logql/first_over_time.go

jeschkies · 2024-01-18T13:14:32Z

pkg/logql/engine.go

-				T: ts,
+				//T: ts,


I'm not sure we can do it like this. What do you think, @cstyan?

I'm not sure what you mean?

The current code on main overrides he timestamps in each vector.

isn't the timestamp from the actual vector samples more accurate than the one got from stepEvaluator.Next()?

Can you help me understand why it won't make sense?

It does make sense for us. I'm just not sure some other component is relying on this overridden timestamp.

pkg/logql/first_over_time.go

…first-over-time

pkg/logql/syntax/ast.go

kavirajk · 2024-01-29T10:58:18Z

pkg/logql/first_last_over_time.go

+// order (see next, moves the current timestamp forward by step). So this
+// means that for each downstreamed shard of a first_over_time, selecting the
+// first sample here in each iterator is getting us the earliest timestamped
+// value. Later on when we merge we select the earliest from all the shards.


I believe this comment should be broken down into pieces and moved to three different iterators.
firstWithTimestampBatch, lastWithTimestampBatch and mergeOverTimeEvaluator?

kavirajk

Overall looks great. Nice work folks! learned few things about sharding in downstream engine.

Added few comments. One question for me is, how do we make sure the correctness of first_over_time and last_over_time running with old engine vs downstream engine.?

I see you added tests in TestMappingEquivalence. But do you think should be also added on other tests like TestRangeMappingEquivalence on downstream engine tests?

kavirajk

LGTM. thanks for addressing the feedbacks.

Still this one applies I think.

I see you added tests in TestMappingEquivalence. But do you think should be also added on other tests like TestRangeMappingEquivalence on downstream engine tests? Rationale being the later tests for more cases of different range aggregation.

approving to unblock.

cstyan · 2024-02-20T00:57:05Z

hey @kavirajk, actually something fun I learned this past week when finally finding the cause of the issues with our quantile_over_time, TestMappingEquivalence is actually testing the results equivalency for each query as a range query. What we don't have (AFAICT) is tests for the results equivalency for instant queries. The test names here are actually a bit misleading; TestMappingEquivalence is testing for equivalency with sharded queries while TestRangeMappingEquivalence is testing for equivalency with time range split queries.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

cstyan · 2024-04-23T22:39:59Z

@jeschkies any opinions on merging this as is vs as behind a feature flag? given that it's not being executed via a probabilistic data structure I don't think we need a feature flag

Signed-off-by: Callum Styan <callumstyan@gmail.com>

pull-request-size bot added the size/L label Jan 8, 2024

jeschkies assigned cstyan Jan 8, 2024

jeschkies assigned cstyan and unassigned cstyan Jan 9, 2024

jeschkies requested a review from cstyan January 9, 2024 17:11

jeschkies unassigned cstyan Jan 9, 2024

jeschkies added 8 commits January 11, 2024 10:39

Step 1

de536b6

Step 2

235e386

Step 3

689c161

Step 4

09f6d66

Step 5

f7463ae

Step 6

750cec7

Step 7

9281f44

Step 8

8f589fe

jeschkies force-pushed the karsten/first-over-time branch from e70ccb5 to 8f589fe Compare January 11, 2024 09:45

cstyan added 10 commits January 12, 2024 17:00

downstreaming of last over time expr

810b07c

Signed-off-by: Callum Styan <callumstyan@gmail.com>

add expr type for merging of last over time

5e106bf

Signed-off-by: Callum Styan <callumstyan@gmail.com>

add shardmapping for last over time

7b3cadc

Signed-off-by: Callum Styan <callumstyan@gmail.com>

wire up new op type in ast

b10c1de

Signed-off-by: Callum Styan <callumstyan@gmail.com>

setup last iterator

928ccfa

Signed-off-by: Callum Styan <callumstyan@gmail.com>

add iterators functionality

f89ea65

Signed-off-by: Callum Styan <callumstyan@gmail.com>

add evaluator

5ddf9df

Signed-off-by: Callum Styan <callumstyan@gmail.com>

add test cases for last over time

104892b

Signed-off-by: Callum Styan <callumstyan@gmail.com>

fix some things I misunderstood

67dd511

Signed-off-by: Callum Styan <callumstyan@gmail.com>

the last timestamp iterator aggregator needs to select the last sample

8e22df0

in each vector, not the first Signed-off-by: Callum Styan <callumstyan@gmail.com>

cstyan reviewed Jan 16, 2024

View reviewed changes

pkg/logql/first_over_time.go Outdated Show resolved Hide resolved

pkg/logql/first_over_time.go Outdated Show resolved Hide resolved

Merge remote-tracking branch 'grafana/main' into karsten/first-over-time

6404308

jeschkies commented Jan 18, 2024

View reviewed changes

Merge branch 'main' into karsten/first-over-time

0d420dc

cstyan mentioned this pull request Jan 23, 2024

[logQL] Allow sharding of first and last over time aggregations #11737

Closed

Merge commit 'f8c9c2bdbf2013cc151e986330263bf08dbeeefc' into karsten/…

2bfd0b5

…first-over-time

pull-request-size bot added size/XL and removed size/L labels Jan 23, 2024

jeschkies added 5 commits January 23, 2024 11:30

Correct vector check.

af67edd

Unify first and last

361babc

Increase relative error

2fb80d7

Document some code

1b6cc02

Merge remote-tracking branch 'grafana/main' into karsten/first-over-time

1f04c45

jeschkies marked this pull request as ready for review January 23, 2024 11:15

jeschkies requested a review from a team as a code owner January 23, 2024 11:15

kavirajk changed the title ~~Shard first_over_time queries.~~ Shard first_over_time and last_over_time queries. Jan 29, 2024

kavirajk reviewed Jan 29, 2024

View reviewed changes

pkg/logql/syntax/ast.go Show resolved Hide resolved

kavirajk reviewed Jan 29, 2024

View reviewed changes

jeschkies added 4 commits January 29, 2024 14:30

Merge remote-tracking branch 'grafana/main' into karsten/first-over-time

d2b055b

Split up documentation

87bbd75

Drop commented TS

307bfc1

Merge branch 'main' into karsten/first-over-time

4dc234f

kavirajk approved these changes Jan 31, 2024

View reviewed changes

jeschkies and others added 3 commits March 6, 2024 10:28

Merge branch 'main' into karsten/first-over-time

995b982

Merge branch 'main' into karsten/first-over-time

5231b6f

Signed-off-by: Callum Styan <callumstyan@gmail.com>

fixes after merging in main

9e10b2c

cstyan changed the title ~~Shard first_over_time and last_over_time queries.~~ feat: improve performance of first_over_time and last_over_time queries by sharding them Apr 23, 2024

cstyan and others added 4 commits May 3, 2024 14:47

Merge branch 'main' into karsten/first-over-time

2325e85

Return an empty result instead of nil

69f504b

fix other minor issues

27a5d66

Signed-off-by: Callum Styan <callumstyan@gmail.com>

handle sharding noop case for first/last over time

1b09ad3

Signed-off-by: Callum Styan <callumstyan@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: improve performance of `first_over_time` and `last_over_time` queries by sharding them #11605

feat: improve performance of `first_over_time` and `last_over_time` queries by sharding them #11605

jeschkies commented Jan 8, 2024 •

edited

github-actions bot commented Jan 8, 2024 •

edited

cstyan left a comment

jeschkies Jan 18, 2024

cstyan Jan 18, 2024

jeschkies Jan 23, 2024

kavirajk Jan 29, 2024

jeschkies Jan 29, 2024

kavirajk Jan 29, 2024 •

edited

jeschkies Jan 29, 2024

kavirajk left a comment

kavirajk left a comment

cstyan commented Feb 20, 2024

cstyan commented Apr 23, 2024

feat: improve performance of first_over_time and last_over_time queries by sharding them #11605

Are you sure you want to change the base?

feat: improve performance of first_over_time and last_over_time queries by sharding them #11605

Conversation

jeschkies commented Jan 8, 2024 • edited

github-actions bot commented Jan 8, 2024 • edited

cstyan left a comment

Choose a reason for hiding this comment

jeschkies Jan 18, 2024

Choose a reason for hiding this comment

cstyan Jan 18, 2024

Choose a reason for hiding this comment

jeschkies Jan 23, 2024

Choose a reason for hiding this comment

kavirajk Jan 29, 2024

Choose a reason for hiding this comment

jeschkies Jan 29, 2024

Choose a reason for hiding this comment

kavirajk Jan 29, 2024 • edited

Choose a reason for hiding this comment

jeschkies Jan 29, 2024

Choose a reason for hiding this comment

kavirajk left a comment

Choose a reason for hiding this comment

kavirajk left a comment

Choose a reason for hiding this comment

cstyan commented Feb 20, 2024

cstyan commented Apr 23, 2024

feat: improve performance of `first_over_time` and `last_over_time` queries by sharding them #11605

feat: improve performance of `first_over_time` and `last_over_time` queries by sharding them #11605

jeschkies commented Jan 8, 2024 •

edited

github-actions bot commented Jan 8, 2024 •

edited

kavirajk Jan 29, 2024 •

edited