Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Labels computation LogQLv2 #2875

Merged
merged 6 commits into from
Nov 5, 2020
Merged

Labels computation LogQLv2 #2875

merged 6 commits into from
Nov 5, 2020

Conversation

cyriltovena
Copy link
Contributor

Improve labels computation for labels in logqlv2 pipeline, by using caches and pushing computation all at the same places.

see benchmark diff:

 benchcmp before.txt after4.txt
benchmark                                                                                                                                old ns/op     new ns/op     delta
BenchmarkBufferedIteratorLabels/{app="foo"}-16                                                                                           42.1          36.4          -13.54%
BenchmarkBufferedIteratorLabels/{app="foo"}_!=_"foo"-16                                                                                  42.3          36.8          -13.00%
BenchmarkBufferedIteratorLabels/{app="foo"}_!=_"foo"_|_logfmt_-16                                                                        85.6          44.9          -47.55%
BenchmarkBufferedIteratorLabels/{app="foo"}_!=_"foo"_|_logfmt_|_duration_>_10ms-16                                                       40.7          39.5          -2.95%
BenchmarkBufferedIteratorLabels/{app="foo"}_!=_"foo"_|_logfmt_|_duration_>_10ms_and_component="tsdb"-16                                  40.3          39.9          -0.99%
BenchmarkBufferedIteratorLabels/rate({app="foo"}[1m])-16                                                                                 30.9          25.3          -18.12%
BenchmarkBufferedIteratorLabels/sum_by_(cluster)_(rate({app="foo"}[10s]))-16                                                             33.4          25.0          -25.15%
BenchmarkBufferedIteratorLabels/sum_by_(cluster)_(rate({app="foo"}_!=_"foo"_|_logfmt[10s]))-16                                           32.2          28.4          -11.80%
BenchmarkBufferedIteratorLabels/sum_by_(caller)_(rate({app="foo"}_!=_"foo"_|_logfmt[10s]))-16                                            33.4          28.1          -15.87%
BenchmarkBufferedIteratorLabels/sum_by_(cluster)_(rate({app="foo"}_!=_"foo"_|_logfmt_|_duration_>_10ms[10s]))-16                         28.3          27.0          -4.59%
BenchmarkBufferedIteratorLabels/sum_by_(cluster)_(rate({app="foo"}_!=_"foo"_|_logfmt_|_duration_>_10ms_and_component="tsdb"[1m]))-16     28.9          27.2          -5.88%

benchmark                                                                                                                                old allocs     new allocs     delta
BenchmarkBufferedIteratorLabels/{app="foo"}-16                                                                                           0              0              +0.00%
BenchmarkBufferedIteratorLabels/{app="foo"}_!=_"foo"-16                                                                                  0              0              +0.00%
BenchmarkBufferedIteratorLabels/{app="foo"}_!=_"foo"_|_logfmt_-16                                                                        0              0              +0.00%
BenchmarkBufferedIteratorLabels/{app="foo"}_!=_"foo"_|_logfmt_|_duration_>_10ms-16                                                       0              0              +0.00%
BenchmarkBufferedIteratorLabels/{app="foo"}_!=_"foo"_|_logfmt_|_duration_>_10ms_and_component="tsdb"-16                                  0              0              +0.00%
BenchmarkBufferedIteratorLabels/rate({app="foo"}[1m])-16                                                                                 0              0              +0.00%
BenchmarkBufferedIteratorLabels/sum_by_(cluster)_(rate({app="foo"}[10s]))-16                                                             0              0              +0.00%
BenchmarkBufferedIteratorLabels/sum_by_(cluster)_(rate({app="foo"}_!=_"foo"_|_logfmt[10s]))-16                                           0              0              +0.00%
BenchmarkBufferedIteratorLabels/sum_by_(caller)_(rate({app="foo"}_!=_"foo"_|_logfmt[10s]))-16                                            0              0              +0.00%
BenchmarkBufferedIteratorLabels/sum_by_(cluster)_(rate({app="foo"}_!=_"foo"_|_logfmt_|_duration_>_10ms[10s]))-16                         0              0              +0.00%
BenchmarkBufferedIteratorLabels/sum_by_(cluster)_(rate({app="foo"}_!=_"foo"_|_logfmt_|_duration_>_10ms_and_component="tsdb"[1m]))-16     0              0              +0.00%

benchmark                                                                                                                                old bytes     new bytes     delta
BenchmarkBufferedIteratorLabels/{app="foo"}-16                                                                                           6             0             -100.00%
BenchmarkBufferedIteratorLabels/{app="foo"}_!=_"foo"-16                                                                                  6             1             -83.33%
BenchmarkBufferedIteratorLabels/{app="foo"}_!=_"foo"_|_logfmt_-16                                                                        32            4             -87.50%
BenchmarkBufferedIteratorLabels/{app="foo"}_!=_"foo"_|_logfmt_|_duration_>_10ms-16                                                       3             1             -66.67%
BenchmarkBufferedIteratorLabels/{app="foo"}_!=_"foo"_|_logfmt_|_duration_>_10ms_and_component="tsdb"-16                                  3             1             -66.67%
BenchmarkBufferedIteratorLabels/rate({app="foo"}[1m])-16                                                                                 4             0             -100.00%
BenchmarkBufferedIteratorLabels/sum_by_(cluster)_(rate({app="foo"}[10s]))-16                                                             4             0             -100.00%
BenchmarkBufferedIteratorLabels/sum_by_(cluster)_(rate({app="foo"}_!=_"foo"_|_logfmt[10s]))-16                                           4             0             -100.00%
BenchmarkBufferedIteratorLabels/sum_by_(caller)_(rate({app="foo"}_!=_"foo"_|_logfmt[10s]))-16                                            4             0             -100.00%
BenchmarkBufferedIteratorLabels/sum_by_(cluster)_(rate({app="foo"}_!=_"foo"_|_logfmt_|_duration_>_10ms[10s]))-16                         1             0             -100.00%
BenchmarkBufferedIteratorLabels/sum_by_(cluster)_(rate({app="foo"}_!=_"foo"_|_logfmt_|_duration_>_10ms_and_component="tsdb"[1m]))-16     1             0             -100.00%

Fixes #2827, metrics queries without pipeline should not suffer from regression anymore.

@cyriltovena cyriltovena changed the title Labels logqlv2 Labels computation LogQLv2 Nov 4, 2020
@cyriltovena
Adding a benchmark to compare before and after.
c87548d 
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
@cyriltovena
Improves labels management in logql v2.
339a4f7 
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
@cyriltovena
Cache grouped result when no changes has occured.
b5e5a04 
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
@cyriltovena
Removes unused methods.
415ad7c 
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
@cyriltovena
Adds docs and tests.
9c20705 
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
sandeepsukhani
sandeepsukhani approved these changes Nov 5, 2020
View reviewed changes
Copy link
Contributor

@sandeepsukhani sandeepsukhani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A lot of this is new for me but I did my best to understand the code. Overall it looks good to me, I have just added some comments which might or might not be valid based on my understanding.

pkg/logql/log/labels.go Show resolved Hide resolved
pkg/logql/log/labels.go Show resolved Hide resolved
pkg/logql/log/labels_test.go Show resolved Hide resolved
@cyriltovena
Uncomment tests.
beadfcd 
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
@codecov-io
Copy link

Codecov Report

Merging #2875 into master will increase coverage by 0.39%.
The diff coverage is 76.38%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2875      +/-   ##
==========================================
+ Coverage   61.23%   61.63%   +0.39%     
==========================================
  Files         181      181              
  Lines       14574    14666      +92     
==========================================
+ Hits         8925     9040     +115     
+ Misses       4829     4807      -22     
+ Partials      820      819       -1
Impacted Files Coverage Δ
pkg/chunkenc/dumb_chunk.go 0.00% <ø> (ø)
pkg/chunkenc/interface.go 85.18% <ø> (ø)
pkg/ingester/tailer.go 32.00% <0.00%> (ø)
pkg/logql/log/pipeline.go 25.64% <0.00%> (+12.00%) ⬆️
pkg/ingester/instance.go 55.73% <50.00%> (ø)
pkg/ingester/stream.go 74.77% <50.00%> (ø)
pkg/logql/functions.go 24.13% <50.00%> (ø)
pkg/storage/batch.go 80.74% <71.42%> (-0.12%) ⬇️
pkg/logql/ast.go 83.97% <75.00%> (ø)
pkg/logql/log/labels.go 90.15% <86.81%> (-3.60%) ⬇️
... and 13 more

@cyriltovena cyriltovena merged commit 53f4aa4 into grafana:master Nov 5, 2020
cyriltovena added a commit to cyriltovena/loki that referenced this pull request Nov 9, 2020
@cyriltovena
Fixes metric query issue with no grouping.
1183ce4 
This would remove all labels for metrics queries that does not have grouping, instead of adding all labels.
It was introduced by grafana#2875

I added a regression test.

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
@cyriltovena cyriltovena mentioned this pull request Nov 9, 2020
Merged
cyriltovena added a commit that referenced this pull request Nov 9, 2020
@cyriltovena
Fixes metric query issue with no grouping. (#2902)
dfd2e20 
This would remove all labels for metrics queries that does not have grouping, instead of adding all labels.
It was introduced by #2875

I added a regression test.

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sandeepsukhani sandeepsukhani sandeepsukhani approved these changes

Assignees
No one assigned
Labels
size/XXL
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

rate() query performance regression in 2.0.0
3 participants
@cyriltovena @codecov-io @sandeepsukhani