Logqv2 optimization #2778

cyriltovena · 2020-10-19T10:40:20Z

benchmark before:

pkg: github.com/grafana/loki/pkg/logql/log
Benchmark_Pipeline
Benchmark_Pipeline-16    	  107580	     10271 ns/op	    6387 B/op	      67 allocs/op
PASS
ok  	github.com/grafana/loki/pkg/logql/log	1.228s

after

pkg: github.com/grafana/loki/pkg/logql/log
Benchmark_Pipeline
Benchmark_Pipeline-16    	  199170	      5670 ns/op	    3307 B/op	      54 allocs/op
PASS
ok  	github.com/grafana/loki/pkg/logql/log	1.201s

Way less allocations !

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

… parsing and moar tests

Also add duration convertion for unwrap. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

The auth middleware was happening after the stats one and so org_id was not set 🤦. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

This patch extends the duration label filter with support for byte sizes such as `1kB` and `42MiB`.

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

…-parser

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

We should never advance an iterator in parallel. Unfortunately before the code was building iterators while advancing previous one, building iterator can advance iterator and thus creates a race condition. This changeset make sure we only fetch chunks in advance and build iterator and iterate over them in sequence. Also add support for labels in the cacheIterator which is required for logqlv2. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

This could cause Loki to crash if a panic happens in the store since it was happening in another goroutine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

benchmark before: ``` pkg: github.com/grafana/loki/pkg/logql/log Benchmark_Pipeline Benchmark_Pipeline-16 107580 10271 ns/op 6387 B/op 67 allocs/op PASS ok github.com/grafana/loki/pkg/logql/log 1.228s ``` after ``` pkg: github.com/grafana/loki/pkg/logql/log Benchmark_Pipeline Benchmark_Pipeline-16 199170 5670 ns/op 3307 B/op 54 allocs/op PASS ok github.com/grafana/loki/pkg/logql/log 1.201s ``` Way less allocations ! Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

codecov-io · 2020-10-19T12:24:57Z

Codecov Report

Merging #2778 into master will increase coverage by 0.04%.
The diff coverage is 66.94%.

@@            Coverage Diff             @@
##           master    #2778      +/-   ##
==========================================
+ Coverage   61.15%   61.19%   +0.04%     
==========================================
  Files         177      177              
  Lines       14278    14325      +47     
==========================================
+ Hits         8731     8766      +35     
- Misses       4737     4746       +9     
- Partials      810      813       +3

Impacted Files	Coverage Δ
pkg/logql/log/filter.go	`73.71% <0.00%> (ø)`
pkg/logql/log/pipeline.go	`13.63% <11.11%> (-2.16%)`	⬇️
pkg/logql/log/metrics_extraction.go	`51.66% <37.50%> (+0.81%)`	⬆️
pkg/logql/log/label_filter.go	`54.16% <38.88%> (+1.71%)`	⬆️
pkg/logql/functions.go	`19.26% <40.00%> (ø)`
pkg/logql/log/labels.go	`93.47% <93.47%> (-6.53%)`	⬇️
pkg/logql/ast.go	`81.79% <100.00%> (ø)`
pkg/logql/log/fmt.go	`77.41% <100.00%> (+1.98%)`	⬆️
pkg/logql/log/parser.go	`91.66% <100.00%> (-0.44%)`	⬇️
pkg/querier/queryrange/downstreamer.go	`95.29% <0.00%> (-2.36%)`	⬇️
... and 2 more

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

owen-d

Few nits then LGTM

pkg/logql/log/labels.go

slim-bean

LGTM for 4am 😉

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

cyriltovena and others added 30 commits October 1, 2020 20:38

Adds logfmt, regexp and json logql parser

03279b1

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

hook the ast with parsers.

31c2f3f

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

hook parser with memchunk.

eaf72bd

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

hook parser with the storage.

7d1dc7b

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

hook parser with ingesters

484afc1

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

fixes all tests

0121a3c

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Refactor to pipeline and implement ast parsing.

62f2829

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Fixes the lexer for duration and range

89c489c

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Fixes all tests and add some for label filters

4238173

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Add label and line format.

88ad104

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Add tests for fmt label and line with validations.

c7791a4

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Polishing parsers and add some more test cases

0651e25

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Finish the unwrap parser, still need to add more tests

4c0570d

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Indent this hell.

01e93c0

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Moar tests and it works.

e455c88

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Add more tests which lead me to find a bug in the lexer

8bc18e5

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Add more tests and fix all engine tests

08d2cf7

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Fixes match stage in promtail pipelines.

b801417

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Hook Pipeline into ingester, tailer and storage.

850b003

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Correctly setup sharding for logqlv2

31c26c0

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Fixes precedences issue with label filters and add moar tests ✌️

b5e11d0

Adds quantile_over_time, grouping for non associate range aggregation…

0fd6018

… parsing and moar tests

Extract with grouping

2ca6677

Adds parsing duration on unwrap

4effb67

Improve the lexer to support more common identifier as functions.

832a977

Also add duration convertion for unwrap. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Fixes the frontend logs to include org_id.

6563d6e

The auth middleware was happening after the stats one and so org_id was not set 🤦. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Merge branch 'fix-orgid-frontend' into logql-parser

92f7c39

Support byte sizes in label filters.

5578dbb

This patch extends the duration label filter with support for byte sizes such as `1kB` and `42MiB`.

Wip on error handling.

13132ad

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Fixes json parser with prometheus label name rules.

db07446

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

cyriltovena added 20 commits October 13, 2020 10:12

Wip error handling, commit before big refactoring.

c054a5d

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Merge branch 'logql-parser' of github.com:cyriltovena/loki into logql…

e7d8234

…-parser

Refactoring in progress.

5ab8b5c

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Work in progress.

5272d91

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Got something that builds and throw __error__ labels properly now.

1aa1609

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Add error handling + fixes groupins and post filtering.

1af9c14

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

400 on pipeline errors.

960ef5a

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Fixes a races in the log pipeline.

de83465

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Unsure the key is parsable and valid.

87c4f00

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Cleanup and code documentation.

50315b1

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Lint.

54dd6c2

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Lint.

a50490f

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Merge remote-tracking branch 'upstream/master' into logql-parser

cbbc37c

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Fixes frontend handler.

801b721

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Fixes old test.

1aee415

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Fix go1.15 local failing test.

1ea917f

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Handle panic in the store goroutine.

87c3c4b

This could cause Loki to crash if a panic happens in the store since it was happening in another goroutine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Now that races are gone we can use a global decoder.

20e9087

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

pull-request-size bot added the size/XXL label Oct 19, 2020

Fixes error label filtering.

a0b00a8

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

Merge remote-tracking branch 'upstream/master' into logqv2-optimization

14598ca

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

pull-request-size bot added size/XL and removed size/XXL labels Oct 20, 2020

owen-d reviewed Oct 20, 2020

View reviewed changes

pkg/logql/log/labels.go Outdated Show resolved Hide resolved

pkg/logql/log/labels.go Show resolved Hide resolved

slim-bean approved these changes Oct 21, 2020

View reviewed changes

Reviews comment.

b4610c4

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

cyriltovena merged commit dd5fceb into grafana:master Oct 21, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Logqv2 optimization #2778

Logqv2 optimization #2778

cyriltovena commented Oct 19, 2020

codecov-io commented Oct 19, 2020 •

edited

Loading

owen-d left a comment

slim-bean left a comment

Logqv2 optimization #2778

Logqv2 optimization #2778

Conversation

cyriltovena commented Oct 19, 2020

codecov-io commented Oct 19, 2020 • edited Loading

Codecov Report

owen-d left a comment

Choose a reason for hiding this comment

slim-bean left a comment

Choose a reason for hiding this comment

codecov-io commented Oct 19, 2020 •

edited

Loading