Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Logqv2 optimization #2778

Merged
merged 56 commits into from
Oct 21, 2020
Merged

Logqv2 optimization #2778

merged 56 commits into from
Oct 21, 2020

Commits on Oct 1, 2020

  1. Adds logfmt, regexp and json logql parser 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    03279b1 View commit details
    Browse the repository at this point in the history

  2. hook the ast with parsers. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    31c2f3f View commit details
    Browse the repository at this point in the history

  3. hook parser with memchunk. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    eaf72bd View commit details
    Browse the repository at this point in the history

  4. hook parser with the storage. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    7d1dc7b View commit details
    Browse the repository at this point in the history

  5. hook parser with ingesters 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    484afc1 View commit details
    Browse the repository at this point in the history

  6. fixes all tests 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    0121a3c View commit details
    Browse the repository at this point in the history

  7. Refactor to pipeline and implement ast parsing. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    62f2829 View commit details
    Browse the repository at this point in the history

  8. Fixes the lexer for duration and range 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    89c489c View commit details
    Browse the repository at this point in the history

  9. Fixes all tests and add some for label filters 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    4238173 View commit details
    Browse the repository at this point in the history

  10. Add label and line format. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    88ad104 View commit details
    Browse the repository at this point in the history

  11. Add tests for fmt label and line with validations. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    c7791a4 View commit details
    Browse the repository at this point in the history

  12. Polishing parsers and add some more test cases 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    0651e25 View commit details
    Browse the repository at this point in the history

  13. Finish the unwrap parser, still need to add more tests 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    4c0570d View commit details
    Browse the repository at this point in the history

  14. Indent this hell. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    01e93c0 View commit details
    Browse the repository at this point in the history

  15. Moar tests and it works. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    e455c88 View commit details
    Browse the repository at this point in the history

  16. Add more tests which lead me to find a bug in the lexer 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    8bc18e5 View commit details
    Browse the repository at this point in the history

  17. Add more tests and fix all engine tests 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    08d2cf7 View commit details
    Browse the repository at this point in the history

  18. Fixes match stage in promtail pipelines. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    b801417 View commit details
    Browse the repository at this point in the history

  19. Hook Pipeline into ingester, tailer and storage. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    850b003 View commit details
    Browse the repository at this point in the history

  20. Correctly setup sharding for logqlv2 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    31c26c0 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2020

  1. Fixes precedences issue with label filters and add moar tests ✌️

    @cyriltovena
    cyriltovena committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    b5e11d0 View commit details
    Browse the repository at this point in the history

  2. Adds quantile_over_time, grouping for non associate range aggregation… 

    … parsing and moar tests
    @cyriltovena
    cyriltovena committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    0fd6018 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2020

  1. Extract with grouping

    @cyriltovena
    cyriltovena committed Oct 3, 2020
    Configuration menu
    Copy the full SHA
    2ca6677 View commit details
    Browse the repository at this point in the history

Commits on Oct 5, 2020

  1. Adds parsing duration on unwrap

    @cyriltovena
    cyriltovena committed Oct 5, 2020
    Configuration menu
    Copy the full SHA
    4effb67 View commit details
    Browse the repository at this point in the history

Commits on Oct 6, 2020

  1. Improve the lexer to support more common identifier as functions. 

    Also add duration convertion for unwrap.

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 6, 2020
    Configuration menu
    Copy the full SHA
    832a977 View commit details
    Browse the repository at this point in the history

  2. Fixes the frontend logs to include org_id. 

    The auth middleware was happening after the stats one and so org_id was not set 🤦.

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 6, 2020
    Configuration menu
    Copy the full SHA
    6563d6e View commit details
    Browse the repository at this point in the history

  3. Merge branch 'fix-orgid-frontend' into logql-parser

    @cyriltovena
    cyriltovena committed Oct 6, 2020
    Configuration menu
    Copy the full SHA
    92f7c39 View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2020

  1. Support byte sizes in label filters. 

    This patch extends the duration label filter with support for byte sizes
such as `1kB` and `42MiB`.
    @jeschkies
    jeschkies committed Oct 9, 2020
    Configuration menu
    Copy the full SHA
    5578dbb View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2020

  1. Wip on error handling. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 12, 2020
    Configuration menu
    Copy the full SHA
    13132ad View commit details
    Browse the repository at this point in the history

  2. Fixes json parser with prometheus label name rules. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 12, 2020
    Configuration menu
    Copy the full SHA
    db07446 View commit details
    Browse the repository at this point in the history

  3. fixup! Support byte sizes in label filters.

    @jeschkies
    jeschkies committed Oct 12, 2020
    Configuration menu
    Copy the full SHA
    78973cf View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'cyril/logql-parser' into karsten/bytes-… 

    …filter
    @jeschkies
    jeschkies committed Oct 12, 2020
    Configuration menu
    Copy the full SHA
    bbacba7 View commit details
    Browse the repository at this point in the history

  5. Merge pull request #5 from jeschkies/karsten/bytes-filter 

    Support byte sizes in label filters.
    @cyriltovena
    cyriltovena authored Oct 12, 2020
    Configuration menu
    Copy the full SHA
    25dd730 View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2020

  1. Wip error handling, commit before big refactoring. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 13, 2020
    Configuration menu
    Copy the full SHA
    c054a5d View commit details
    Browse the repository at this point in the history

  2. Merge branch 'logql-parser' of github.com:cyriltovena/loki into logql… 

    …-parser
    @cyriltovena
    cyriltovena committed Oct 13, 2020
    Configuration menu
    Copy the full SHA
    e7d8234 View commit details
    Browse the repository at this point in the history

  3. Refactoring in progress. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 13, 2020
    Configuration menu
    Copy the full SHA
    5ab8b5c View commit details
    Browse the repository at this point in the history

  4. Work in progress. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 13, 2020
    Configuration menu
    Copy the full SHA
    5272d91 View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2020

  1. Got something that builds and throw __error__ labels properly now. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 14, 2020
    Configuration menu
    Copy the full SHA
    1aa1609 View commit details
    Browse the repository at this point in the history

  2. Add error handling + fixes groupins and post filtering. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 14, 2020
    Configuration menu
    Copy the full SHA
    1af9c14 View commit details
    Browse the repository at this point in the history

  3. 400 on pipeline errors. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 14, 2020
    Configuration menu
    Copy the full SHA
    960ef5a View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2020

  1. Fixes a races in the log pipeline. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 15, 2020
    Configuration menu
    Copy the full SHA
    de83465 View commit details
    Browse the repository at this point in the history

  2. Unsure the key is parsable and valid. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 15, 2020
    Configuration menu
    Copy the full SHA
    87c4f00 View commit details
    Browse the repository at this point in the history

  3. Cleanup and code documentation. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 15, 2020
    Configuration menu
    Copy the full SHA
    50315b1 View commit details
    Browse the repository at this point in the history

  4. Lint. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 15, 2020
    Configuration menu
    Copy the full SHA
    54dd6c2 View commit details
    Browse the repository at this point in the history

  5. Lint. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 15, 2020
    Configuration menu
    Copy the full SHA
    a50490f View commit details
    Browse the repository at this point in the history

  6. Merge remote-tracking branch 'upstream/master' into logql-parser 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 15, 2020
    Configuration menu
    Copy the full SHA
    cbbc37c View commit details
    Browse the repository at this point in the history

  7. Fixes frontend handler. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 15, 2020
    Configuration menu
    Copy the full SHA
    801b721 View commit details
    Browse the repository at this point in the history

  8. Fixes old test. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 15, 2020
    Configuration menu
    Copy the full SHA
    1aee415 View commit details
    Browse the repository at this point in the history

  9. Fix go1.15 local failing test. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 15, 2020
    Configuration menu
    Copy the full SHA
    1ea917f View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2020

  1. Fixes race conditions in the batch iterator. 

    We should never advance an iterator in parallel. Unfortunately before the code was building iterators while advancing previous one, building iterator can advance iterator and thus creates a race condition. This changeset make sure we only fetch chunks in advance and build iterator and iterate over them in sequence.

Also add support for labels in the cacheIterator which is required for logqlv2.

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 16, 2020
    Configuration menu
    Copy the full SHA
    faecc80 View commit details
    Browse the repository at this point in the history

  2. Handle panic in the store goroutine. 

    This could cause Loki to crash if a panic happens in the store since it was happening in another goroutine.

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 16, 2020
    Configuration menu
    Copy the full SHA
    87c3c4b View commit details
    Browse the repository at this point in the history

  3. Now that races are gone we can use a global decoder. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 16, 2020
    Configuration menu
    Copy the full SHA
    20e9087 View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2020

  1. Improve labels handling in all LogQL stages. 

    benchmark before:

```
pkg: github.com/grafana/loki/pkg/logql/log
Benchmark_Pipeline
Benchmark_Pipeline-16    	  107580	     10271 ns/op	    6387 B/op	      67 allocs/op
PASS
ok  	github.com/grafana/loki/pkg/logql/log	1.228s
```

after
```
pkg: github.com/grafana/loki/pkg/logql/log
Benchmark_Pipeline
Benchmark_Pipeline-16    	  199170	      5670 ns/op	    3307 B/op	      54 allocs/op
PASS
ok  	github.com/grafana/loki/pkg/logql/log	1.201s
```

Way less allocations !

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 19, 2020
    Configuration menu
    Copy the full SHA
    40694fc View commit details
    Browse the repository at this point in the history

  2. Fixes error label filtering. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 19, 2020
    Configuration menu
    Copy the full SHA
    a0b00a8 View commit details
    Browse the repository at this point in the history

Commits on Oct 20, 2020

  1. Merge remote-tracking branch 'upstream/master' into logqv2-optimization 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 20, 2020
    Configuration menu
    Copy the full SHA
    14598ca View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2020

  1. Reviews comment. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Oct 21, 2020
    Configuration menu
    Copy the full SHA
    b4610c4 View commit details
    Browse the repository at this point in the history