Improve JSON parser and add labels parser hints. #3080
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is a rework of the json parser, that now uses iteration over reflection allowing to avoid allocations. I've also implemented a way to guess and hints what labels needs to be parsed when doing metric queries, this means we can extract only the few labels required from the log line. benchcmp ``` ❯ benchcmp before.txt after.txt5 benchmark old ns/op new ns/op delta Benchmark_Parser/json/no_labels_hints-16 9889 3281 -66.82% Benchmark_Parser/logfmt/no_labels_hints-16 1624 1671 +2.89% Benchmark_Parser/logfmt/labels_hints-16 1601 790 -50.66% benchmark old allocs new allocs delta Benchmark_Parser/json/no_labels_hints-16 139 56 -59.71% Benchmark_Parser/logfmt/no_labels_hints-16 31 31 +0.00% Benchmark_Parser/logfmt/labels_hints-16 31 5 -83.87% benchmark old bytes new bytes delta Benchmark_Parser/json/no_labels_hints-16 3671 912 -75.16% Benchmark_Parser/logfmt/no_labels_hints-16 464 464 +0.00% Benchmark_Parser/logfmt/labels_hints-16 464 144 -68.97% ``` I've experienced a 2x to 4x improvement in my cluster. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
owen-d
reviewed
Jan 4, 2021
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Codecov Report
@@ Coverage Diff @@
## master #3080 +/- ##
==========================================
+ Coverage 62.88% 62.91% +0.03%
==========================================
Files 186 187 +1
Lines 15949 16089 +140
==========================================
+ Hits 10030 10123 +93
- Misses 4987 5029 +42
- Partials 932 937 +5
|
owen-d
approved these changes
Jan 5, 2021
cyriltovena
pushed a commit
to cyriltovena/loki
that referenced
this pull request
Jun 11, 2021
…fana#3080) Signed-off-by: Marco Pracucci <marco@pracucci.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a rework of the json parser, that now uses iteration over reflection allowing to avoid allocations.
I've also implemented a way to guess and hints what labels needs to be parsed when doing metric queries, this means we can extract only the few labels required from the log line.
benchcmp
I've experienced a 2x to 4x improvement in my cluster.
Signed-off-by: Cyril Tovena cyril.tovena@gmail.com