Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Uses custom json-iter decoder for log entries. #3163

Merged
merged 2 commits into from
Jan 19, 2021
Merged

Uses custom json-iter decoder for log entries. #3163

merged 2 commits into from
Jan 19, 2021

Commits on Jan 12, 2021

  1. Uses custom json-iter decoder for log entries. 

    Previously we were using json.Unmarshal for each line. However json-iter uses a Pool for each calls and I believe this can cause to increase memory usage.

For each line we would put in a pool the iterator to re-use it, once put in a pool, the last data is retained, since we handle millions of lines, this can cause problem, using a custom extensions, keep using a pool but at the root object only, not for each line.

On top of that we're going to process that json payload 50% faster.

```
❯ benchcmp  before.txt after.txt2
benchmark                          old ns/op     new ns/op     delta
Benchmark_DecodePushRequest-16     13509236      6677037       -50.57%
benchmark                          old allocs     new allocs     delta
Benchmark_DecodePushRequest-16     106149         38719          -63.52%
benchmark                          old bytes     new bytes     delta
Benchmark_DecodePushRequest-16     10350362      5222989       -49.54%
```

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Jan 12, 2021
    Configuration menu
    Copy the full SHA
    7385735 View commit details
    Browse the repository at this point in the history

  2. Fixes query-tee tests. 

    Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
    @cyriltovena
    cyriltovena committed Jan 12, 2021
    Configuration menu
    Copy the full SHA
    1e6544e View commit details
    Browse the repository at this point in the history