Skip to content
This repository was archived by the owner on Jul 19, 2023. It is now read-only.

Conversation

cyriltovena
Copy link
Collaborator

This adds a flush benchmark and an optimization to reduce memory usage.

❯ benchstat before.txt after.txt                      
name      old time/op    new time/op    delta
Flush-16     400ms ± 8%     249ms ±10%  -37.76%  (p=0.008 n=5+5)

name      old alloc/op   new alloc/op   delta
Flush-16    1.03GB ± 0%    0.20GB ± 1%  -80.12%  (p=0.008 n=5+5)

name      old allocs/op  new allocs/op  delta
Flush-16     8.26k ± 1%     3.43k ± 1%  -58.54%  (p=0.008 n=5+5)

@simonswine
Copy link
Collaborator

We also need to call Flush() to ensure the rowGroup boundaries are still respected.

So that one segment in becomes one row group in the final file. Otherwise the ordering is incorrect. Still significant win:

❯ benchstat before.txt after2.txt
name     old time/op    new time/op    delta
Flush-8     533ms ±13%     294ms ± 6%  -44.84%  (p=0.008 n=5+5)

name     old alloc/op   new alloc/op   delta
Flush-8    1.03GB ± 0%    0.20GB ± 1%  -80.64%  (p=0.016 n=4+5)

name     old allocs/op  new allocs/op  delta
Flush-8     8.00k ± 0%     5.07k ± 1%  -36.62%  (p=0.016 n=4+5)

Copy link
Collaborator

@simonswine simonswine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your bit LGTM

@cyriltovena cyriltovena merged commit a3190eb into main Mar 21, 2023
@cyriltovena cyriltovena deleted the improve-flush-alloc branch March 21, 2023 10:30
simonswine added a commit to simonswine/pyroscope that referenced this pull request Jun 30, 2023
* Improve profiles flushing using ReadRowsFrom

* Updates comment

* Flush to ensure to cut a row group

---------

Co-authored-by: Christian Simon <simon@swine.de>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants