Append to small bz1 blocks #3735
Conversation
|
@benbjohnson -- rebase your changes to pick up the test I have skipped. |
benbjohnson
force-pushed
the
append-threshold
branch
2 times, most recently
from
5313724
to
06eb4b9
Compare
|
@otoolep I rebased and updated the |
|
Is this an optimization? It that its purpose? |
|
@otoolep flushes can happen from the WAL due to memory pressure. The goal of this is to ensure that compressed blocks are big enough to actually get value from the compression. If you compress a single data point, your ratio won't be nearly as good as when you compress 1,000. Also, lumping these together lessons the number of unique keys in the DB. |
|
I see, a fix, makes sense @pauldix |
|
@benbjohnson does the quick test cover all three scenarios? If so, +1 |
|
@pauldix It should but I'll add a more explicit append test for it. |
|
+1 once that test is in. Also good to get this cherry picked into the 0.9.3 branch. |
benbjohnson
force-pushed
the
append-threshold
branch
2 times, most recently
from
207b058
to
f7ae6f8
Compare
This commit changes the bz1 append to check for a small ending block first. If the block is below the threshold for block size then it is rewritten with the new data points instead of having a new block written.
benbjohnson
force-pushed
the
append-threshold
branch
from
f7ae6f8
to
e57d602
Compare
Append to small bz1 blocks
Overview
This pull request changes the
bz1append to check for a small ending block first. If the block is below the threshold for block size then it is rewritten with the new data points instead of having a new block written./cc @pauldix