Shrink bloom filters when serializing partitions #1172
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tobim
reviewed
Nov 16, 2020
lava
force-pushed
the
topic/bloom-filter-sizing
branch
3 times, most recently
from
bb70f24
to
d2bd1ee
Compare
So 50-80% reduction in size? |
lava
force-pushed
the
topic/bloom-filter-sizing
branch
from
d2bd1ee
to
b26b66b
Compare
tobim
requested changes
Nov 18, 2020
lava
force-pushed
the
topic/bloom-filter-sizing
branch
2 times, most recently
from
f65da53
to
de33222
Compare
lava
force-pushed
the
topic/bloom-filter-sizing
branch
2 times, most recently
from
2aad26d
to
1d09499
Compare
tobim
reviewed
Nov 20, 2020
tobim
reviewed
Nov 20, 2020
tobim
reviewed
Nov 20, 2020
tobim
reviewed
Nov 20, 2020
lava
force-pushed
the
topic/bloom-filter-sizing
branch
from
5c41d8b
to
6ef3a77
Compare
When creating a new address synopsis structure, it is sized for the worst-case scenario of every event in the synopsis having a unique ip address. When finalizing a synopsis, the exact number of unique ips is known and the minimal size for the synopsis can be computed, so that's what we do.
The `bloom_filter_parameters` class has no input validation, so users have to rely on the `evaluate` function to see if a given combination of parameters is valid. However, this function can die with an assertion failure if some parameters were set incorrectly, creating a catch-22 situation. Additionally, this meant that these checks were not performed in release builds with assertions disabled, potentially leading to bloom filters with nonsensical settings.
lava
force-pushed
the
topic/bloom-filter-sizing
branch
2 times, most recently
from
ea443e7
to
0211bd7
Compare
Use a std::set over a std::vector + bloom_filter combination, since the latter confused multiple independent reviewers. This also has the benefit of allowing for stricter checks on where and how the buffered_address_synopsis is used, and we don't have to rely on the `type` member being annotated with stringified bloom filter parameters. Also, pass the actor handle of the index together with the persist atom, as opposed to when spawning a partition actor.
lava
force-pushed
the
topic/bloom-filter-sizing
branch
from
0211bd7
to
5c42cde
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
馃摂 Description
Shrink bloom filters when storing partitions.
Running this on the 5GiB suricata dataset and comparing against a CI build from latest master there seems to be a reduction of 10-15 MiB per partition, so a significant chunk of the 19 MiB we expect for synopsis bloom filters in total.
Looking at the address synopsis in particular: (for reference, all of these were exactly
1.2MiBbefore the changes in this PR)馃摑 Checklist
馃幆 Review Instructions