add bloom filter index #1738
Closed
add bloom filter index #1738
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
imay
reviewed
Sep 4, 2019
gaodayue
requested changes
Sep 9, 2019
There was a problem hiding this comment.
Besides implementation, I have several questions about the design
- Why do we choose to store one BF per column per row block (1024 rows)? This seems to have a fairly high storage and search overhead. For example, let's say a column contains 10M rows and thus 10K blocks. Under the default fpp which 5%, the size of each BF is around 800 bytes. So the total size of this BF index is ~8MB which is very large(for comparison, the uncompressed size of int column with 10M rows is 40MB). To evaluate a "in" predicate with
nvalues,10KBFs need to be tested onnvalues, which could also take a long time. - Size of BF depends on
expected_entries, which is the number of unique values that will get inserted. Currentlyexpected_entriesis set to num_rows_per_block, which is too large for row block that contains many repeated values. How about allocating one BF for k distinct values instead of k rows? We can use the BF we're building to de-duplicate inputs. - Classic BF (the one Doris is currently using) is known to be cache unfriendly. Have you considered BlockedBloomFilter which is used by Parquet and RocksDB? For more information on BlockedBloomFilter, you can refer to https://github.com/apache/parquet-format/blob/master/BloomFilter.md
good suggestion! Tks |
kangpinghuang
force-pushed
the
add_bloom_filter
branch
from
b2eb947
to
853213e
Compare
imay
reviewed
Sep 24, 2019
| if (_num_inserted > _block_size) { | ||
| int flush_round = _num_inserted / _block_size; | ||
| for (int i = 0; i < flush_round; ++i) { | ||
| flush(); |
| Slice _data; | ||
| size_t _block_num; | ||
| uint32_t _expected_num; | ||
| std::vector<std::shared_ptr<BloomFilter>> _bloom_filters; |
kangpinghuang
force-pushed
the
add_bloom_filter
branch
from
853213e
to
167f9f1
Compare
SWJTU-ZhangLei
pushed a commit
to SWJTU-ZhangLei/incubator-doris
that referenced
this pull request
Jul 25, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add bloom filter index to prune data when loading segment