[CARBONDATA-2454][DataMap] Add fpp property for bloom datamap #2279
Conversation
add fpp(false positive probability) property to configure bloom filter that used by bloom datamap.
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5722/ |
|
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4561/ |
| /** | ||
| * property for fpp(false-positive-probability) of bloom filter | ||
| */ | ||
| private static final String BLOOM_FPP = "bloom_fpp"; |
There was a problem hiding this comment.
Do user need to configure this? What is the relationship with DEFAULT_BLOOM_FILTER_SIZE
There was a problem hiding this comment.
yes, it is an configuration for bloom filter. In the previous implementation, p is
a fixed value 0.00001.
If
n : bloomfilterSize
p : fpp
then
p will decide the number of hash functions used internally by bloom filter
n and p together decide the length of byte used internally by bloom filter
There was a problem hiding this comment.
Can you add a testcase for it
There was a problem hiding this comment.
User can always control the bloom filter memory size by setting n?
There was a problem hiding this comment.
yes, by setting n and p, user can control the bloomfiltersize (for memory) and the number of hash function(for cpu).
There was a problem hiding this comment.
Testcase will be added after your PR #2255 is merged. It makes lots of changes to the code.
|
LGTM |
add fpp(false positive probability) property to configure bloom filter that used by bloom datamap. This closes apache#2279
add fpp(false positive probability) property to configure bloom filter
that used by bloom datamap.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
changed only internal used interfacesNONOPlease provide details on
- Whether new unit test cases have been added or why no new tests are required?
Tests will be added in the future- How it is tested? Please attach test report.
tested in standalone 3-node cluster- Is it a performance related change? Please attach the performance test report.
Yes, proper fpp can improve the query performance- Any additional information to help reviewers in testing this change.
NANA