Skip to content

Commit

Permalink
[CARBONDATA-2800][Doc] Add useful tips about bloomfilter datamap
Browse files Browse the repository at this point in the history
add useful tips about bloomfilter datamap

This closes #2581
  • Loading branch information
xuchuanyin authored and ravipesala committed Aug 9, 2018
1 parent b2972ce commit 415b18f
Show file tree
Hide file tree
Showing 2 changed files with 30 additions and 1 deletion.
27 changes: 26 additions & 1 deletion docs/datamap/bloomfilter-datamap-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
* [Loading Data](#loading-data)
* [Querying Data](#querying-data)
* [Data Management](#data-management-with-bloomfilter-datamap)
* [Useful Tips](#useful-tips)

#### DataMap Management
Creating BloomFilter DataMap
Expand Down Expand Up @@ -102,4 +103,28 @@ which will show the transformed logical plan, and thus user can check whether th
If the datamap does not prune blocklets well, you can try to increase the value of property `BLOOM_SIZE` and decrease the value of property `BLOOM_FPP`.

## Data Management With BloomFilter DataMap
Data management with BloomFilter datamap has no difference with that on Lucene datamap. You can refer to the corresponding section in `CarbonData BloomFilter DataMap`.
Data management with BloomFilter datamap has no difference with that on Lucene datamap.
You can refer to the corresponding section in `CarbonData Lucene DataMap`.

## Useful Tips
+ BloomFilter DataMap is suggested to be created on the high cardinality columns.
Query conditions on these columns are always simple `equal` or `in`,
such as 'col1=XX', 'col1 in (XX, YY)'.
+ We can create multiple BloomFilter datamaps on one table,
but we do recommend you to create one BloomFilter datamap that contains multiple index columns,
because the data loading and query performance will be better.
+ `BLOOM_FPP` is only the expected number from user, the actually FPP may be worse.
If the BloomFilter datamap does not work well,
you can try to increase `BLOOM_SIZE` and decrease `BLOOM_FPP` at the same time.
Notice that bigger `BLOOM_SIZE` will increase the size of index file
and smaller `BLOOM_FPP` will increase runtime calculation while performing query.
+ '0' skipped blocklets of BloomFilter datamap in explain output indicates that
BloomFilter datamap does not prune better than Main datamap.
(For example since the data is not ordered, a specific value may be contained in many blocklets. In this case, bloom may not work better than Main DataMap.)
If this occurs very often, it means that current BloomFilter is useless. You can disable or drop it.
Sometimes we cannot see any pruning result about BloomFilter datamap in the explain output,
this indicates that the previous datamap has pruned all the blocklets and there is no need to continue pruning.
+ In some scenarios, the BloomFilter datamap may not enhance the query performance significantly
but if it can reduce the number of spark task,
there is still a chance that BloomFilter datamap can enhance the performance for concurrent query.
+ Note that BloomFilter datamap will decrease the data loading performance and may cause slightly storage expansion (for datamap index file).
4 changes: 4 additions & 0 deletions docs/useful-tips-on-carbondata.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,10 @@
TBLPROPERTIES ('SORT_COLUMNS'='Dime_1, HOST, MSISDN')
```

**NOTE:**
+ BloomFilter can be created to enhance performance for queries with precise equal/in conditions. You can find more information about it in BloomFilter datamap [document](https://github.com/apache/carbondata/blob/master/docs/datamap/bloomfilter-datamap-guide.md).


## Configuration for Optimizing Data Loading performance for Massive Data


Expand Down

0 comments on commit 415b18f

Please sign in to comment.