[CARBONDATA-2750] Updated documentation on Local Dictionary Supoort #2590
Conversation
| @@ -126,20 +126,20 @@ This tutorial is going to introduce all commands and data operations on CarbonDa | |||
|
|
|||
| - **Local Dictionary Configuration** | |||
|
|
|||
| Local Dictionary is generated only for no-dictionary string/varchar datatype columns. It helps in: | |||
| Local Dictionary is generated only for string/varchar datatype columns which are not included in dictionary include. It helps in: | |||
There was a problem hiding this comment.
Please add one Note and list which data type don't support.
There was a problem hiding this comment.
Add a small sentence on what local dictionary means
| | LOCAL_DICTIONARY_INCLUDE | all no-dictionary string/varchar columns | Columns for which Local Dictionary is generated. | | ||
| | LOCAL_DICTIONARY_ENABLE | false | By default, local dictionary will be disabled for the table | | ||
| | LOCAL_DICTIONARY_THRESHOLD | 10000 | The maximum cardinality for local dictionary generation (maximum - 100000) | | ||
| | LOCAL_DICTIONARY_INCLUDE | all string/varchar columns which are not included in dictionary include| Columns for which Local Dictionary is generated. | |
There was a problem hiding this comment.
"which are not included in dictionary include" -- please refine.
| @@ -126,20 +126,20 @@ This tutorial is going to introduce all commands and data operations on CarbonDa | |||
|
|
|||
| - **Local Dictionary Configuration** | |||
|
|
|||
| Local Dictionary is generated only for no-dictionary string/varchar datatype columns. It helps in: | |||
| Local Dictionary is generated only for string/varchar datatype columns which are not included in dictionary include. It helps in: | |||
| 1. Getting more compression on dimension columns with less cardinality. | |||
| 2. Filter queries and full scan queries on No-dictionary columns with local dictionary will be faster as filter will be done on encoded data. | |||
| 3. Reducing the store size and memory footprint as only unique values will be stored as part of local dictionary and corresponding data will be stored as encoded data. | |||
|
|
|||
There was a problem hiding this comment.
Please explain : what is the cost for enabling local dictionary.
| @@ -508,6 +511,9 @@ Users can specify which columns to include and exclude for local dictionary gene | |||
| ``` | |||
| ALTER TABLE tablename UNSET TBLPROPERTIES('LOCAL_DICTIONARY_ENABLE','LOCAL_DICTIONARY_THRESHOLD','LOCAL_DICTIONARY_INCLUDE','LOCAL_DICTIONARY_EXCLUDE') | |||
| ``` | |||
|
|
|||
| **NOTE:** For old tables, by default, local dictionary is disabled. If user wants local dictionary, he/she can enable/disable local dictionary for new data on those tables at his/her discretion. | |||
There was a problem hiding this comment.
"he/she" change to "user"
praveenmeenakshi56
force-pushed
the
local_doc
branch
from
540066d
to
44ad8fb
Compare
|
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6078/ |
|
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6396/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7675/ |
|
|
||
| By default, Local Dictionary will be enabled and generated for all no-dictionary string/varchar datatype columns. | ||
|
|
||
| **Bottleneck for Local Dictionary:** The memory size will increase when local dictionary is enabled. |
There was a problem hiding this comment.
Please change "bottleneck" to "The cost"
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7683/ |
|
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6407/ |
|
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6083/ |
praveenmeenakshi56
force-pushed
the
local_doc
branch
from
44ad8fb
to
54e299f
Compare
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7708/ |
|
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6434/ |
|
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6100/ |
| @@ -126,20 +126,20 @@ This tutorial is going to introduce all commands and data operations on CarbonDa | |||
|
|
|||
| - **Local Dictionary Configuration** | |||
|
|
|||
| Local Dictionary is generated only for no-dictionary string/varchar datatype columns. It helps in: | |||
| Local Dictionary is generated only for string/varchar datatype columns which are not included in dictionary include. It helps in: | |||
There was a problem hiding this comment.
Add a small sentence on what local dictionary means
| @@ -126,20 +126,33 @@ This tutorial is going to introduce all commands and data operations on CarbonDa | |||
|
|
|||
| - **Local Dictionary Configuration** | |||
|
|
|||
| Local Dictionary is generated only for no-dictionary string/varchar datatype columns. It helps in: | |||
| Local Dictionary is generated only for string/varchar datatype columns which are not included in dictionary include. It helps in: | |||
| 1. Getting more compression on dimension columns with less cardinality. | |||
| 2. Filter queries and full scan queries on No-dictionary columns with local dictionary will be faster as filter will be done on encoded data. | |||
|
|
||
| By default, Local Dictionary will be enabled and generated for all no-dictionary string/varchar datatype columns. | ||
|
|
||
| **The cost for Local Dictionary:** The memory size will increase when local dictionary is enabled. |
There was a problem hiding this comment.
can add a sentence as to why it will increase
| | LOCAL_DICTIONARY_THRESHOLD | 10000 | The maximum cardinality for local dictionary generation (range- 1000 to 100000) | | ||
| | LOCAL_DICTIONARY_INCLUDE | all no-dictionary string/varchar columns | Columns for which Local Dictionary is generated. | | ||
| | LOCAL_DICTIONARY_ENABLE | false | By default, local dictionary will be disabled for the table | | ||
| | LOCAL_DICTIONARY_THRESHOLD | 10000 | The maximum cardinality for local dictionary generation (maximum - 100000) | |
There was a problem hiding this comment.
description not correct. need to explain what threshold means and what happens beyond threshold
| | LOCAL_DICTIONARY_INCLUDE | all no-dictionary string/varchar columns | Columns for which Local Dictionary is generated. | | ||
| | LOCAL_DICTIONARY_ENABLE | false | By default, local dictionary will be disabled for the table | | ||
| | LOCAL_DICTIONARY_THRESHOLD | 10000 | The maximum cardinality for local dictionary generation (maximum - 100000) | | ||
| | LOCAL_DICTIONARY_INCLUDE | all string/varchar columns not specified in dictionary include| Columns for which Local Dictionary is generated. | |
There was a problem hiding this comment.
if i don't specify this property, what is the behaviour?
| | LOCAL_DICTIONARY_INCLUDE | all no-dictionary string/varchar columns | Columns for which Local Dictionary is generated. | | ||
| | LOCAL_DICTIONARY_ENABLE | false | By default, local dictionary will be disabled for the table | | ||
| | LOCAL_DICTIONARY_THRESHOLD | 10000 | The maximum cardinality for local dictionary generation (maximum - 100000) | | ||
| | LOCAL_DICTIONARY_INCLUDE | all string/varchar columns not specified in dictionary include| Columns for which Local Dictionary is generated. | | ||
| | LOCAL_DICTIONARY_EXCLUDE | none | Columns for which Local Dictionary is not generated | | ||
|
|
||
| **NOTE:** If the cardinality exceeds the threshold, this column will not use local dictionary encoding. And in this case, the data loading performance will decrease since there is a rollback procedure for local dictionary encoding. |
There was a problem hiding this comment.
For line 149 (162):
Encoded data and Actual data are both stored when Local Dictionary is enabled.
please change it to:
Encoded data with & without Local dictionary are both stored when Local Dictionary is enabled during data loading, so it requires more memory than before.
| @@ -170,6 +183,9 @@ This tutorial is going to introduce all commands and data operations on CarbonDa | |||
| TBLPROPERTIES('LOCAL_DICTIONARY_ENABLE'='true','LOCAL_DICTIONARY_THRESHOLD'='1000', | |||
| 'LOCAL_DICTIONARY_INCLUDE'='column1','LOCAL_DICTIONARY_EXCLUDE'='column2') | |||
| ``` | |||
|
|
|||
There was a problem hiding this comment.
sentence not easy to understand. need simpler language to explain the reason
| @@ -524,6 +540,9 @@ Users can specify which columns to include and exclude for local dictionary gene | |||
| ``` | |||
| ALTER TABLE tablename UNSET TBLPROPERTIES('LOCAL_DICTIONARY_ENABLE','LOCAL_DICTIONARY_THRESHOLD','LOCAL_DICTIONARY_INCLUDE','LOCAL_DICTIONARY_EXCLUDE') | |||
| ``` | |||
|
|
|||
| **NOTE:** For old tables, by default, local dictionary is disabled. If user wants local dictionary, user can enable/disable local dictionary for new data on those tables at their discretion. | |||
There was a problem hiding this comment.
local dictionary is disabled for new tables also.Need to mention it can be enabled for old tables also
| * DECIMAL | ||
| * TIMESTAMP | ||
| * DATE | ||
| * CHAR |
There was a problem hiding this comment.
Why is CHAR not supported? As I know, SparkSQL treat both varchar and char as string, so in carbon data we actually see string.
There was a problem hiding this comment.
what about complex? you didn't mention it
| | LOCAL_DICTIONARY_INCLUDE | all no-dictionary string/varchar columns | Columns for which Local Dictionary is generated. | | ||
| | LOCAL_DICTIONARY_ENABLE | false | By default, local dictionary will be disabled for the table | | ||
| | LOCAL_DICTIONARY_THRESHOLD | 10000 | The maximum cardinality for local dictionary generation (maximum - 100000) | | ||
| | LOCAL_DICTIONARY_INCLUDE | all string/varchar columns not specified in dictionary include| Columns for which Local Dictionary is generated. | | ||
| | LOCAL_DICTIONARY_EXCLUDE | none | Columns for which Local Dictionary is not generated | | ||
|
|
||
| **NOTE:** If the cardinality exceeds the threshold, this column will not use local dictionary encoding. And in this case, the data loading performance will decrease since there is a rollback procedure for local dictionary encoding. |
There was a problem hiding this comment.
For line 149 (162):
Encoded data and Actual data are both stored when Local Dictionary is enabled.
please change it to:
Encoded data with & without Local dictionary are both stored when Local Dictionary is enabled during data loading, so it requires more memory than before.
83930b5
to
ebd75b8
Compare
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7741/ |
|
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6466/ |
|
retest this please |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7747/ |
|
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6472/ |
praveenmeenakshi56
force-pushed
the
local_doc
branch
from
ebd75b8
to
0523b96
Compare
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7759/ |
|
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6484/ |
|
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6130/ |
praveenmeenakshi56
force-pushed
the
local_doc
branch
from
0523b96
to
9c7960e
Compare
|
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6132/ |
|
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6491/ |
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7767/ |
|
Lgtm |
|
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6143/ |
|
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6150/ |
|
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6160/ |
|
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6502/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7778/ |
praveenmeenakshi56
force-pushed
the
local_doc
branch
3 times, most recently
from
9c44be5
to
f41e5bb
Compare
|
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6161/ |
|
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6162/ |
|
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6163/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7780/ |
|
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6505/ |
praveenmeenakshi56
force-pushed
the
local_doc
branch
from
f41e5bb
to
50afb0b
Compare
Conflicts: docs/data-management-on-carbondata.md
praveenmeenakshi56
force-pushed
the
local_doc
branch
from
50afb0b
to
6d6dd3b
Compare
|
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6169/ |
|
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6515/ |
|
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6170/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7790/ |
|
LGTM |
Updated Documentation on Local Dictionary Support. Changed default scenario for Local dictionary to false This closes apache#2590
Updated Documentation on Local Dictionary Support. Changed default scenario for Local dictionary to false This closes #2590
Updated Documentation on Local Dictionary Support. Changed default scenario for Local dictionary to false.
NOTE: The Bad Records path was moved in between Local Dictionary documentation in another PR. In this, that has just been moved into Bad Records Handling
NA
NA
Document Updated
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
NA
NA