[CARBONDATA-2016] Exception displays while executing compaction with alter query #1839
Conversation
|
@jackylk i accidently deleted my old branch i have created this new branch for same issue please merge this one |
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3000/ |
|
retest this please |
|
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1770/ |
|
retest this please |
|
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1774/ |
|
retest this please |
|
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1775/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3005/ |
anubhav100
force-pushed
the
bugfix/CARBONDATA-2016
branch
from
eee0a23
to
591e9b0
Compare
|
@jackylk please review |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3121/ |
|
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1889/ |
|
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3100/ |
| "alter table customer1 add columns (longfield bigint) TBLPROPERTIES ('DEFAULT.VALUE.longfield'='10')") | ||
|
|
||
| sql("alter table customer1 compact 'minor' ").show() | ||
| assert(true) |
There was a problem hiding this comment.
- Remove try catch block..its not required..in case of any exception anyways the test case will fail.
- In the add column DDL's. give the default value as MAX value of the corresponding datatype so that the same is validated after compaction in the select query validation.
- After compaction match the newly added column values as select query using checkAnswer.
There was a problem hiding this comment.
@manishgupta88 done
| @@ -360,6 +393,8 @@ class HorizontalCompactionTestCase extends QueryTest with BeforeAndAfterAll { | |||
| CarbonProperties.getInstance() | |||
| .addProperty(CarbonCommonConstants.isHorizontalCompactionEnabled , "true") | |||
| sql("""drop table if exists t_carbn01""") | |||
| sql("""drop table if exists customer1""") | |||
|
|
|||
There was a problem hiding this comment.
@manishgupta88 done you can review
Reason: When we apply the alter table command to add column with default value it is always storing it as long object for all measures,it is wrongly written in restructure util we should return the value as the same type as that of the measure,it was causing the compaction to fail with class cast exception because the data type and its corresponding value does not have same data type Solution:Correct the wrong logic in restructure util the type of returning value object should be same as that of measure
anubhav100
force-pushed
the
bugfix/CARBONDATA-2016
branch
from
591e9b0
to
d30cf9c
Compare
|
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1979/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3213/ |
|
LGTM |
|
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3174/ |
…alter query Reason: When we apply the alter table command to add column with default value it is always storing it as long object for all measures,it is wrongly written in restructure util we should return the value as the same type as that of the measure,it was causing the compaction to fail with class cast exception because the data type and its corresponding value does not have same data type Solution: Correct the wrong logic in restructure util the type of returning value object should be same as that of measure This closes apache#1839
Root Cause
When we apply the alter table command to add column with default value it is storing it as long object,it is wrongly written in restructure util we should get the value as the same type
as that of datatype of columnschema in restructure util earlier in master branch in restuctureutil class if our data type is long or short or int we are always returning back a long object which is wong due to same reason compaction was failing
if it was applied after alter table add columns command with default value because in sortdatarows there was mismatch between data type and its corresponding value
Testing:
1.mvn clean install is passing
2.added new test case for same