[CARBONDATA-4149] Fix query issues after alter add partition. #4107

ShreelekhyaG · 2021-03-15T11:52:12Z

Why is this PR needed?

Query with SI after add partition based on location on partition table gives incorrect results.

While pruning, if it's an external segment, it should use ExternalSegmentResolver , and no need to use ImplicitIncludeFilterExecutor as an external segment is not added in the SI table.
If the partition table has external partitions, after compaction the new files are loaded to the external path.
Data is not loaded to the child table(MV) after executing add partition command

What changes were proposed in this PR?

add path to loadMetadataDetails for external partition. It is used to identify it as an external segment.
After compaction, to not maintain any link to the external partition, the compacted files will be added as a new partition in the table. To update partition spec details in hive metastore, (drop partition + add partition) operations performed.
Add Load Pre and Post listener's in CarbonAlterTableAddHivePartitionCommand to trigger data load to materialized view.

Does this PR introduce any user interface change?

No

Is any new testcase added?

Yes

CarbonDataQA2 · 2021-03-15T14:04:26Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5568/

CarbonDataQA2 · 2021-03-15T14:13:34Z

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3802/

CarbonDataQA2 · 2021-03-17T21:34:56Z

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3811/

CarbonDataQA2 · 2021-03-17T21:39:13Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5577/

CarbonDataQA2 · 2021-03-18T09:45:19Z

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3815/

CarbonDataQA2 · 2021-03-18T09:48:34Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5581/

core/src/main/java/org/apache/carbondata/core/indexstore/ExtendedBlockletWrapper.java

core/src/main/java/org/apache/carbondata/core/readcommitter/TableStatusReadCommittedScope.java

...g/apache/spark/sql/execution/command/partition/CarbonAlterTableAddHivePartitionCommand.scala

core/src/main/java/org/apache/carbondata/core/index/Segment.java

integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonTableCompactor.scala

...rc/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestSIWithPartition.scala

integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonMergerRDD.scala

integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonTableCompactor.scala

CarbonDataQA2 · 2021-03-18T18:17:01Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5593/

CarbonDataQA2 · 2021-03-18T18:19:18Z

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3827/

CarbonDataQA2 · 2021-03-18T22:30:28Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5594/

CarbonDataQA2 · 2021-03-18T22:38:13Z

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3828/

integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonTableCompactor.scala

Indhumathi27 · 2021-03-19T06:51:08Z

@ShreelekhyaG please update PR description

update partitionspec path after compaction load data to mv after add partition rework fix for muliple partitions removed old,new

Indhumathi27 · 2021-03-19T08:58:46Z

LGTM

CarbonDataQA2 · 2021-03-19T10:21:12Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5602/

CarbonDataQA2 · 2021-03-19T10:25:16Z

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3836/

Why is this PR needed? Query with SI after add partition based on location on partition table gives incorrect results. 1. While pruning, if it's an external segment, it should use ExternalSegmentResolver , and no need to use ImplicitIncludeFilterExecutor as an external segment is not added in the SI table. 2. If the partition table has external partitions, after compaction the new files are loaded to the external path. 3. Data is not loaded to the child table(MV) after executing add partition command What changes were proposed in this PR? 1. add path to loadMetadataDetails for external partition. It is used to identify it as an external segment. 2. After compaction, to not maintain any link to the external partition, the compacted files will be added as a new partition in the table. To update partition spec details in hive metastore, (drop partition + add partition) operations performed. 3. Add Load Pre and Post listener's in CarbonAlterTableAddHivePartitionCommand to trigger data load to materialized view. Does this PR introduce any user interface change? No Is any new testcase added? Yes This closes apache#4107

ShreelekhyaG changed the title ~~[CARBONDATA-4149] Query with SI after add partition based on location on partition table gives incorrect results~~ [WIP] Query with SI after add partition based on location on partition table gives incorrect results Mar 16, 2021

ShreelekhyaG force-pushed the partitionAddSeg branch from 7146add to e7a92ed Compare March 17, 2021 19:30

ShreelekhyaG changed the title ~~[WIP] Query with SI after add partition based on location on partition table gives incorrect results~~ [CARBONDATA-4149] Query with SI after add partition based on location on partition table gives incorrect results Mar 18, 2021