[CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed. #4104

liuhe0702 · 2021-03-10T13:15:10Z

Why is this PR needed?
During compact execution, the status of the new segment is set to success before index files are merged.
After index files are merged, the carbonindex files are deleted.
As a result, the query task cannot find the cached carbonindex files.

What changes were proposed in this PR?
Set the status of the new segment to succeeded after index files are merged.

Does this PR introduce any user interface change?
No

Is any new testcase added?
No

CarbonDataQA2 · 2021-03-10T13:17:08Z

Can one of the admins verify this patch?

brijoobopanna · 2021-03-11T04:56:32Z

add to whitelist

CarbonDataQA2 · 2021-03-11T07:25:36Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3780/

CarbonDataQA2 · 2021-03-11T07:26:03Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5545/

…e status" is displayed. query is normal after the "drop metacache on table" command is executed. Why is this PR needed? During compact execution, the status of the new segment is set to success before index files are merged. After index files are merged, the carbonindex files are deleted. As a result, the query task cannot find the cached carbonindex files. What changes were proposed in this PR? Set the status of the new segment to succeeded after index files are merged. Does this PR introduce any user interface change? No Is any new testcase added? No

CarbonDataQA2 · 2021-03-11T12:04:16Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5554/

CarbonDataQA2 · 2021-03-11T12:08:25Z

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3788/

liuhe0702 · 2021-03-12T00:26:19Z

retest this please

CarbonDataQA2 · 2021-03-12T02:26:25Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3792/

CarbonDataQA2 · 2021-03-12T02:28:59Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5558/

liuhe0702 · 2021-03-12T03:06:16Z

retest this please

CarbonDataQA2 · 2021-03-12T05:08:08Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3794/

CarbonDataQA2 · 2021-03-12T05:18:25Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5560/

liuhe0702 · 2021-03-12T06:02:45Z

retest this please

CarbonDataQA2 · 2021-03-12T08:09:35Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5561/

CarbonDataQA2 · 2021-03-12T08:14:01Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3795/

liuhe0702 · 2021-03-12T09:06:06Z

retest this please

CarbonDataQA2 · 2021-03-12T11:19:09Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5563/

CarbonDataQA2 · 2021-03-12T11:27:13Z

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3797/

liuhe0702 · 2021-03-12T11:30:10Z

retest this please

CarbonDataQA2 · 2021-03-12T13:40:35Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5565/

CarbonDataQA2 · 2021-03-12T13:45:30Z

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3799/

akashrn5 · 2021-03-16T07:35:41Z

@liuhe0702 i think this issue is already being handled in #3988 , so i think no need of this PR, please refer the jira to the jira of #3988 and close as duplicate. @ShreelekhyaG please confirm the same here, where this scenario is being handled or not in your PR.

ShreelekhyaG · 2021-03-16T07:39:39Z

Yes @akashrn5/ @liuhe0702 , the same scenario being handled in PR #3988

akashrn5 · 2021-03-16T07:41:24Z

@liuhe0702 please make the jira as duplicate and comment there as its handled in #3988 and close both jira and PR, thanks

ShreelekhyaG · 2021-03-19T06:09:11Z

...ing/src/main/java/org/apache/carbondata/processing/merger/CompactionResultSortProcessor.java

@@ -190,7 +200,7 @@ public boolean execute(List<RawResultIterator> unsortedResultIteratorList,
      LOGGER.error(e.getLocalizedMessage(), e);
      throw e;
    } finally {
-      if (partitionSpec != null) {


Instead of passing boolean value, cant we directly get CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT from properties? and the if check should be - if merge index not enabled, then write segment file. As below,

``` boolean isMergeIndexEnable = Boolean.parseBoolean(CarbonProperties.getInstance().getProperty( CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT, CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT_DEFAULT)); if (partitionSpec != null && !isMergeIndexEnable) { ```

yes, @liuhe0702 instead of making parameter changes or adding constructors, check property here only and can avoid changes in MergerRDD

CarbonDataQA2 · 2021-03-19T10:53:00Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5603/

CarbonDataQA2 · 2021-03-19T10:55:43Z

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3837/

CarbonDataQA2 · 2021-03-21T17:06:06Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5059/

CarbonDataQA2 · 2021-03-21T17:14:39Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3307/

liuhe0702 · 2021-03-22T00:56:03Z

retest this please

CarbonDataQA2 · 2021-03-22T02:28:17Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5069/

CarbonDataQA2 · 2021-03-22T02:36:05Z

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3316/

liuhe0702 · 2021-03-22T03:33:37Z

retest this please

CarbonDataQA2 · 2021-03-22T05:07:28Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5072/

akashrn5 · 2021-03-22T05:16:48Z

...ing/src/main/java/org/apache/carbondata/processing/merger/CompactionResultSortProcessor.java

+          CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT_DEFAULT));
+      // mergeIndex is true, the segment file not need to be wrotten
+      // and will be wrotten during merging index
+      if (partitionSpec != null && !isMergeIndex) {


wrotten to written

CarbonDataQA2 · 2021-03-22T05:19:41Z

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3319/

…e status" is displayed. query is normal after the "drop metacache on table" command is executed. not write temp segment file when mergeindex is true

CarbonDataQA2 · 2021-03-22T08:09:43Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5074/

CarbonDataQA2 · 2021-03-22T08:17:04Z

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3321/

liuhe0702 · 2021-03-22T08:28:50Z

retest this please

CarbonDataQA2 · 2021-03-22T10:00:06Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5076/

CarbonDataQA2 · 2021-03-22T10:03:53Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3323/

liuhe0702 · 2021-03-22T11:19:01Z

retest this please

CarbonDataQA2 · 2021-03-22T12:51:50Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5077/

CarbonDataQA2 · 2021-03-22T12:56:53Z

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3324/

ShreelekhyaG · 2021-03-23T09:32:36Z

LGTM

akashrn5 · 2021-03-23T09:45:48Z

LGTM

…e status" is displayed. query is normal after the "drop metacache on table" command is executed. Why is this PR needed? During compact execution, the status of the new segment is set to success before index files are merged. After index files are merged, the carbonindex files are deleted. As a result, the query task cannot find the cached carbonindex files. What changes were proposed in this PR? Set the status of the new segment to succeeded after index files are merged. Does this PR introduce any user interface change? No Is any new testcase added? No This closes apache#4104

liuhe0702 force-pushed the MergeIndexFilesFirst branch from 2e79296 to 57999ab Compare March 11, 2021 09:56

liuhe0702 closed this Mar 17, 2021

liuhe0702 reopened this Mar 18, 2021

ShreelekhyaG reviewed Mar 19, 2021

View reviewed changes

liuhe0702 force-pushed the MergeIndexFilesFirst branch 3 times, most recently from 5c197b4 to 937f644 Compare March 19, 2021 08:46

akashrn5 reviewed Mar 22, 2021

View reviewed changes

[CARBONDATA-4146]Query fails and the error message "unable to get fil…

f4c4c0c

…e status" is displayed. query is normal after the "drop metacache on table" command is executed. not write temp segment file when mergeindex is true

liuhe0702 force-pushed the MergeIndexFilesFirst branch from 937f644 to f4c4c0c Compare March 22, 2021 06:34

asfgit closed this in 6ab3647 Mar 23, 2021

[CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed. #4104

[CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed. #4104

Conversation

liuhe0702 commented Mar 10, 2021

CarbonDataQA2 commented Mar 10, 2021

brijoobopanna commented Mar 11, 2021

CarbonDataQA2 commented Mar 11, 2021

CarbonDataQA2 commented Mar 11, 2021

CarbonDataQA2 commented Mar 11, 2021

CarbonDataQA2 commented Mar 11, 2021

liuhe0702 commented Mar 12, 2021

CarbonDataQA2 commented Mar 12, 2021

CarbonDataQA2 commented Mar 12, 2021

liuhe0702 commented Mar 12, 2021

CarbonDataQA2 commented Mar 12, 2021

CarbonDataQA2 commented Mar 12, 2021

liuhe0702 commented Mar 12, 2021

CarbonDataQA2 commented Mar 12, 2021

CarbonDataQA2 commented Mar 12, 2021

liuhe0702 commented Mar 12, 2021

CarbonDataQA2 commented Mar 12, 2021

CarbonDataQA2 commented Mar 12, 2021

liuhe0702 commented Mar 12, 2021

CarbonDataQA2 commented Mar 12, 2021

CarbonDataQA2 commented Mar 12, 2021

akashrn5 commented Mar 16, 2021 • edited

ShreelekhyaG commented Mar 16, 2021

akashrn5 commented Mar 16, 2021

ShreelekhyaG Mar 19, 2021

Choose a reason for hiding this comment

akashrn5 Mar 19, 2021

Choose a reason for hiding this comment

liuhe0702 Mar 19, 2021

Choose a reason for hiding this comment

CarbonDataQA2 commented Mar 19, 2021

CarbonDataQA2 commented Mar 19, 2021

CarbonDataQA2 commented Mar 21, 2021

CarbonDataQA2 commented Mar 21, 2021

liuhe0702 commented Mar 22, 2021

CarbonDataQA2 commented Mar 22, 2021

CarbonDataQA2 commented Mar 22, 2021

liuhe0702 commented Mar 22, 2021

CarbonDataQA2 commented Mar 22, 2021

akashrn5 Mar 22, 2021

Choose a reason for hiding this comment

liuhe0702 Mar 22, 2021

Choose a reason for hiding this comment

CarbonDataQA2 commented Mar 22, 2021

CarbonDataQA2 commented Mar 22, 2021

CarbonDataQA2 commented Mar 22, 2021

liuhe0702 commented Mar 22, 2021

CarbonDataQA2 commented Mar 22, 2021

CarbonDataQA2 commented Mar 22, 2021

liuhe0702 commented Mar 22, 2021

CarbonDataQA2 commented Mar 22, 2021

CarbonDataQA2 commented Mar 22, 2021

ShreelekhyaG commented Mar 23, 2021

akashrn5 commented Mar 23, 2021

akashrn5 commented Mar 16, 2021 •

edited