[WIP] Support Global Unique Id for SegmentNo #3934

marchpure · 2020-09-16T11:14:26Z

Why is this PR needed?

What changes were proposed in this PR?

Does this PR introduce any user interface change?

No
Yes. (please explain the change and update document)

Is any new testcase added?

No
Yes

CarbonDataQA1 · 2020-09-16T11:46:27Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2360/

CarbonDataQA1 · 2020-09-16T11:47:33Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4102/

CarbonDataQA1 · 2020-09-16T13:21:23Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2363/

CarbonDataQA1 · 2020-09-16T13:23:22Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4105/

CarbonDataQA1 · 2020-09-16T13:57:59Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2364/

CarbonDataQA1 · 2020-09-16T14:00:12Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4106/

CarbonDataQA1 · 2020-09-16T18:48:25Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2369/

CarbonDataQA1 · 2020-09-16T18:59:46Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4111/

… compaction process

CarbonDataQA1 · 2020-09-24T06:05:44Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2457/

CarbonDataQA1 · 2020-09-24T06:12:51Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4200/

CarbonDataQA1 · 2020-09-25T11:16:30Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4217/

CarbonDataQA1 · 2020-09-25T11:20:20Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2474/

CarbonDataQA1 · 2020-09-25T11:26:09Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4218/

CarbonDataQA1 · 2020-09-25T11:30:09Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2475/

marchpure · 2020-09-25T11:42:40Z

retest this please

CarbonDataQA1 · 2020-09-25T11:46:00Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4219/

CarbonDataQA1 · 2020-09-25T11:50:16Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2476/

CarbonDataQA1 · 2020-09-25T14:43:20Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2478/

CarbonDataQA1 · 2020-09-25T14:45:31Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4221/

CarbonDataQA1 · 2020-09-27T02:39:47Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2487/

CarbonDataQA1 · 2020-09-27T02:40:52Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4231/

CarbonDataQA1 · 2020-09-27T03:19:51Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2490/

CarbonDataQA1 · 2020-09-27T03:20:48Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4234/

CarbonDataQA1 · 2020-09-27T05:23:11Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4237/

CarbonDataQA1 · 2020-09-27T05:24:40Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2493/

CarbonDataQA1 · 2020-09-27T13:23:32Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4242/

CarbonDataQA1 · 2020-09-27T13:27:25Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2498/

CarbonDataQA1 · 2020-09-27T17:40:32Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4247/

CarbonDataQA1 · 2020-09-27T17:43:02Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2502/

marchpure · 2020-10-09T08:27:49Z

retest this please

CarbonDataQA1 · 2020-10-09T10:07:08Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4344/

CarbonDataQA1 · 2020-10-09T10:24:52Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2594/

ajantha-bhat · 2020-10-15T05:49:16Z

@marchpure and @QiangCai : The solution was to use segmentid_UUID right ? I didn't see UUID in the PR changes. please explain the solution or update the PR description

vikramahuja1001 · 2020-10-15T06:10:19Z

core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java

    String tablePath = identifier.getTablePath();

    if (filePath.startsWith(tablePath)) {
      if (!isTransactionalTable || isStandardTable) {
-        blockId = "Part0" + CarbonCommonConstants.FILE_SEPARATOR + "Segment_" + segmentId
+        blockId = "Part0" + CarbonCommonConstants.FILE_SEPARATOR + "Segment_"
+            + segmentId + segmentId


why is segmentID used twice instead of a UUID?

QiangCai · 2020-10-19T01:25:34Z

please close this PR and raise another PR to fix the listFiles issue.

ajantha-bhat · 2020-10-22T12:32:24Z

@QiangCai :I have fixed the list files issue in #3994 as it was urgent.

@marchpure : If you can use UUID and make it work. we can still have this feature. If not better to close this PR as the actual issue is fixed.

marchpure force-pushed the segmentno_uniqueid branch from 0f53e8d to b2cf16f Compare September 16, 2020 13:06

marchpure force-pushed the segmentno_uniqueid branch from b2cf16f to 75e8069 Compare September 16, 2020 13:25

marchpure force-pushed the segmentno_uniqueid branch 2 times, most recently from a658b65 to c7f20ef Compare September 16, 2020 18:24

[CARBONDATA-3993] Remove deletePartialLoadData command in loading and…

d543c98

… compaction process

ajantha-bhat mentioned this pull request Sep 18, 2020

[CARBONDATA-3993] Remove auto data deletion in IUD processs #3935

Closed

marchpure force-pushed the segmentno_uniqueid branch from c7f20ef to 9248ad3 Compare September 24, 2020 04:10

marchpure force-pushed the segmentno_uniqueid branch from 9248ad3 to 78d5ed2 Compare September 25, 2020 11:12

marchpure force-pushed the segmentno_uniqueid branch from 78d5ed2 to d4c7ade Compare September 25, 2020 11:22

marchpure force-pushed the segmentno_uniqueid branch from d4c7ade to ef10dcb Compare September 25, 2020 12:41

marchpure force-pushed the segmentno_uniqueid branch from ef10dcb to f909066 Compare September 27, 2020 02:36

marchpure force-pushed the segmentno_uniqueid branch from f909066 to 0f1dc0e Compare September 27, 2020 03:13

marchpure force-pushed the segmentno_uniqueid branch from 0f1dc0e to 09b0def Compare September 27, 2020 03:27

marchpure force-pushed the segmentno_uniqueid branch from 09b0def to efb4443 Compare September 27, 2020 11:31

test

11c61d5

marchpure force-pushed the segmentno_uniqueid branch from efb4443 to 11c61d5 Compare September 27, 2020 15:49

vikramahuja1001 reviewed Oct 15, 2020

View reviewed changes

marchpure closed this Oct 27, 2020

[WIP] Support Global Unique Id for SegmentNo #3934

[WIP] Support Global Unique Id for SegmentNo #3934

Conversation

marchpure commented Sep 16, 2020 • edited

Why is this PR needed?

What changes were proposed in this PR?

Does this PR introduce any user interface change?

Is any new testcase added?

CarbonDataQA1 commented Sep 16, 2020

CarbonDataQA1 commented Sep 16, 2020

CarbonDataQA1 commented Sep 16, 2020

CarbonDataQA1 commented Sep 16, 2020

CarbonDataQA1 commented Sep 16, 2020

CarbonDataQA1 commented Sep 16, 2020

CarbonDataQA1 commented Sep 16, 2020

CarbonDataQA1 commented Sep 16, 2020

CarbonDataQA1 commented Sep 24, 2020

CarbonDataQA1 commented Sep 24, 2020

CarbonDataQA1 commented Sep 25, 2020

CarbonDataQA1 commented Sep 25, 2020

CarbonDataQA1 commented Sep 25, 2020

CarbonDataQA1 commented Sep 25, 2020

marchpure commented Sep 25, 2020

CarbonDataQA1 commented Sep 25, 2020

CarbonDataQA1 commented Sep 25, 2020

CarbonDataQA1 commented Sep 25, 2020

CarbonDataQA1 commented Sep 25, 2020

CarbonDataQA1 commented Sep 27, 2020

CarbonDataQA1 commented Sep 27, 2020

CarbonDataQA1 commented Sep 27, 2020

CarbonDataQA1 commented Sep 27, 2020

CarbonDataQA1 commented Sep 27, 2020

CarbonDataQA1 commented Sep 27, 2020

CarbonDataQA1 commented Sep 27, 2020

CarbonDataQA1 commented Sep 27, 2020

CarbonDataQA1 commented Sep 27, 2020

CarbonDataQA1 commented Sep 27, 2020

marchpure commented Oct 9, 2020

CarbonDataQA1 commented Oct 9, 2020

CarbonDataQA1 commented Oct 9, 2020

ajantha-bhat commented Oct 15, 2020

vikramahuja1001 Oct 15, 2020

Choose a reason for hiding this comment

QiangCai commented Oct 19, 2020

ajantha-bhat commented Oct 22, 2020

marchpure commented Sep 16, 2020 •

edited