[CARBONDATA-3097] Optimize getVersionDetails #2919

xubo245 · 2018-11-14T09:24:37Z

This PR optimize getVersionDetails, including:
1.Support folder path in getVersionDetails
2.support getVersionDetails in CSDK
3.support get version from S3

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

Any interfaces changed?
add
Any backward compatibility impacted?
no
Document update required?
Yes
Testing done
add test case in SDK and CSDK
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
JIRA2951

CarbonDataQA · 2018-11-14T09:34:26Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1402/

CarbonDataQA · 2018-11-14T10:33:32Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1612/

CarbonDataQA · 2018-11-14T10:33:51Z

Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9660/

xubo245 · 2018-11-15T01:43:22Z

@KanakaKumar @jackylk @ajantha-bhat @kunal642 Please review it.

xubo245 · 2018-11-19T09:23:02Z

@KanakaKumar @jackylk @QiangCai @ajantha-bhat Please review it.

CarbonDataQA · 2018-11-22T04:14:51Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1504/

CarbonDataQA · 2018-11-22T04:26:24Z

Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9760/

CarbonDataQA · 2018-11-22T05:02:07Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1712/

xubo245 · 2018-11-22T07:15:45Z

@KanakaKumar @jackylk @QiangCai @ajantha-bhat @kunal642 Rebased, Please review it.

CarbonDataQA · 2018-12-03T09:44:28Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1619/

CarbonDataQA · 2018-12-03T10:47:37Z

Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9879/

CarbonDataQA · 2018-12-03T10:48:28Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1830/

xubo245 · 2018-12-04T01:39:18Z

retest this please

CarbonDataQA · 2018-12-04T01:49:19Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1625/

CarbonDataQA · 2018-12-04T02:49:03Z

Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9885/

CarbonDataQA · 2018-12-04T02:49:24Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1836/

xubo245 · 2018-12-04T04:04:30Z

Rebased and CI pass，@KanakaKumar @jackylk @QiangCai @ajantha-bhat @kunal642 Please review it.

KanakaKumar · 2018-12-17T06:23:35Z

store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonSchemaReader.java

   * @throws IOException
   */
-  public static String getVersionDetails(String dataFilePath) throws IOException {
+  public static String getVersionDetails(String path, boolean validate) throws IOException {


I think it is not correct to validate readability through version details. In general new version jars can read all old version files.

Please remove this method.

CarbonDataQA · 2018-12-18T08:23:09Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1818/

CarbonDataQA · 2018-12-18T08:39:26Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2029/

CarbonDataQA · 2018-12-18T09:25:05Z

Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10078/

CarbonDataQA · 2018-12-18T10:04:01Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1823/

CarbonDataQA · 2018-12-18T10:54:54Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2033/

CarbonDataQA · 2018-12-18T11:33:38Z

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10081/

KanakaKumar · 2018-12-18T15:17:27Z

store/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonSchemaReaderTest.java

+      } catch (Exception e) {
+        Assert.assertTrue(e.getMessage()
+            .equalsIgnoreCase("Version details is different between different files."));
+        Assert.fail();


What is the purpose of this test case? "Version details is different between different files" exception we never get now right? If thee is no separate validation required, we can remove this test.

ok, removed.

KanakaKumar · 2018-12-18T15:42:51Z

store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonSchemaReader.java

-  public static String getVersionDetails(String dataFilePath) throws IOException {
+  public static String getVersionDetails(String path) throws IOException {
+    if (path.endsWith(INDEX_FILE_EXT)) {
+      throw new RuntimeException("Can't get version details from carbonindex file.");


RuntimeException is mentioned in the signature. Should throw IOException.
Also some places in this class throws CarbonDataLoadingException but if not intentional we can change them to IOException as per signature instead of a subclass of RuntimeException (if it was not intentional)

IOException is from org.apache.carbondata.sdk.file.CarbonSchemaReader#getVersionDetailsFromDataFile, which is from org.apache.carbondata.core.datastore.FileReader#readByteBuffer, It's better to keep API, don't change it. @KanakaKumar

CarbonDataQA · 2018-12-18T15:46:58Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1833/

CarbonDataQA · 2018-12-18T16:42:31Z

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10090/

CarbonDataQA · 2018-12-18T17:17:07Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2041/

xubo245 · 2018-12-19T02:44:46Z

retest this please

CarbonDataQA · 2018-12-19T02:54:35Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1838/

CarbonDataQA · 2018-12-19T02:57:25Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1839/

CarbonDataQA · 2018-12-19T02:58:06Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2047/

CarbonDataQA · 2018-12-19T02:59:40Z

Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10096/

CarbonDataQA · 2018-12-19T03:24:06Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1840/

CarbonDataQA · 2018-12-19T04:17:00Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2048/

CarbonDataQA · 2018-12-19T04:22:18Z

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10097/

CarbonDataQA · 2018-12-27T02:51:47Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1975/

xubo245 · 2018-12-27T08:12:21Z

@KanakaKumar Please check it again.

CarbonDataQA · 2018-12-27T15:22:15Z

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10227/

CarbonDataQA · 2018-12-27T16:51:28Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2248/

xubo245 · 2018-12-28T03:52:53Z

@KanakaKumar Please check it again.

…t getVersionDetails in CSDK support get Verson from S3 optimize

CarbonDataQA · 2019-01-07T03:13:02Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2181/

CarbonDataQA · 2019-01-07T03:56:48Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2396/

CarbonDataQA · 2019-01-07T04:24:57Z

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10437/

xubo245 · 2019-01-07T08:00:14Z

@KanakaKumar Please review it again.

CarbonDataQA · 2019-07-24T09:05:34Z

Build Failed with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/49/

CarbonDataQA · 2019-07-24T09:36:09Z

Build Failed with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/48/

CarbonDataQA · 2019-07-24T09:48:57Z

Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/43/

CarbonDataQA1 · 2020-02-03T07:02:36Z

Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/50/

xubo245 changed the title ~~[CARBONDATA-3097] Support folder path in getVersionDetails and support getVersionDetails in CSDK~~ [CARBONDATA-3097] Support folder path in getVersionDetails and support getVersionDetails in SDK/CSDK Nov 14, 2018

xubo245 changed the title ~~[CARBONDATA-3097] Support folder path in getVersionDetails and support getVersionDetails in SDK/CSDK~~ [CARBONDATA-3097] Support folder path in getVersionDetails and support getVersionDetails in CSDK Nov 14, 2018

xubo245 force-pushed the CARBONDATA-3097_getVersionDetails branch from 22a6bd3 to 0a774fc Compare November 22, 2018 02:38

xubo245 force-pushed the CARBONDATA-3097_getVersionDetails branch from 0a774fc to bedddaa Compare December 3, 2018 09:33

KanakaKumar reviewed Dec 17, 2018

View reviewed changes

xubo245 force-pushed the CARBONDATA-3097_getVersionDetails branch from bedddaa to e29a2c4 Compare December 18, 2018 07:29

KanakaKumar reviewed Dec 18, 2018

View reviewed changes

xubo245 force-pushed the CARBONDATA-3097_getVersionDetails branch from c058ac7 to b41eff1 Compare December 19, 2018 02:51

xubo245 changed the title ~~[CARBONDATA-3097] Support folder path in getVersionDetails and support getVersionDetails in CSDK~~ [CARBONDATA-3097] Optimize getVersionDetails Dec 19, 2018

[CARBONDATA-3097] Support folder path in getVersionDetails and suppor…

1de02bc

…t getVersionDetails in CSDK support get Verson from S3 optimize

xubo245 force-pushed the CARBONDATA-3097_getVersionDetails branch from f487130 to 1de02bc Compare January 7, 2019 02:26

xubo245 closed this Nov 5, 2023

[CARBONDATA-3097] Optimize getVersionDetails #2919

[CARBONDATA-3097] Optimize getVersionDetails #2919

Conversation

xubo245 commented Nov 14, 2018 • edited

CarbonDataQA commented Nov 14, 2018

CarbonDataQA commented Nov 14, 2018

CarbonDataQA commented Nov 14, 2018

xubo245 commented Nov 15, 2018

xubo245 commented Nov 19, 2018

CarbonDataQA commented Nov 22, 2018

CarbonDataQA commented Nov 22, 2018

CarbonDataQA commented Nov 22, 2018

xubo245 commented Nov 22, 2018

CarbonDataQA commented Dec 3, 2018

CarbonDataQA commented Dec 3, 2018

CarbonDataQA commented Dec 3, 2018

xubo245 commented Dec 4, 2018

CarbonDataQA commented Dec 4, 2018

CarbonDataQA commented Dec 4, 2018

CarbonDataQA commented Dec 4, 2018

xubo245 commented Dec 4, 2018

KanakaKumar Dec 17, 2018

Choose a reason for hiding this comment

xubo245 Dec 18, 2018

Choose a reason for hiding this comment

CarbonDataQA commented Dec 18, 2018

CarbonDataQA commented Dec 18, 2018

CarbonDataQA commented Dec 18, 2018

CarbonDataQA commented Dec 18, 2018

CarbonDataQA commented Dec 18, 2018

CarbonDataQA commented Dec 18, 2018

KanakaKumar Dec 18, 2018

Choose a reason for hiding this comment

xubo245 Dec 18, 2018

Choose a reason for hiding this comment

KanakaKumar Dec 18, 2018

Choose a reason for hiding this comment

xubo245 Dec 27, 2018

Choose a reason for hiding this comment

CarbonDataQA commented Dec 18, 2018

CarbonDataQA commented Dec 18, 2018

CarbonDataQA commented Dec 18, 2018

xubo245 commented Dec 19, 2018

CarbonDataQA commented Dec 19, 2018

CarbonDataQA commented Dec 19, 2018

CarbonDataQA commented Dec 19, 2018

CarbonDataQA commented Dec 19, 2018

CarbonDataQA commented Dec 19, 2018

CarbonDataQA commented Dec 19, 2018

CarbonDataQA commented Dec 19, 2018

CarbonDataQA commented Dec 27, 2018

xubo245 commented Dec 27, 2018

CarbonDataQA commented Dec 27, 2018

CarbonDataQA commented Dec 27, 2018

xubo245 commented Dec 28, 2018

CarbonDataQA commented Jan 7, 2019

CarbonDataQA commented Jan 7, 2019

CarbonDataQA commented Jan 7, 2019

xubo245 commented Jan 7, 2019

CarbonDataQA commented Jul 24, 2019

CarbonDataQA commented Jul 24, 2019

CarbonDataQA commented Jul 24, 2019

CarbonDataQA1 commented Feb 3, 2020

xubo245 commented Nov 14, 2018 •

edited