[CARBONDATA-3514] Support Spark 2.4.4 integration #3378

jackylk · 2019-09-06T01:37:54Z

This PR adds integration with Spark 2.4.4

Any interfaces changed?
Any backward compatibility impacted?
Document update required?
Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

CarbonDataQA · 2019-09-06T01:46:35Z

Build Failed with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/424/

CarbonDataQA · 2019-09-06T01:49:26Z

Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/427/

CarbonDataQA · 2019-09-06T03:23:16Z

Build Failed with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/427/

CarbonDataQA · 2019-09-06T14:29:24Z

Build Failed with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/430/

CarbonDataQA · 2019-09-06T14:35:11Z

Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/433/

CarbonDataQA · 2019-09-06T15:39:42Z

Build Failed with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/433/

CarbonDataQA · 2019-09-08T10:40:21Z

Build Failed with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/435/

CarbonDataQA · 2019-09-08T10:43:04Z

Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/438/

CarbonDataQA · 2019-09-08T12:29:07Z

Build Success with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/437/

CarbonDataQA · 2019-09-08T12:35:49Z

Build Failed with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/436/

CarbonDataQA · 2019-09-08T12:41:27Z

Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/439/

CarbonDataQA · 2019-09-08T15:15:18Z

Build Failed with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/438/

IceMimosa · 2019-09-19T11:54:16Z

Can this patch be used for production?

zzcclp · 2019-09-19T14:43:28Z

@IceMimosa it can't. this pr is WIP one and all test cases don't pass.

CarbonDataQA · 2019-11-14T13:08:11Z

Build Failed with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/871/

CarbonDataQA · 2019-11-14T13:08:20Z

Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/869/

CarbonDataQA · 2019-11-14T13:13:56Z

Build Failed with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/863/

CarbonDataQA · 2019-11-15T08:53:26Z

Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/880/

CarbonDataQA · 2019-11-15T08:53:26Z

Build Failed with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/874/

CarbonDataQA · 2019-11-15T08:53:26Z

Build Failed with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/882/

CarbonDataQA · 2019-11-15T10:31:18Z

Build Failed with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/888/

CarbonDataQA · 2019-11-15T10:35:26Z

Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/886/

CarbonDataQA · 2019-11-15T10:36:12Z

Build Failed with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/880/

CarbonDataQA · 2019-11-15T13:02:33Z

Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/893/

CarbonDataQA1 · 2019-12-13T19:17:37Z

Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1180/

CarbonDataQA1 · 2019-12-13T20:16:07Z

Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1189/

CarbonDataQA1 · 2019-12-13T20:48:55Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1199/

zzcclp · 2019-12-15T02:30:38Z

datamap/mv/plan/src/main/scala/org/apache/carbondata/mv/plans/util/BirdcageOptimizer.scala

-        SimplifyCreateStructOps,
-        SimplifyCreateArrayOps,
-        SimplifyCreateMapOps) ++


why remove these three rules?

I reverted back, fixed

zzcclp · 2019-12-15T02:32:37Z

integration/spark-common/pom.xml

+    <profile>
+      <id>build-all</id>
+      <properties>
+        <spark.version>2.2.1</spark.version>


use version 2.3.4

zzcclp · 2019-12-15T02:33:05Z

integration/spark-common/pom.xml

+    </profile>
+    <profile>
+      <id>spark-2.2</id>
+      <activation>


remove activeByDefault

zzcclp · 2019-12-15T02:33:46Z

integration/spark-common/pom.xml

+      </build>
+    </profile>
+    <profile>
+      <id>spark-2.3</id>


set activeByDefault as true

zzcclp · 2019-12-15T02:33:57Z

integration/spark-common/pom.xml

+    <profile>
+      <id>spark-2.3</id>
+      <properties>
+        <spark.version>2.3.2</spark.version>


use version 2.3.4

zzcclp · 2019-12-15T02:59:26Z

integration/spark2/src/main/spark2.4/org/apache/spark/sql/CarbonToSparkAdapter.scala

+
+  /**
+   * As a part of SPARK-24085 Hive tables supports scala subquery for
+   * parition tables,so Carbon also needs to supports


add blank before 'so'

zzcclp · 2019-12-15T04:06:16Z

BTW, currently CarbonData uses maven profile to indicate which spark version in use and manager many source path, such as commonTo2.1And2.2, commonTo2.1And2.3, to support multiple spark versions (2.1 to 2.4) in one module spark2, in the other hand, this way can not deploy jar for each spark version. So one suggestion, we can seperate mudole spark2 into multiple modules for each spark version, such as spark21, spark22, spark23, in my opinion, this way is more clear. what do you think about? @jackylk @ravipesala

jackylk · 2019-12-16T09:00:36Z

If we seperate mudole spark2 into multiple modules for each spark version, such as spark21, spark22, spark23, we will need to duplicate more code, like for the code in commonTo2.1And2.2. Then we need to make a new module for commonTo2.1And2.2 only. So, it will become too many profiles to manage. Because of this, it is very hard to keep code common and still adapt to multiple spark versions.

CarbonDataQA1 · 2019-12-16T10:34:03Z

Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1184/

CarbonDataQA1 · 2019-12-16T10:34:03Z

Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1193/

CarbonDataQA1 · 2019-12-16T10:43:06Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1203/

CarbonDataQA1 · 2019-12-22T18:06:27Z

Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1250/

CarbonDataQA1 · 2019-12-22T19:04:30Z

Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1260/

CarbonDataQA1 · 2019-12-22T19:27:44Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1271/

CarbonDataQA1 · 2020-01-05T08:57:17Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1473/

jackylk · 2020-01-05T10:53:05Z

retest this please

CarbonDataQA1 · 2020-01-05T12:18:45Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1481/

ravipesala · 2020-01-08T07:13:40Z

integration/spark-common/pom.xml

@@ -191,4 +191,183 @@
    </plugins>
  </build>

+  <profiles>


Why this is needed here?

ravipesala · 2020-01-08T07:15:17Z

integration/spark2/pom.xml

@@ -376,6 +385,55 @@
                  <sources>
                    <source>src/main/spark2.3</source>
                    <source>src/main/commonTo2.2And2.3</source>
+                    <source>src/main/commonTo2.2AndAbove</source>


It becomes so messy to add so many folders, when we are planning to remove 2.1 and 2.2 support?

In 2.0 we will remove spark 2.1 and spark 2.2 support. It is really messy if we still keep the adaptation code spark 2.1, 2.2, shall I raise to delete them? Any suggestion?

Removed spark 2.1 and 2.2 integration support in this PR

CarbonDataQA1 · 2020-01-08T17:00:11Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1542/

ravipesala · 2020-01-09T10:19:46Z

LGTM

jackylk changed the title ~~Support Spark 2.4.4 integration~~ [CARBONDATA-3514] Support Spark 2.4.4 integration Sep 6, 2019

jackylk force-pushed the spark244 branch from a24fb0b to 407de26 Compare September 8, 2019 10:32

jackylk force-pushed the spark244 branch from 407de26 to 0fb4c2e Compare September 8, 2019 12:27

jackylk force-pushed the spark244 branch from 0fb4c2e to a7d21eb Compare November 14, 2019 12:55

jackylk force-pushed the spark244 branch from a7d21eb to ea8f7e3 Compare November 15, 2019 08:23

jackylk force-pushed the spark244 branch from e9b863a to 45f0184 Compare November 15, 2019 12:41

jackylk force-pushed the spark244 branch from 7076e50 to 4dbf2b1 Compare December 13, 2019 18:19

zzcclp reviewed Dec 15, 2019

View reviewed changes

jackylk force-pushed the spark244 branch from 1277019 to 254fe91 Compare December 22, 2019 17:36

jackylk force-pushed the spark244 branch from 254fe91 to 3a90913 Compare January 5, 2020 07:37

ravipesala reviewed Jan 8, 2020

View reviewed changes

integration/spark-common/pom.xml

@@ -191,4 +191,183 @@

</plugins>

</build>

<profiles>

Copy link

Contributor

ravipesala Jan 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this is needed here?

ravipesala reviewed Jan 8, 2020

View reviewed changes

support spark 2.4 and remove spark 2.1&2.2

9177ec6

jackylk force-pushed the spark244 branch from 3a90913 to 9177ec6 Compare January 8, 2020 15:36

asfgit closed this in ba35a02 Jan 9, 2020

[CARBONDATA-3514] Support Spark 2.4.4 integration #3378

[CARBONDATA-3514] Support Spark 2.4.4 integration #3378

Conversation

jackylk commented Sep 6, 2019

CarbonDataQA commented Sep 6, 2019

CarbonDataQA commented Sep 6, 2019

CarbonDataQA commented Sep 6, 2019

CarbonDataQA commented Sep 6, 2019

CarbonDataQA commented Sep 6, 2019

CarbonDataQA commented Sep 6, 2019

CarbonDataQA commented Sep 8, 2019

CarbonDataQA commented Sep 8, 2019

CarbonDataQA commented Sep 8, 2019

CarbonDataQA commented Sep 8, 2019

CarbonDataQA commented Sep 8, 2019

CarbonDataQA commented Sep 8, 2019

IceMimosa commented Sep 19, 2019

zzcclp commented Sep 19, 2019

CarbonDataQA commented Nov 14, 2019

CarbonDataQA commented Nov 14, 2019

CarbonDataQA commented Nov 14, 2019

CarbonDataQA commented Nov 15, 2019

CarbonDataQA commented Nov 15, 2019

CarbonDataQA commented Nov 15, 2019

CarbonDataQA commented Nov 15, 2019

CarbonDataQA commented Nov 15, 2019

CarbonDataQA commented Nov 15, 2019

CarbonDataQA commented Nov 15, 2019

CarbonDataQA1 commented Dec 13, 2019

CarbonDataQA1 commented Dec 13, 2019

CarbonDataQA1 commented Dec 13, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zzcclp commented Dec 15, 2019

jackylk commented Dec 16, 2019

CarbonDataQA1 commented Dec 16, 2019

CarbonDataQA1 commented Dec 16, 2019

CarbonDataQA1 commented Dec 16, 2019

CarbonDataQA1 commented Dec 22, 2019

CarbonDataQA1 commented Dec 22, 2019

CarbonDataQA1 commented Dec 22, 2019

CarbonDataQA1 commented Jan 5, 2020

jackylk commented Jan 5, 2020

CarbonDataQA1 commented Jan 5, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jackylk Jan 8, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarbonDataQA1 commented Jan 8, 2020

ravipesala commented Jan 9, 2020

jackylk Jan 8, 2020 •

edited