[feat] upgrade hadoop\spark\hive default vertion to 3.x #4263

GuoPhilipse · 2023-02-24T09:04:48Z

What is the purpose of the change

we now support different hadoop.hive,spark vesion, so we can upgrade hadoop\spark\hive default vertion to 3.x to solve possible security issue

Related issues/PRs

Related issues: #4262
Related pr:#4263

Brief change log

upgrade hadoop from 2.7.2 ->3.3.4
upgrade hive from 2.3.3 ->3.1.3
upgrade spark from 2.4.3 ->3.2.1
remove profile spark-3.2 for spark will be 3.x by default
remove profile hadoop-3.3 for hadoop will be 3.x by default
modify profile spark-2.4-hadoop-3.3 to spark-2.4 for hadoop vesion will be 3.x by default
upgrade default curator\json4s\scala\hadoop-hdfs-client.artifact properties to fit for hadoop3/spark3
update doc introduction
update known depdency

Checklist

I have read the Contributing Guidelines on pull requests.
I have explained the need for this PR and the problem it solves
I have explained the changes or the new features added to this PR
I have added tests corresponding to this change
I have updated the documentation to reflect this change
I have verified that this change is backward compatible (If not, please discuss on the Linkis mailing list first)
If this is a code change: I have written unit tests to fully verify the new behavior.

jackxu2011

the spark-2.7 should change some properties in global, such as json4s

pom.xml

jackxu2011 · 2023-02-26T13:29:51Z

linkis-engineconn-plugins/spark/pom.xml

-      <scope>provided</scope>
+      <groupId>org.apache.linkis</groupId>
+      <artifactId>linkis-hadoop-hdfs-client-shade</artifactId>
+      <version>${project.version}</version>


this dependency is not needed by spark 3.2, it's just for spark2.4 working with hadoop3

yes, will try variable to make it suitable only for spark2.4, just keep hadoop-common and hadoop-hdfs provided if no t profile spark2.4

jackxu2011

LGTM

peacewong · 2023-03-04T11:07:22Z

...application/src/main/java/org/apache/linkis/cli/application/operator/ujes/UJESConstants.java

@@ -41,6 +41,6 @@ public class UJESConstants {

  public static final int DEFAULT_PAGE_SIZE = 500;

-  public static final String DEFAULT_SPARK_ENGINE = "spark-2.4.3";
-  public static final String DEFAULT_HIVE_ENGINE = "hive-1.2.1";
+  public static final String DEFAULT_SPARK_ENGINE = "spark-3.2.1";


peacewong · 2023-03-04T11:09:40Z

...nce-common/src/main/scala/org/apache/linkis/governance/common/conf/GovernaceCommonConf.scala

@@ -23,9 +23,9 @@ object GovernanceCommonConf {

  val CONF_FILTER_RM = "wds.linkis.rm"

-  val SPARK_ENGINE_VERSION = CommonVars("wds.linkis.spark.engine.version", "2.4.3")
+  val SPARK_ENGINE_VERSION = CommonVars("wds.linkis.spark.engine.version", "3.2.1")


Should use LabelCommonConfig:HIVE_ENGINE_VERSION and SPARK_ENGINE_VERSION

peacewong · 2023-03-04T11:10:01Z

...omputation-client/src/test/java/org/apache/linkis/computation/client/InteractiveJobTest.java

@@ -29,7 +29,7 @@ public static void main(String[] args) {
    SubmittableInteractiveJob job =
        LinkisJobClient.interactive()
            .builder()
-            .setEngineType("hive-2.3.3")
+            .setEngineType("hive-3.1.3")


Should use org.apache.linkis.manager.label.conf.LabelCommonConfig#HIVE_ENGINE_VERSION

peacewong · 2023-03-04T11:10:33Z

.../linkis-entrance/src/main/scala/org/apache/linkis/entrance/parser/CommonEntranceParser.scala

@@ -134,7 +134,7 @@ class CommonEntranceParser(val persistenceManager: PersistenceManager)
  private def checkEngineTypeLabel(labels: util.Map[String, Label[_]]): Unit = {
    val engineTypeLabel = labels.getOrDefault(LabelKeyConstant.ENGINE_TYPE_KEY, null)
    if (null == engineTypeLabel) {
-      val msg = s"You need to specify engineTypeLabel in labels, such as spark-2.4.3"
+      val msg = s"You need to specify engineTypeLabel in labels, such as spark-3.2.1"


Should use org.apache.linkis.manager.label.conf.LabelCommonConfig#SPARK_ENGINE_VERSION

peacewong · 2023-03-04T11:10:52Z

...ager/linkis-label-common/src/test/java/org/apache/linkis/manager/label/TestLabelBuilder.java

@@ -27,7 +27,7 @@ public class TestLabelBuilder {

  public static void main(String[] args) throws LabelErrorException {
    LabelBuilderFactory labelBuilderFactory = LabelBuilderFactoryContext.getLabelBuilderFactory();
-    Label<?> engineType = labelBuilderFactory.createLabel("engineType", "hive-1.2.1");
+    Label<?> engineType = labelBuilderFactory.createLabel("engineType", "hive-3.1.3");


Should use org.apache.linkis.manager.label.conf.LabelCommonConfig#HIVE_ENGINE_VERSION

peacewong · 2023-03-04T11:11:20Z

...-manager-common/src/main/scala/org/apache/linkis/manager/common/conf/ManagerCommonConf.scala

@@ -23,7 +23,7 @@ object ManagerCommonConf {

  val DEFAULT_ENGINE_TYPE = CommonVars("wds.linkis.default.engine.type", "spark")

-  val DEFAULT_ENGINE_VERSION = CommonVars("wds.linkis.default.engine.version", "2.4.3")
+  val DEFAULT_ENGINE_VERSION = CommonVars("wds.linkis.default.engine.version", "3.2.1")


peacewong · 2023-03-04T11:13:29Z

linkis-dist/helm/charts/linkis/templates/configmap-init-sql.yaml

@@ -1183,12 +1183,12 @@ data:
    (select `relation`.`config_key_id` AS `config_key_id`, '' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` FROM linkis_ps_configuration_key_engine_relation relation
    INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = '*-*,*-*');

-    -- spark2.4.3 default configuration
+    -- spark3.2.1 default configuration


can remove 3.2.1

peacewong · 2023-03-04T11:13:36Z

linkis-dist/helm/charts/linkis/templates/configmap-init-sql.yaml

    insert into `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`)
    (select `relation`.`config_key_id` AS `config_key_id`, '' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` FROM linkis_ps_configuration_key_engine_relation relation
    INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = @SPARK_ALL);

-    -- hive1.2.1 default configuration
+    -- hive3.1.3 default configuration


can remove 3.1.3

peacewong · 2023-03-04T11:14:13Z

linkis-dist/helm/scripts/resources/ldh/configmaps/configmap-spark.yaml

@@ -141,7 +141,7 @@ data:
    spark.sql.autoBroadcastJoinThreshold 26214400
    spark.sql.hive.convertMetastoreOrc true
    spark.sql.hive.metastore.jars /opt/ldh/current/spark/jars/*
-    spark.sql.hive.metastore.version 2.3.3
+    spark.sql.hive.metastore.version 3.1.3


should use 2.3.3?

spark3.2.1 use hive meta 2.3.9 by default ,maybe we can change as 2.3.9 ?

peacewong · 2023-03-04T11:15:08Z

linkis-dist/package/db/linkis_dml.sql

 -- Global Settings
 insert into `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`)
 (select config.id as `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config
 INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type is null and label.label_value = "*-*,*-*");

-- spark-2.4.3(Here choose to associate all spark type Key values with spark2.4.3)
+-- spark-3.2.1(Here choose to associate all spark type Key values with spark3.2.1)


can remove 3.2.1 and 3.1.3

peacewong

LGTM.

tclxgf13871 added 3 commits February 24, 2023 11:48

upgrade hive\spark\hadoop to default 3.x

9aa26e9

fix unknown dependency

9241fd7

fix unknown dependency

1f3c76f

jackxu2011 requested changes Feb 25, 2023

View reviewed changes

pom.xml Show resolved Hide resolved

move spark profile to global

ac30c55

jackxu2011 reviewed Feb 26, 2023

View reviewed changes

adjust dependency fit for spark2.4

12c5eb7

jackxu2011 approved these changes Feb 27, 2023

View reviewed changes

peacewong approved these changes Mar 4, 2023

View reviewed changes

GuoPhilipse added 2 commits March 5, 2023 20:14

use variable to represent version

97ab86d

fix checkstyle

c53b7ec

peacewong approved these changes Mar 6, 2023

View reviewed changes

peacewong merged commit eb55412 into apache:dev-1.4.0 Mar 6, 2023

GuoPhilipse deleted the upgradeversion branch March 7, 2023 15:59

aiceflower mentioned this pull request May 31, 2023

[Feature] Linkis1.4.0 requirements #4590

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feat] upgrade hadoop\spark\hive default vertion to 3.x #4263

[feat] upgrade hadoop\spark\hive default vertion to 3.x #4263

GuoPhilipse commented Feb 24, 2023

jackxu2011 left a comment

jackxu2011 Feb 26, 2023

GuoPhilipse Feb 26, 2023

jackxu2011 left a comment

peacewong Mar 4, 2023

peacewong Mar 4, 2023

peacewong Mar 4, 2023

peacewong Mar 4, 2023

peacewong Mar 4, 2023

peacewong Mar 4, 2023

peacewong Mar 4, 2023

peacewong Mar 4, 2023

peacewong Mar 4, 2023

GuoPhilipse Mar 4, 2023

peacewong Mar 4, 2023

peacewong left a comment

[feat] upgrade hadoop\spark\hive default vertion to 3.x #4263

[feat] upgrade hadoop\spark\hive default vertion to 3.x #4263

Conversation

GuoPhilipse commented Feb 24, 2023

What is the purpose of the change

Related issues/PRs

Brief change log

Checklist

jackxu2011 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jackxu2011 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peacewong left a comment

Choose a reason for hiding this comment