[KYUUBI #3600] [SPARK][K8S] Respect default static port for web UI on Spark K8s cluster mode

### _Why are the changes needed?_

Now kyuubi will help set web ui port to 0 while scalaing up spark sql engine, which is good for the general situation, help avoid port conflicts.

But on k8s case, kyuubi should respect the user's default settings, here means web ui should be `4040`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3600 from zwangsheng/improve/fix_spark_sql_engine_web_ui_on_k8s.

Closes #3600

0174e22 [zwangsheng] comment
ac2dbed [zwangsheng] fix
1a2bff7 [zwangsheng] fix
bd4c35c [zwangsheng] fix

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit e80a7db)
Signed-off-by: Cheng Pan <chengpan@apache.org>

Author: zwangsheng

externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala

@@ -147,7 +147,6 @@ object SparkSQLEngine extends Logging {
     _sparkConf.setIfMissing("spark.sql.execution.topKSortFallbackThreshold", "10000")
     _sparkConf.setIfMissing("spark.sql.legacy.castComplexTypesToString.enabled", "true")
     _sparkConf.setIfMissing("spark.master", "local")
-    _sparkConf.setIfMissing("spark.ui.port", "0")
     _sparkConf.set(
       "spark.redaction.regex",
       _sparkConf.get("spark.redaction.regex", "(?i)secret|password|token|access[.]key")
@@ -171,6 +170,11 @@ object SparkSQLEngine extends Logging {
       kyuubiConf.setIfMissing(FRONTEND_CONNECTION_URL_USE_HOSTNAME, false)
     }
 
+    // Set web ui port 0 to avoid port conflicts during non-k8s cluster mode
+    if (!isOnK8sClusterMode) {
+      _sparkConf.setIfMissing("spark.ui.port", "0")
+    }
+
     // Pass kyuubi config from spark with `spark.kyuubi`
     val sparkToKyuubiPrefix = "spark.kyuubi."
     _sparkConf.getAllWithPrefix(sparkToKyuubiPrefix).foreach { case (k, v) =>
@@ -309,4 +313,9 @@ object SparkSQLEngine extends Logging {
       },
       "CreateSparkTimeoutChecker").start()
   }
+
+  private def isOnK8sClusterMode: Boolean = {
+    // only spark driver pod will build with `SPARK_APPLICATION_ID` env.
+    Utils.isOnK8s && sys.env.contains("SPARK_APPLICATION_ID")
+  }
 }