[KYUUBI #3914] [K8S][SPARK] Set spark driver host to ip instead of po…

…d name when kyuubi on k8s submit spark with client deploy-mode

### _Why are the changes needed?_

Set `spark.driver.host` to ip instead of pod name when kyuubi on k8s submit spark with client deploy-mode.

When Kyuubi On Kubernetes submit spark with client deploy mode, spark driver will using kyuubi pod name as `spark.driver.host`, which can't be recognized by executors(Exclude kyuubi deployed as Statefulset + serivce case).

We have done this in #1596

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3914 from zwangsheng/k8s/set_driver_host_if_client.

Closes #3914

8fb8965 [zwangsheng] fix style
fc38143 [Binjie Yang] Merge branch 'master' into k8s/set_driver_host_if_client
4a31365 [zwangsheng] fix style
c61b7b8 [zwangsheng] fix

Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Binjie Yang <52876270+zwangsheng@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>

externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala

@@ -17,6 +17,7 @@
 
 package org.apache.kyuubi.engine.spark
 
+import java.net.InetAddress
 import java.time.Instant
 import java.util.{Locale, UUID}
 import java.util.concurrent.{CountDownLatch, ScheduledExecutorService, ThreadPoolExecutor, TimeUnit}
@@ -201,6 +202,11 @@ object SparkSQLEngine extends Logging {
       _sparkConf.setIfMissing(
         "spark.kubernetes.executor.podNamePrefix",
         generateExecutorPodNamePrefixForK8s(user))
+
+      if (!isOnK8sClusterMode) {
+        // set driver host to ip instead of kyuubi pod name
+        _sparkConf.set("spark.driver.host", InetAddress.getLocalHost.getHostAddress)
+      }
     }
 
     // Set web ui port 0 to avoid port conflicts during non-k8s cluster mode

integration-tests/kyuubi-kubernetes-it/src/test/scala/org/apache/kyuubi/kubernetes/test/deployment/KyuubiOnKubernetesTestsSuite.scala

@@ -78,7 +78,6 @@ class KyuubiOnKubernetesWithClientSparkTestsSuite
   override protected def connectionConf: Map[String, String] = {
     super.connectionConf ++ Map(
       "spark.submit.deployMode" -> "client",
-      "spark.driver.host" -> kyuubiServerIp,
       "kyuubi.frontend.connection.url.use.hostname" -> "false",
       "spark.kubernetes.executor.label.kyuubi-it-test" -> "client")
   }