[KYUUBI #3272] Synchronize graceful shutdown with main stop sequence

### _Why are the changes needed?_

Fixes #3272

Kyuubi server stop sequence doesn't respect graceful shutdown triggered in the discovery clients' `DeRegisterWatcher` loops. This leads to frontend service components being stopped before sessions close, resetting connections and leading to client disruption.

PR synchronizes these threads to ensure that service components wait for client sessions to close before continuing the stop sequence.
* Existing service discovery deregistration functionality is maintained to ensure that new connections are established with healthy instances once the stop sequence has begun.
* Existing Hadoop shutdown hooks ensure that the watcher can be timed out, and are configurable via `hadoop.service.shutdown.timeout`. The Kyuubi server will always shut down within 30 seconds when using default config.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3291 from bgrams/fix/graceful-shutdown.

Closes #3272

ffec2a2 [Brandon Grams] Synchronize graceful session shutdown

Authored-by: Brandon Grams <brandongr@zillowgroup.com>
Signed-off-by: Kent Yao <yao@apache.org>
(cherry picked from commit bc16852)
Signed-off-by: Kent Yao <yao@apache.org>

Author: bgrams

kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/client/KyuubiServiceDiscovery.scala

@@ -32,6 +32,7 @@ class KyuubiServiceDiscovery(
     if (!isServerLost.get()) {
       discoveryClient.deregisterService()
       discoveryClient.closeClient()
+      gracefulShutdownLatch.await() // wait for graceful shutdown triggered by watcher
     } else {
       warn(s"The Zookeeper ensemble is LOST")
     }

kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/client/ServiceDiscovery.scala

@@ -17,6 +17,7 @@
 
 package org.apache.kyuubi.ha.client
 
+import java.util.concurrent.CountDownLatch
 import java.util.concurrent.atomic.AtomicBoolean
 
 import org.apache.kyuubi.Logging
@@ -34,6 +35,7 @@ abstract class ServiceDiscovery(
     name: String,
     val fe: FrontendService) extends AbstractService(name) {
 
+  protected val gracefulShutdownLatch = new CountDownLatch(1)
   protected val isServerLost = new AtomicBoolean(false)
 
   /**
@@ -68,6 +70,7 @@ abstract class ServiceDiscovery(
       Thread.sleep(1000 * 60)
     }
     isServerLost.set(isLost)
+    gracefulShutdownLatch.countDown()
     fe.serverable.stop()
   }
 }