Supports fully asynchronous Engine initialization

[HanShuhang]

Why are the changes needed?

When SESSION_ENGINE_LAUNCH_ASYNC is set to True, the client should not block while waiting for the engine to initialize. This is particularly important in scenarios where cluster resources are constrained, as engine initialization can be significantly delayed, potentially causing the client to block during execute calls. This can lead to serious issues—for example, we found that PyHive may lose connection with Kyuubi under such circumstances. This PR enables fully asynchronous engine startup, thereby reducing client response latency and improving the overall user experience.

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 22 lines in your changes missing coverage. Please review.

Project coverage is 0.00%. Comparing base (ba854d3) to head (219a1cf).
Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
.../org/apache/kyuubi/session/KyuubiSessionImpl.scala	0.00%	11 Missing ⚠️
...org/apache/kyuubi/operation/ExecuteStatement.scala	0.00%	10 Missing ⚠️
.../org/apache/kyuubi/operation/KyuubiOperation.scala	0.00%	1 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##           master   #7020   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         695     695           
  Lines       42819   42830   +11     
  Branches     5830    5832    +2     
======================================
- Misses      42819   42830   +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[turboFei]

could you check the UT failure?

[Copilot]

Copilot wasn't able to review any files in this pull request.

Files not reviewed (3)

• kyuubi-server/src/main/scala/org/apache/kyuubi/operation/ExecuteStatement.scala: Language not supported
• kyuubi-server/src/main/scala/org/apache/kyuubi/operation/KyuubiOperation.scala: Language not supported
• kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiSessionImpl.scala: Language not supported

[turboFei]

This PR is a little tricky for me.

Did you test whether this PR resolve the issue?

Could you check the KyuubiConnection?

kyuubi/kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiConnection.java

Lines 195 to 199 in 4cbff4d

	if (!isBeeLineMode) {
	showLaunchEngineLog();
	waitLaunchEngineToComplete();
	executeInitSql();
	}

Was your issue caused by waitLaunchEngineToComplete?

[HanShuhang]

Our users connect to kyuubi by pyhive with CONNECTION mode. If the engine is not started for a long time (maybe dozens of minutes) due to waiting for resources, the client's execute sql will be blocked in kyuubi. The final phenomenon is that even if the spark engine is successfully started finally, the client can not receive any response. This PR solved this problem, execute request (async) from pyhive can return quickly without blocking while waiting for the engine to start.
In addition, asynchronous execution should not be blocked waiting for resources in theory, which is obviously not asynchronous enough.