[SPARK-53897][CONNECT][TESTS] Add dependency checks for Python-related tests in the connect module

[LuciferYang]

What changes were proposed in this pull request?

This pr adds a pre-check for module dependencies to the test cases in the connect module that utilize Python client code. This ensures that when the required Python modules are missing, the relevant tests are skipped rather than failing.

Why are the changes needed?

Some test cases in the connect module involve interactions between Python client code and the Connect server. These tests currently fail due to the absence of the required Python modules.

build/sbt 'connect/test'

[info] - Pipeline with selective full_refresh *** FAILED *** (374 milliseconds)
[info]   java.lang.RuntimeException: Pipeline update process failed with exit code 1.
[info] Output: 
[info] Error: Traceback (most recent call last):
[info]   File "/Users/yangjie01/SourceCode/git/spark-sbt/python/pyspark/sql/connect/utils.py", line 47, in require_minimum_grpc_version
[info]     import grpc
[info] ModuleNotFoundError: No module named 'grpc'
[info] 
[info] The above exception was the direct cause of the following exception:
[info] 
[info] Traceback (most recent call last):
[info]   File "/Users/yangjie01/SourceCode/git/spark-sbt/python/pyspark/pipelines/cli.py", line 36, in <module>
[info]     from pyspark.pipelines.block_session_mutations import block_session_mutations
[info]   File "/Users/yangjie01/SourceCode/git/spark-sbt/python/pyspark/pipelines/block_session_mutations.py", line 21, in <module>
[info]     from pyspark.sql.connect.catalog import Catalog
[info]   File "/Users/yangjie01/SourceCode/git/spark-sbt/python/pyspark/sql/connect/catalog.py", line 20, in <module>
[info]     check_dependencies(__name__)
[info]   File "/Users/yangjie01/SourceCode/git/spark-sbt/python/pyspark/sql/connect/utils.py", line 37, in check_dependencies
[info]     require_minimum_grpc_version()
[info]   File "/Users/yangjie01/SourceCode/git/spark-sbt/python/pyspark/sql/connect/utils.py", line 49, in require_minimum_grpc_version
[info]     raise PySparkImportError(
[info] pyspark.errors.exceptions.base.PySparkImportError: [PACKAGE_NOT_INSTALLED] grpcio >= 1.48.1 must be installed; however, it was not found.
...
raceback (most recent call last):
  File "/Users/yangjie01/SourceCode/git/spark-sbt/python/pyspark/sql/connect/utils.py", line 47, in require_minimum_grpc_version
    import grpc
ModuleNotFoundError: No module named 'grpc'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/yangjie01/SourceCode/gravitino/.gradle/python/MacOSX/Miniforge3/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/yangjie01/SourceCode/gravitino/.gradle/python/MacOSX/Miniforge3/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/yangjie01/SourceCode/git/spark-sbt/python/pyspark/sql/connect/streaming/worker/foreach_batch_worker.py", line 32, in <module>
    from pyspark.sql.connect.session import SparkSession
  File "/Users/yangjie01/SourceCode/git/spark-sbt/python/pyspark/sql/connect/session.py", line 20, in <module>
    check_dependencies(__name__)
  File "/Users/yangjie01/SourceCode/git/spark-sbt/python/pyspark/sql/connect/utils.py", line 37, in check_dependencies
    require_minimum_grpc_version()
  File "/Users/yangjie01/SourceCode/git/spark-sbt/python/pyspark/sql/connect/utils.py", line 49, in require_minimum_grpc_version
    raise PySparkImportError(
pyspark.errors.exceptions.base.PySparkImportError: [PACKAGE_NOT_INSTALLED] grpcio >= 1.48.1 must be installed; however, it was not found.
[info] - python foreachBatch process: process terminates after query is stopped *** FAILED *** (14 seconds, 188 milliseconds)
[info]   org.apache.spark.SparkException: Python worker failed to connect back.

...
[info] *** 48 TESTS FAILED ***
[error] Failed tests:
[error]   org.apache.spark.sql.connect.pipelines.PythonPipelineSuite
[error]   org.apache.spark.sql.connect.pipelines.EndToEndAPISuite
[error]   org.apache.spark.sql.connect.service.SparkConnectSessionHolderSuite
[error] (connect / Test / test) sbt.TestsFailedException: Tests unsuccessful

Does this PR introduce any user-facing change?

No

How was this patch tested?

• Pass Github Actions
• Manually verified by running build/sbt 'connect/test'

[info] - temporary views works !!! CANCELED !!! (0 milliseconds)
[info]   org.apache.spark.sql.connect.PythonTestDepsChecker.isConnectDepsAvailable was false (PythonPipelineSuite.scala:52)
...
[info] - create named flow with multipart name will fail !!! CANCELED !!! (1 millisecond)
[info]   org.apache.spark.sql.connect.PythonTestDepsChecker.isConnectDepsAvailable was false (PythonPipelineSuite.scala:521)
...
[info] - create flow with multipart target and no explicit name succeeds !!! CANCELED !!! (1 millisecond)
[info]   org.apache.spark.sql.connect.PythonTestDepsChecker.isConnectDepsAvailable was false (PythonPipelineSuite.scala:52)
...
[info] Run completed in 3 minutes, 27 seconds.
[info] Total number of tests run: 1088
[info] Suites: completed 37, aborted 0
[info] Tests: succeeded 1088, failed 0, canceled 45, ignored 0, pending 0
[info] All tests passed.

Was this patch authored or co-authored using generative AI tooling?

No

[LuciferYang]

Test first

[LuciferYang]

End-to-end testing for SDP requires pyyaml

[LuciferYang]

Merged into master. Thanks @HyukjinKwon