[SPARK-45822][CONNECT] SparkConnectSessionManager may look up a stopped sparkcontext

[yaooqinn]

What changes were proposed in this pull request?

This PR checks whether the sc is still functional before cloning a new isolated session from it.

Why are the changes needed?

SparkSession.active is a thread-local value and not be updated by other thread.

This causes https://github.com/LuciferYang/spark/actions/runs/6767960232/job/18426049162

- ReleaseSession: session with different session_id or user_id allowed after release *** FAILED *** (9 milliseconds)
[info]   org.apache.spark.SparkException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
[info] This stopped SparkContext was created at:
[info] 
[info] org.apache.spark.sql.connect.service.SparkConnectSessionHolderSuite.beforeAll(SparkConnectSessionHolderSuite.scala:37)
[info] org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)

For shared spark sessions in tests, these sessions are created, stopped, and retrieved in different threads.

Does this PR introduce any user-facing change?

no

How was this patch tested?

I ran build/sbt "connect/testOnly *SparkConnect*" locally and the test consistently failed w/o this patch. Otherwise, it passed.

Was this patch authored or co-authored using generative AI tooling?

no

[LuciferYang]

cc @panbingkun he is also concerned about this issue

also cc @juliuszsompolski

[LuciferYang]

If the pandas environment is available, run

build/sbt "connect/testOnly *SparkConnectSessionHolderSuite *SparkConnectServiceE2ESuite"

also can reproduce this issue locally w/o this patch

[LuciferYang]

+1, LGTM

I have verified it locally and it is effective. Thank you very much @yaooqinn

[juliuszsompolski]

Thank you very much @yaooqinn !
I believe @grundprinzip was reporting hitting that to me, and it was on my list to look into.

[LuciferYang]

Merged into master for Spark 4.0. Thanks @yaooqinn @grundprinzip @juliuszsompolski

[yaooqinn]

Thank all!