[SPARK-43574][PYTHON][SQL] Support to set Python executable for UDF and pandas function APIs in workers during runtime

[HyukjinKwon]

What changes were proposed in this pull request?

This PR proposes a new configuration spark.sql.execution.pyspark.python that sets the Python executable on worker nodes.

Note that, if the Python executable is different from the one previously ran, it will creates a new Python worker processes because we reuse Python workers but they are unique by both executable path and env variables as a key:

spark/core/src/main/scala/org/apache/spark/SparkEnv.scala

Lines 123 to 124 in d7a8b85

	val key = (pythonExec, envVars)
	pythonWorkers.getOrElseUpdate(key, new PythonWorkerFactory(pythonExec, envVars)).create()

This PR is also a basework for Spark Connect to support a different set of dependencies.

Why are the changes needed?

This can be useful especially when you want to run your Python with a different set of dependencies during runtime (see also https://www.databricks.com/blog/2020/12/22/how-to-manage-python-dependencies-in-pyspark.html)

Does this PR introduce any user-facing change?

No, this PR adds a configuration but that's internal for now.

How was this patch tested?

Manually tested as below:

import sys
from pyspark.sql.functions import udf
spark.range(1).select(udf(lambda x: sys.executable)("id")).show(truncate=False)
spark.conf.set("spark.sql.execution.pyspark.python", "/.../miniconda3/envs/another-python/bin/python")
spark.range(1).select(udf(lambda x: sys.executable)("id")).show(truncate=False)

+------------------------------------------+
|<lambda>(id)                              |
+------------------------------------------+
|/.../miniconda3/envs/python3.9/bin/python3|
+------------------------------------------+

+----------------------------------------------+
|<lambda>(id)                                  |
+----------------------------------------------+
|/.../miniconda3/envs/another-python/bin/python|
+----------------------------------------------+

[dongjoon-hyun]

+1, LGTM.
Merged to master for Apache Spark 3.5.0.