[WIP][SPARK-40309][PYTHON][PS] Introduce sql_conf context manager for pyspark.sql

[xinrong-meng]

What changes were proposed in this pull request?

Introduce sql_conf context manager for pyspark.sql.

Why are the changes needed?

That simplifies the control of Spark SQL configuration as below

from

original_value = spark.conf.get("key")
spark.conf.set("key", "value")
...
spark.conf.set("key", original_value)

to

with sql_conf({"key": "value"}):
    ...

Here is such a context manager is in Pandas API on Spark.

We should introduce one in pyspark.sql, and deduplicate code if possible.

Does this PR introduce any user-facing change?

Yes. Users may use the context manager to manage the Spark SQL configuration for a code block.

For example,

>>> from pyspark.sql.utils import sql_conf
>>> with sql_conf({"spark.sql.execution.arrow.pyspark.enabled": True}):
...    pdf = sdf.toPandas()

How was this patch tested?

Unit tests.

[xinrong-meng]

Remain the assertion to be consistent with pyspark.pandas.utils.sql_conf.

[AmplabJenkins]

Can one of the admins verify this patch?

[HyukjinKwon]

Should probably name it as sqlConf since we follow camelcase in the API. In addition, I think this should be able to be imported via something like from pyspark.sql import sqlConf. Should also document it in the API reference at python/docs