[SPARK-52308][CORE] Fix the bug that the actual configs of the driver are overwritten

[huangxiaopingRD]

What changes were proposed in this pull request?

After Spark Driver is started, it is not allowed to overwrite Driver-related configurations.

Why are the changes needed?

Fix the problem that the driver-related configuration was overwritten by the user by mistake, resulting in the final spark UI display error.

Does this PR introduce any user-facing change?

Yes. The values ​​of driver-related configurations on the Spark UI will show the actual effective values

How was this patch tested?

The manual test is as follows:
Test Scripts：
./bin/spark-submit --conf spark.driver.memory=10g --conf spark.driver.memoryOverhead=2g --queue dev test_pyspark.py

test_pyspark.py

from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext, HiveContext, SparkSession

SPARK_CONF = SparkConf().set('spark.driver.memory','2.5G').set('spark.driver.memoryOverhead','2.5G')

sc = SparkContext(appName="test pyspark", conf=SPARK_CONF)

spark = SparkSession.builder.enableHiveSupport().getOrCreate()

df = spark.sql("select 1")
df.show()
spark.stop()

Spark UI display before repair：

Spark UI display after repair：

BTW, Spark does not support double type values ​​for memory configuration ​​by default. So when we used Spark code to parse the event log, we encountered this problem. This PR is to avoid this kind of problem from the source.

Was this patch authored or co-authored using generative AI tooling?

No

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!