[SPARK-31193][CORE] set spark.master and spark.app.name conf default value

[Le-Dai]

What changes were proposed in this pull request?

set spark.master and spark.app.name default config

Why are the changes needed?

I see the default value of master setting in spark-submit client

// Global defaults. These should be keep to minimum to avoid confusing behavior.
master = Option(master).getOrElse("local[*]")

but during our development and debugging, We will encounter this kind of problem

Exception in thread "main" org.apache.spark.SparkException: A master URL must be set in your configuration

This conflicts with the default setting

//If we do
val sparkConf = new SparkConf().setAppName(“app”)
//When using the client to submit tasks to the cluster, the matser will be overwritten by the local
sparkConf.set("spark.master", "local[*]")

so we have to do like this

val sparkConf = new SparkConf().setAppName(“app”)
//Because the program runs to set the priority of the master, we have to first determine whether to set the master to avoid submitting the cluster to run.
sparkConf.set("spark.master",sparkConf.get("spark.master","local[*]"))

so is spark.app.name

Is it better for users to handle it like submit client ?

Optimize the development process

Does this PR introduce any user-facing change?

no

How was this patch tested?

do not set config test

[AmplabJenkins]

Can one of the admins verify this patch?

[Le-Dai]

cc @maropu @cloud-fan

[srowen]

Your code shouldn't set a master; it should be set when you submit your code. It does have to be set, and already defaults in the submit client code, yes. I don't think we should duplicate the logic.
If you did for some reason want to do this directly, yes, you have to set a master.

[Le-Dai]

Your code shouldn't set a master; it should be set when you submit your code. It does have to be set, and already defaults in the submit client code, yes. I don't think we should duplicate the logic.
If you did for some reason want to do this directly, yes, you have to set a master.

In most cases, we will choose to submit to the cluster after the local debugging is completed, but because we manually set the master in the code, the attribute of the submit client will be overwritten during the submission process. because the local development process is separate from the submit, why not assigning default values when Context initialization?

[srowen]

Simply do not set it in your code? That is the intended usage here.

[maropu]

Simply do not set it in your code? That is the intended usage here.

Yea, I think so, too...

[dongjoon-hyun]

Thank you for making a PR, @MrDLontheway . However, could you close this PR?

[Le-Dai]

ok thanks

[dongjoon-hyun]

Thank you!