[SPARK-45756][CORE] Support spark.master.useAppNameAsAppId.enabled

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to support spark.master.useAppNameAsAppId.enabled as an experimental feature in Spark Standalone cluster.

Why are the changes needed?

This allows the users to control the appID completely.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manual tests with the following procedure.

$ SPARK_MASTER_OPTS="-Dspark.master.useAppNameAsAppId.enabled=true" sbin/start-master.sh
$ bin/spark-shell --master spark://max.local:7077

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

Could you review this experimental feature, @viirya ?

[viirya]

Hmm, how Spark responses to duplicate appId? If two submitted applications have same app names.

[viirya]

Should we add nextAppNumber as postfix to the appId as usual?

[dongjoon-hyun]

This experimental design is for the advanced users who maintains unique appName.

[viirya]

Looks good especially this is still internal config.

[dongjoon-hyun]

Thank you, @viirya !

[dongjoon-hyun]

Oops. I used the umbrella JIRA id.

[dongjoon-hyun]

• Let me switch JIRAs.

[mridulm]

What is the behavior if this is turned on and there is a conflict ? Will it result in application submission failure ? Overwriting running app state with new app (or vice versa) ?
In other words, what is the benefit of doing this ?

[dongjoon-hyun]

Thank you for review, @mridulm .

What is the behavior if this is turned on and there is a conflict ? Will it result in application submission failure ? Overwriting running app state with new app (or vice versa) ?

Yes, it will. Nothing will work correctly. It's a user responsibility to keep it unique as a ID.

In other words, what is the benefit of doing this ?

This feature gives the users lots of controllability. For simple examples,

1. Human-recognizable App ID (which was the goal of App Name).
2. Human-recognizable event log file names based on (1) in the remote storage like S3.
3. Human-recognizable driver log file names based on (1) in the remote storage like S3.

Based on (1), (2) and (3), they can build their service very easily by linking jobs and logs (event/driver) effortless with their patterns.

[mridulm]

Why not continue to use the existing scheme (iirc prefix + time: I am afk, so can't confirm), but make the prefix customizable instead ?
That keeps it human readable, while minimizing conflicts.

Pushing responsibility to user and documenting it as such would still result in hard to debug failure modes :-)

I dont use standalone mode - so not sure how common this can become, but something to think about.

[dongjoon-hyun]

@mridulm . It seems that you mean SPARK-45754 (Support spark.deploy.appIdPattern) feature which I already delivered that level of support.

Why not continue to use the existing scheme (iirc prefix + time: I am afk, so can't confirm), but make the prefix customizable instead ?

Let me give you the context, @mridulm . There is an umbrella JIRA and most customizable configurations are delivered already.

• https://issues.apache.org/jira/browse/SPARK-45869 (Revisit and Improve Spark Standalone Cluster)

[mridulm]

This is not answering my query @dongjoon-hyun - as to why we are allowing expicitly setting an application id, and introducing hard to debug modes; instead of relying on minimizing the issue.
I was not specifically referring to SPARK-45754, but that is indeed a good option as well.

[dongjoon-hyun]

To @mridulm . The above was the exact answer for your following question. What I meant is that Apache Spark already provide more patterns including the prefix customazable.

Why not continue to use the existing scheme (iirc prefix + time: I am afk, so can't confirm), but make the prefix customizable instead ?

For the following question, that is the exact design goal of this PR where a upper service system uses Apache Spark Standalone cluster as an execution subsystem. The upper service layer already provides many features like the unique appName and security layers. So, there is no such things like hard to debug or minimizing the issue.

why we are allowing explicitly setting an application id, and introducing hard to debug modes; instead of relying on minimizing the issue.

In addition, I already answered with the examples. Let me reiterate that. You can generate a SHS link like https://shs/history/app-202311140014-0204/ with appId while you cannot with appName. appId is used as a foreign key across Apache Spark systems (Storage like S3 event log and Spark driver logs, SHS link, Spark Master link). It opens a wide range of opportunity to us.

• [SPARK-45756][CORE] Support spark.master.useAppNameAsAppId.enabled #43743 (comment)

Lastly, this configuration is introduced as Experimental as you see because we are already aware of the risk. We don't recommend this configuration for novice users. This is more like automated subsystems in the industrial production usages.

spark/core/src/main/scala/org/apache/spark/internal/config/package.scala

Line 1852 in 2cac768

.doc("(Experimental) If true, Spark master uses the user-provided appName for appId.")

[mridulm]

In a specific ecosystem, this can be designed away from being an issue (have narrow waist submission layer above spark which ensures uniqueness, among other things) - and so no risk to debugging in that ecosystem.
Features in Apache Spark are used by all users - where others might not have such guarantees or might not even be aware of these requirement, and my comment about debuggability was in that context: how can a Spark user leverage this experimental feature ? What should they be aware of ? Currently, if they use this feature and there is an issue, as you mentioned behavior would be undefined.
At a minimum, we should document the expectations/risks.

Thoughts ?

[dongjoon-hyun]

Sure, I love to provide more document on this because I want to advertise Spark Standalone Cluster in Apache Spark 4.0.0 more.

Currently, I'm revisiting the documentation on Spark Standalone Cluster. The first one was #43774 and I'll make a few more PRs including this too.

Does it meet your requirement, @mridulm ?

[mridulm]

Sounds good to me, thanks @dongjoon-hyun !