[K8S] Update Comment for k8s applicationId

[zhangmeng0426]

What changes were proposed in this pull request?

  update comment of applicationid, add case of k8s.

(Please fill in changes proposed in this fix)

How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

Please review https://spark.apache.org/contributing.html before opening a pull request.

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

@skonto is this correct?

[skonto]

@srowen This is not true for cluster mode. In cluster mode application_id is set here in the Spark Context and this value is retrieved from here. But it does not default to:

  private val appId = "spark-application-" + System.currentTimeMillis

as it is overridden in here:

  override def applicationId(): String = backend.applicationId()

The K8s backend will set that value here:

 conf.getOption("spark.app.id").map(_.toString).getOrElse(super.applicationId)

For cluster mode the spark.app.id is always set here to be:

val kubernetesAppId = s"spark-${UUID.randomUUID().toString.replaceAll("-", "")}"

An example: Spark Application Id: spark-3d4ef94a42e1428c9658f13573abfa38

[skonto]

@zhangmeng0426 pls update the PR title to include the related jira ticket that addresses this issue and also add related component eg. [SPARK-XXXXX][K8s]

[dongjoon-hyun]

@zhangjiajin . This is not spark-24791. Did you create a JIRA ID? You need to use your JIRA id for this.

• https://issues.apache.org/jira/browse/SPARK-24791 (Spark Structured Streaming randomly does not process batch)

[srowen]

Per #24791 (comment) this looks incorrect anyway.