[SPARK-30519][CORE] Use spark.executorEnv.HADOOP_USER_NAME to set executor's hadoop user

[apeming]

What changes were proposed in this pull request?

Let executor can specify hadoop user by using spark.executorEnv.HADOOP_USER_NAME

Why are the changes needed?

We can use HADOOP_USER_NAME to specify hadoop user while submitting a job on driver, but invalid to set executor's hadoop user even by using spark.executorEnv.HADOOP_USER_NAME.

Does this PR introduce any user-facing change?

No

How was this patch tested?

[AmplabJenkins]

Can one of the admins verify this patch?

[seanli-rallyhealth]

is "HADOOP_USER_NAME" constant defined?

[apeming]

Yes, it's specified by spark.executorEnv.HADOOP_USER_NAME and all executors will use it

[jiangxb1987]

This is a behavior change, and I don't think we should introduce this change by the approach proposed this way.

If both SPARK_USER and HADOOP_USER_NAME are specified in the env, we should prioritize SPARK_USER over HADOOP_USER_NAME. Also, I'm hesitate whether we should support HADOOP_USER_NAME at all.

[apeming]

This is a behavior change, and I don't think we should introduce this change by the approach proposed this way.

If both SPARK_USER and HADOOP_USER_NAME are specified in the env, we should prioritize SPARK_USER over HADOOP_USER_NAME. Also, I'm hesitate whether we should support HADOOP_USER_NAME at all.

But SPARK_USER is used for scheduler such as Mesos, Kubernetes and HADOOP_USER_NAME is used for Hadoop cluster such HDFS. At Spark on Kubernetes, SPARK_USER is not over HADOOP_USER_NAME

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!