[SPARK-43370] Switch spark user only when run driver and executor

[Yikun]

What changes were proposed in this pull request?

Switch spark user only when run driver and executor

Why are the changes needed?

Address doi comments: question 7 [1]

[1] docker-library/official-images#13089 (comment)
[2] docker-library/official-images#13089 (comment)

Does this PR introduce any user-facing change?

Yes

How was this patch tested?

1. test mannuly

cd ~/spark-docker/3.4.0/scala2.12-java11-ubuntu
$ docker build . -t spark-test

$ docker run -ti spark-test bash
spark@afa78af05cf8:/opt/spark/work-dir$

$ docker run  --user root  -ti spark-test bash
root@095e0d7651fd:/opt/spark/work-dir#

2. ci passed

Closes: #44

[Yikun]

1. spark as default USER (this PR)

• Run directly: spark user will be used when enter docker image

$ docker run -ti spark-test bash
spark@afa78af05cf8:/opt/spark/work-dir$

• Base image: If users want to build image based on base image, users will have to switch root to install extra apt pkg and then switch back to spark.
• In previous docker (v3.3.0), docker run -ti apache/spark:v3.3.0 bash, 185 user will be used.
• K8s: we pin to spark user.

2. root as default USER ([SPARK-43370] Switch spark user only when run driver and executor and set root as default #44)

• Run directly: root user will be used when enter docker image

$ docker run -ti spark-test bash
root@afa78af05cf8:/opt/spark/work-dir$

• Base image: If users want to build image based on base image, they can install directly
• K8s: we pin to spark user.

So, looks like root user is more convenient (but bring some behavior change compare to previous apache/spark image). Wait for feedback from DOI.

[yosifkit]

This should have exec as well. Simplified, it could be exec "$@". (Applies here or to #44)

[Yikun]

According the suggestion of docker-library/official-images#13089 (comment)

I recommend that if the entrypoint needs to do setup work as root before stepping down then using the default USER root is appropriate (like mysql), otherwise USER configured to an app-specific user in the Dockerfile is better from a security perspective.

For spark, root USER is not required by any setup work, so I think spark user is enough. So I am going to merge this PR, and close #44 .

[Yikun]

@HyukjinKwon @yosifkit Thanks for review.

Merged.