[SPARK-47040][CONNECT] Allow Spark Connect Server Script to wait

[grundprinzip]

What changes were proposed in this pull request?

Add an option to the command line of ./sbin/start-connect-server.sh that leaves it running in the foreground for easier debugging.

./sbin/start-connect-server.sh --wait

Why are the changes needed?

Usability

Does this PR introduce any user-facing change?

How was this patch tested?

Manual

Was this patch authored or co-authored using generative AI tooling?

No

[dongjoon-hyun]

+1, LGTM.

[pan3793]

... that leaves it running in the foreground ...

could it be achieved by the following command?

SPARK_NO_DAEMONIZE=1 ./sbin/start-connect-server.sh

[dongjoon-hyun]

Oh, right. Does it work in the same with this?

[dongjoon-hyun]

If then, we need to revert this. Could you confirm that the above works for you, @grundprinzip ?

[grundprinzip]

If then, we need to revert this. Could you confirm that the above works for you, @grundprinzip ?

I will not revert the patch because it has significantly better developer ergonomics over obscure environment variables. I created a PR to use the env variable instead of calling spark-submit directly.

#45117

[pan3793]

... it has significantly better developer ergonomics over obscure environment variables.

I would recommend removing --wait, just use env var and update https://spark.apache.org/developer-tools.html to mention that.

@grundprinzip There are several server components in Spark, like Thrift Server, History Server, Master, Worker, only adding --wait on the Connect Server makes things inconsistent, while env vars approach is adopted widely in Spark for different develop/test purposes. Additionally, --wait is not quite accurate, it also affects stdout/stderr redirection not just process exiting.

[dongjoon-hyun]

@pan3793 's point makes sense to me in terms of the consistency inside Apache Spark.

How do you think about that, @grundprinzip ?