[SPARK-51334][CONNECT] Add java/scala version in analyze spark_version response

[garlandz-db]

What changes were proposed in this pull request?

• piggyback off the spark version response and include other env properties like java/scala version

Why are the changes needed?

• client is not able to know what type of java/scala version the server uses which can be problematic with udfs and client has different build env than server

Does this PR introduce any user-facing change?

• no. the proto includes an additional response which doesn't change for existing users

How was this patch tested?

existing tests

Was this patch authored or co-authored using generative AI tooling?

No

[the-sakthi]

I think we could add some tests here to verify the change in the response?

[garlandz-db]

@the-sakthi do you know where i can write them? im not seeing a relevant test suite that already includes tests for analyze plan handler

[grundprinzip]

Generally LGTM, please add a test via the Scala E2E tests.

[dongjoon-hyun]

Is this correct, @garlandz-db ?

[grundprinzip]

Agree this is the JRE version not the built version. Same is true for the Scala version likely.

[garlandz-db]

fixed

[dongjoon-hyun]

IIUC, this PR provides the runtime Java version instead of The java version the Spark Connect Service is built with.

[dongjoon-hyun]

In addition to #50102 (review), this PR seems to cause a semantic confusion to SparkConnect eco-system because SparkVersion is completely irrelevant from the underlying runtime Java version which can be Java 17 or Java 21.

message SparkVersion {

[grundprinzip]

@dongjoon-hyun IMHO it is ok to return these values, for language agnostic clients they can simply inquire the server about how it runs. I don't think this creates any inconsistency.

[garlandz-db]

@dongjoon-hyun please take another pass thanks!

[dongjoon-hyun]

To @grundprinzip and @garlandz-db , could you propose new messages instead of touching the existing message SparkVersion ? This kind of piggy-backing is not a good design choice because someone might want to add Python version later and then Photon or (Comet) version later.

SparkVersion is not a gateway to look up the server installation, ins't it?

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!