[SPARK-37121] [HIVE][TEST] Fix Python version detection bug in TestUtils used by HiveExternalCatalogVersionsSuite

[xkrogen]

What changes were proposed in this pull request?

Fix a bug in TestUtils.isPythonVersionAtLeast38 to allow for HiveExternalCatalogVersionsSuite to test against Spark 2.x releases in environments with Python <= 3.7.

Why are the changes needed?

The logic in TestUtils.isPythonVersionAtLeast38 was added in #30044 to prevent Spark 2.4 from being run in an environment where the Python3 version installed was >= Python 3.8, which is not compatible with Spark 2.4. However, this method always returns true, so only Spark 3.x versions will ever be included in the version set for HiveExternalCatalogVersionsSuite, regardless of the system-installed version of Python.

The problem is here:

spark/core/src/main/scala/org/apache/spark/TestUtils.scala

Lines 280 to 291 in 951efb8

	def isPythonVersionAtLeast38(): Boolean = {
	val attempt = if (Utils.isWindows) {
	Try(Process(Seq("cmd.exe", "/C", "python3 --version"))
	.run(ProcessLogger(s => s.startsWith("Python 3.8") || s.startsWith("Python 3.9")))
	.exitValue())
	} else {
	Try(Process(Seq("sh", "-c", "python3 --version"))
	.run(ProcessLogger(s => s.startsWith("Python 3.8") || s.startsWith("Python 3.9")))
	.exitValue())
	}
	attempt.isSuccess && attempt.get == 0
	}

It's trying to evaluate the version of Python using a ProcessLogger, but the logger accepts a String => Unit function, i.e., it does not make use of the return value in any way (since it's meant for logging). So the result of the startsWith checks are thrown away, and attempt.isSuccess && attempt.get == 0 will always be true as long as your system has a python3 binary (of any version).

Does this PR introduce any user-facing change?

No, test changes only.

How was this patch tested?

Confirmed by checking that HiveExternalCatalogVersionsSuite downloads binary distros for Spark 2.x lines as well as 3.x when I symlink my python3 to Python 3.7, and only downloads distros for the 3.x lines when I symlink my python3 to Python 3.9.

brew link --force python@3.7
# run HiveExternalCatalogVersionsSuite and validate that 2.x and 3.x tests get executed
brew unlink python@3.7
brew link --force python@3.9
# run HiveExternalCatalogVersionsSuite and validate that only 3.x tests get executed

[xkrogen]

cc @dongjoon-hyun @HyukjinKwon

[SparkQA]

Test build #144627 has started for PR 34395 at commit 09769f9.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49097/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49097/

[dongjoon-hyun]

Thank you for pinging me, @xkrogen .

[dongjoon-hyun]

+1, LGTM.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49101/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49101/

[SparkQA]

Test build #144631 has finished for PR 34395 at commit bfb9883.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49138/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49138/

[SparkQA]

Test build #144670 has finished for PR 34395 at commit 535fe44.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Thanks @xkrogen.

Merged to master and branch-3.2.