[SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12

[HyukjinKwon]

What changes were proposed in this pull request?

This PR proposes to skip doctest, TorchDistributorLocalUnitTests.test_end_to_end_run_locally with Python 3.12.

Why are the changes needed?

• For the proper test for Python 3.12. It is failing, see SPARK-46149 and https://github.com/apache/spark/actions/runs/7020654429/job/19100964890

• Current dev torch version does not work very well with Python 3.12. It should be enabled together with SPARK-46078.

• The build with Python 3.12 halts so can't see if other tests pass fine.

Does this PR introduce any user-facing change?

No, dev-only.

How was this patch tested?

Yes:

./python/run-tests --python-executables=python3  --testnames 'pyspark.ml.torch.tests.test_distributor TorchDistributorLocalUnitTests'
...
Tests passed in 14 seconds

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

+1, LGTM.

[dongjoon-hyun]

Thank you for taking care of them, @HyukjinKwon !

[HyukjinKwon]

It was my bad. I am fixing it together:

Python 3.11.5 (main, Sep 11 2023, 08:19:27) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version_info > (3, 12)
False
>>> sys.version_info > (3, 11)
True

>>> (3, 12, 0) > (3, 12)
True
>>> (3, 11, 0) > (3, 12)
False

[dongjoon-hyun]

Oh, interesting.

[dongjoon-hyun]

To @HyukjinKwon , do we need to reevaluate every instances like this? It looks a little counter-intuitive.

$ git grep 'sys.version_info >'
python/pyspark/pandas/__init__.py:    if sys.version_info >= (3, 7):
python/pyspark/pandas/frame.py:        can_return_named_tuples = sys.version_info >= (3, 7) or len(self.columns) + index < 255
python/pyspark/pandas/tests/computation/test_apply_func.py:        if sys.version_info >= (3, 8):
python/pyspark/pandas/tests/test_typedef.py:            if sys.version_info >= (3, 8):
python/pyspark/pandas/typedef/typehints.py:    if sys.version_info >= (3, 8):
python/pyspark/sql/connect/_typing.py:if sys.version_info >= (3, 8):
python/pyspark/sql/connect/proto/base_pb2.pyi:if sys.version_info >= (3, 10):
python/pyspark/sql/connect/proto/catalog_pb2.pyi:if sys.version_info >= (3, 8):
python/pyspark/sql/connect/proto/commands_pb2.pyi:if sys.version_info >= (3, 10):
python/pyspark/sql/connect/proto/common_pb2.pyi:if sys.version_info >= (3, 8):
python/pyspark/sql/connect/proto/example_plugins_pb2.pyi:if sys.version_info >= (3, 8):
python/pyspark/sql/connect/proto/expressions_pb2.pyi:if sys.version_info >= (3, 10):
python/pyspark/sql/connect/proto/relations_pb2.pyi:if sys.version_info >= (3, 10):
python/pyspark/sql/connect/proto/types_pb2.pyi:if sys.version_info >= (3, 8):
python/pyspark/tests/test_worker.py:    @unittest.skipIf(sys.version_info > (3, 11), "SPARK-46130: Flaky with Python 3.12")

$ git grep 'sys.version_info <'
core/src/main/scala/org/apache/spark/TestUtils.scala:    val pythonSnippet = s"import sys; sys.exit(sys.version_info < ($major, $minor, $reversion))"
dev/run-tests:PYTHON_VERSION_CHECK=$(python3 -c 'import sys; print(sys.version_info < (3, 8, 0))')
dev/run-tests-jenkins:PYTHON_VERSION_CHECK=$(python3 -c 'import sys; print(sys.version_info < (3, 8, 0))')
python/pyspark/cloudpickle/compat.py:if sys.version_info < (3, 8):
python/pyspark/pandas/typedef/typehints.py:    if sys.version_info < (3, 11):
python/pyspark/shuffle.py:        if sys.version_info < (3, 11):
python/pyspark/sql/tests/pandas/test_pandas_udf_typehints.py:    @unittest.skipIf(sys.version_info < (3, 9), "Type hinting generics require Python 3.9.")
python/pyspark/sql/tests/pandas/test_pandas_udf_typehints_with_future_annotations.py:        sys.version_info < (3, 9),
python/run-tests:PYTHON_VERSION_CHECK=$(python3 -c 'import sys; print(sys.version_info < (3, 8, 0))')

[HyukjinKwon]

sure - lemme check them

[dongjoon-hyun]

So, is it impossible to be equal because one is 3-item tuple and the other is 2-item tuple?

>>> (3, 12, 0) == (3, 12)
False

>>> (3, 12, 0) <= (3, 12)
False

[HyukjinKwon]

Yeah, I think so actually. For that, let me take a separate look for individual (with reading https://docs.python.org/3/reference/expressions.html#value-comparisons carefully), and fix them if there's anything wrong (or that looks weird).

[HyukjinKwon]

Merged to master.