Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-46131][PYTHON][INFRA] Install torchvision for Python 3.12 build
### What changes were proposed in this pull request? This PR adds `torchvision` into the testing image for Python 3.12. ### Why are the changes needed? To continue Python 3.12 build, and see what are failing. Currently it fails as below: https://github.com/apache/spark/actions/runs/7006848931/job/19059702169#step:12:4236 ``` ====================================================================== ERROR [0.001s]: test_end_to_end_run_distributedly (pyspark.ml.tests.connect.test_parity_torch_distributor.TorchDistributorDistributedUnitTestsOnConnect.test_end_to_end_run_distributedly) ---------------------------------------------------------------------- Traceback (most recent call last): File "/__w/spark/spark/python/pyspark/ml/torch/tests/test_distributor.py", line 495, in test_end_to_end_run_distributedly train_fn = create_training_function(self.mnist_dir_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/__w/spark/spark/python/pyspark/ml/torch/tests/test_distributor.py", line 60, in create_training_function from torchvision import transforms, datasets ModuleNotFoundError: No module named 'torchvision' ====================================================================== ERROR [0.001s]: test_end_to_end_run_locally (pyspark.ml.tests.connect.test_parity_torch_distributor.TorchDistributorLocalUnitTestsIIOnConnect.test_end_to_end_run_locally) ---------------------------------------------------------------------- Traceback (most recent call last): File "/__w/spark/spark/python/pyspark/ml/torch/tests/test_distributor.py", line 402, in test_end_to_end_run_locally train_fn = create_training_function(self.mnist_dir_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/__w/spark/spark/python/pyspark/ml/torch/tests/test_distributor.py", line 60, in create_training_function from torchvision import transforms, datasets ModuleNotFoundError: No module named 'torchvision' ====================================================================== ERROR [0.001s]: test_end_to_end_run_locally (pyspark.ml.tests.connect.test_parity_torch_distributor.TorchDistributorLocalUnitTestsOnConnect.test_end_to_end_run_locally) ---------------------------------------------------------------------- Traceback (most recent call last): File "/__w/spark/spark/python/pyspark/ml/torch/tests/test_distributor.py", line 402, in test_end_to_end_run_locally train_fn = create_training_function(self.mnist_dir_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/__w/spark/spark/python/pyspark/ml/torch/tests/test_distributor.py", line 60, in create_training_function from torchvision import transforms, datasets ModuleNotFoundError: No module named 'torchvision' ---------------------------------------------------------------------- Ran 23 tests in 50.860s ``` and this pr fixes it ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Manually tested. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #44045 from HyukjinKwon/SPARK-46131. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
- Loading branch information