[SPARK-42181][PYTHON][TESTS] Skip torch tests when torch is not installed#39737
Closed
dongjoon-hyun wants to merge 3 commits intoapache:masterfrom
dongjoon-hyun:SPARK-42181
Closed
[SPARK-42181][PYTHON][TESTS] Skip torch tests when torch is not installed#39737dongjoon-hyun wants to merge 3 commits intoapache:masterfrom dongjoon-hyun:SPARK-42181
torch tests when torch is not installed#39737dongjoon-hyun wants to merge 3 commits intoapache:masterfrom
dongjoon-hyun:SPARK-42181
Conversation
Member
Author
|
Could you review this, @HyukjinKwon ? |
HyukjinKwon
reviewed
Jan 25, 2023
| self.delete_env_vars(input_env_vars) | ||
|
|
||
|
|
||
| @unittest.skipIf(not have_torch, "torch is required") # type: ignore |
Member
There was a problem hiding this comment.
actually we should exclude this in mypy test.I will do it separately.
HyukjinKwon
approved these changes
Jan 25, 2023
HyukjinKwon
changed the title
torch tests when torch is not installedtorch tests when torch is not installed
Member
|
Merged to master and branch-3.4. |
HyukjinKwon
added a commit
that referenced
this pull request
Jan 25, 2023
…nstalled
### What changes were proposed in this pull request?
This PR aims to skip `torch` tests when `torch` is not installed.
### Why are the changes needed?
This will enable CI environments to run tests without `torch` test dependency requirements.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs and do the manual tests on a system without `torch`.
```
$ python/run-tests --testnames pyspark.ml.torch.tests.test_distributor
Running PySpark tests. Output is in /Users/dongjoon/APACHE/spark-merge/python/unit-tests.log
Will test against the following Python executables: ['python3.9']
Will test the following Python tests: ['pyspark.ml.torch.tests.test_distributor']
python3.9 python_implementation is CPython
python3.9 version is: Python 3.9.16
Starting test(python3.9): pyspark.ml.torch.tests.test_distributor (temp output: /Users/dongjoon/APACHE/spark-merge/python/target/9347caa4-6c4d-44cb-aeec-da87c2aa2749/python3.9__pyspark.ml.torch.tests.test_distributor__jobzp1bh.log)
Finished test(python3.9): pyspark.ml.torch.tests.test_distributor (0s) ... 17 tests were skipped
Tests passed in 0 seconds
Skipped tests in pyspark.ml.torch.tests.test_distributor with python3.9:
test_create_torchrun_command (pyspark.ml.torch.tests.test_distributor.TorchDistributorBaselineUnitTests) ... SKIP (0.000s)
test_encryption_fails (pyspark.ml.torch.tests.test_distributor.TorchDistributorBaselineUnitTests) ... SKIP (0.000s)
test_encryption_passes (pyspark.ml.torch.tests.test_distributor.TorchDistributorBaselineUnitTests) ... SKIP (0.000s)
test_get_num_tasks_fails (pyspark.ml.torch.tests.test_distributor.TorchDistributorBaselineUnitTests) ... SKIP (0.000s)
test_validate_correct_inputs (pyspark.ml.torch.tests.test_distributor.TorchDistributorBaselineUnitTests) ... SKIP (0.000s)
test_validate_incorrect_inputs (pyspark.ml.torch.tests.test_distributor.TorchDistributorBaselineUnitTests) ... SKIP (0.000s)
test_dist_training_succeeds (pyspark.ml.torch.tests.test_distributor.TorchDistributorDistributedUnitTests) ... SKIP (0.000s)
test_distributed_file_with_pytorch (pyspark.ml.torch.tests.test_distributor.TorchDistributorDistributedUnitTests) ... SKIP (0.000s)
test_end_to_end_run_distributedly (pyspark.ml.torch.tests.test_distributor.TorchDistributorDistributedUnitTests) ... SKIP (0.000s)
test_get_num_tasks_distributed (pyspark.ml.torch.tests.test_distributor.TorchDistributorDistributedUnitTests) ... SKIP (0.000s)
test_end_to_end_run_locally (pyspark.ml.torch.tests.test_distributor.TorchDistributorLocalUnitTests) ... SKIP (0.000s)
test_get_gpus_owned_local (pyspark.ml.torch.tests.test_distributor.TorchDistributorLocalUnitTests) ... SKIP (0.000s)
test_get_num_tasks_locally (pyspark.ml.torch.tests.test_distributor.TorchDistributorLocalUnitTests) ... SKIP (0.000s)
test_local_file_with_pytorch (pyspark.ml.torch.tests.test_distributor.TorchDistributorLocalUnitTests) ... SKIP (0.000s)
test_local_training_succeeds (pyspark.ml.torch.tests.test_distributor.TorchDistributorLocalUnitTests) ... SKIP (0.000s)
test_check_parent_alive (pyspark.ml.torch.tests.test_distributor.TorchWrapperUnitTests) ... SKIP (0.000s)
test_clean_and_terminate (pyspark.ml.torch.tests.test_distributor.TorchWrapperUnitTests) ... SKIP (0.000s)
```
Closes #39737 from dongjoon-hyun/SPARK-42181.
Lead-authored-by: Dongjoon Hyun <dongjoon@apache.org>
Co-authored-by: Hyukjin Kwon <gurwls223@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 9835476)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Member
Author
|
Thank you! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR aims to skip
torchtests whentorchis not installed.Why are the changes needed?
This will enable CI environments to run tests without
torchtest dependency requirements.Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass the CIs and do the manual tests on a system without
torch.