[SPARK-35832][CORE][ML][K8S][TESTS] Add LocalRootDirsTest trait #32986
Conversation
|
Could you review this when you have some time, @HyukjinKwon and @viirya ? |
| import org.apache.spark.util.Utils | ||
|
|
||
|
|
||
| trait LocalRootDirsTest extends SparkFunSuite with LocalSparkContext { |
There was a problem hiding this comment.
This functionality came from SortShuffleSuite which had the most complete clean-up logic.
There was a problem hiding this comment.
Can we add some comments for this trait? So others in the future can know quickly what this trait is for.
dongjoon-hyun
changed the title
|
This is also a patch to fix flaky tests, @gengliangwang . |
|
Kubernetes integration test starting |
|
Kubernetes integration test unable to build dist. exiting with code: 1 |
|
Kubernetes integration test status success |
|
Test build #140042 has finished for PR 32986 at commit
|
|
Test build #140044 has finished for PR 32986 at commit
|
| @@ -192,7 +195,7 @@ class KubernetesLocalDiskShuffleDataIOSuite extends SparkFunSuite with LocalSpar | |||
| val rdd3 = rdd2.reduceByKey(_ + _) | |||
| val rdd4 = rdd3.sortByKey() | |||
|
|
|||
| assert(rdd4.count() === 3) | |||
| assert(rdd4.count() === 2) | |||
|
Thank you, @viirya . Yes, that was a hidden bug and uncovered by this. I mentioned in the PR description like the following.
Merged to master. |
|
Thank you @dongjoon-hyun |
|
lgtm2 |
### What changes were proposed in this pull request? To make the test suite more robust, this PR aims to add a new trait, `LocalRootDirsTest`, by refactoring `SortShuffleSuite`'s helper functions and applying it to the following: - ShuffleNettySuite - ShuffleOldFetchProtocolSuite - ExternalShuffleServiceSuite - KubernetesLocalDiskShuffleDataIOSuite - LocalDirsSuite - RDDCleanerSuite - ALSCleanerSuite In addition, this fixes a UT in `KubernetesLocalDiskShuffleDataIOSuite`. ### Why are the changes needed? `ShuffleSuite` is extended by four classes but only `SortShuffleSuite` does the clean-up correctly. ``` ShuffleSuite - SortShuffleSuite - ShuffleNettySuite - ShuffleOldFetchProtocolSuite - ExternalShuffleServiceSuite ``` Since `KubernetesLocalDiskShuffleDataIOSuite` is looking for the other storage directory, the leftover of `ShuffleSuite` causes flakiness. - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-3.2/2649/testReport/junit/org.apache.spark.shuffle/KubernetesLocalDiskShuffleDataIOSuite/recompute_is_not_blocked_by_the_recovery/ ``` org.apache.spark.SparkException: Job aborted due to stage failure: task 0.0 in stage 1.0 (TID 3) had a not serializable result: org.apache.spark.ShuffleSuite$NonJavaSerializableClass ... org.apache.spark.shuffle.KubernetesLocalDiskShuffleDataIOSuite.$anonfun$new$2(KubernetesLocalDiskShuffleDataIOSuite.scala:52) ``` For the other suites, the clean-up implementation is used but not complete. So, they are refactored to use new trait. ### Does this PR introduce _any_ user-facing change? No, this is a test-only change. ### How was this patch tested? Pass the CIs. Closes apache#32986 from dongjoon-hyun/SPARK-35832. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
What changes were proposed in this pull request?
To make the test suite more robust, this PR aims to add a new trait,
LocalRootDirsTest, by refactoringSortShuffleSuite's helper functions and applying it to the following:In addition, this fixes a UT in
KubernetesLocalDiskShuffleDataIOSuite.Why are the changes needed?
ShuffleSuiteis extended by four classes but onlySortShuffleSuitedoes the clean-up correctly.Since
KubernetesLocalDiskShuffleDataIOSuiteis looking for the other storage directory, the leftover ofShuffleSuitecauses flakiness.For the other suites, the clean-up implementation is used but not complete. So, they are refactored to use new trait.
Does this PR introduce any user-facing change?
No, this is a test-only change.
How was this patch tested?
Pass the CIs.