[SPARK-44049][K8S][TESTS] Fix KubernetesSuite to use inNamespace for validating driver pod cleanup#41586
Closed
dongjoon-hyun wants to merge 1 commit intoapache:masterfrom
dongjoon-hyun:SPARK-44049
Closed
[SPARK-44049][K8S][TESTS] Fix KubernetesSuite to use inNamespace for validating driver pod cleanup#41586dongjoon-hyun wants to merge 1 commit intoapache:masterfrom dongjoon-hyun:SPARK-44049
inNamespace for validating driver pod cleanup#41586dongjoon-hyun wants to merge 1 commit intoapache:masterfrom
dongjoon-hyun:SPARK-44049
Conversation
…r validating driver pod cleanup
dongjoon-hyun
commented
Jun 14, 2023
| Eventually.eventually(TIMEOUT, INTERVAL) { | ||
| assert(kubernetesTestComponents.kubernetesClient | ||
| .pods() | ||
| .inNamespace(kubernetesTestComponents.namespace) |
Member
Author
There was a problem hiding this comment.
This is the same code pattern with line 610.
Member
Author
|
Could you review this when you have some time, @viirya ? |
pan3793
approved these changes
Jun 14, 2023
viirya
approved these changes
Jun 14, 2023
Member
Author
czxm
pushed a commit
to czxm/spark
that referenced
this pull request
Jun 19, 2023
…r validating driver pod cleanup ### What changes were proposed in this pull request? This PR aims to fix `KubernetesSuite` to use `inNamespace` API for validating driver pod cleanup. ### Why are the changes needed? This is a trick bug because of the following two reasons. - Although all test cases passed, currently K8s integration tests are running extremely slowly. - The individual test case running time shows correctly. - The slowness happens during the transition from a test to another test. The main root cause is that K8s test shows `namespace not specified` error after passing tests and this bug blocks every test case at the driver pod clean-up and validation stage `up to 3 minutes` (the maximum timeouts). ``` [info] The code passed to eventually never returned normally. Attempted 190 times over 3.011156453483333 minutes. Last failure message: namespace not specified for an operation requiring one and no default was found in the Config.. (KubernetesSuite.scala:612) ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. Also, I manually tested that the suite took 13 minutes correctly. Previously, it took over 1 hour. ``` [info] YuniKornSuite: [info] - SPARK-42190: Run SparkPi with local[*] (17 seconds, 144 milliseconds) [info] - Run SparkPi with no resources (20 seconds, 406 milliseconds) [info] - Run SparkPi with no resources & statefulset allocation (15 seconds, 531 milliseconds) ... [info] Run completed in 13 minutes, 46 seconds. [info] Total number of tests run: 27 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 27, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. [success] Total time: 842 s (14:02), completed Jun 13, 2023, 9:33:02 PM ``` Closes apache#41586 from dongjoon-hyun/SPARK-44049. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR aims to fix
KubernetesSuiteto useinNamespaceAPI for validating driver pod cleanup.Why are the changes needed?
This is a trick bug because of the following two reasons.
The main root cause is that K8s test shows
namespace not specifiederror after passing tests and this bug blocks every test case at the driver pod clean-up and validation stageup to 3 minutes(the maximum timeouts).Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass the CIs.
Also, I manually tested that the suite took 13 minutes correctly. Previously, it took over 1 hour.