-
Notifications
You must be signed in to change notification settings - Fork 364
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
improve test stability #394
improve test stability #394
Conversation
84dd8f1
to
f38add5
Compare
4829dd1
to
7770740
Compare
@Jeffwan I did some investigation, it looks like the root cause for the failure and long waiting time is because in our test environment, the initial delay seconds is too small. I am working on this right now. |
52a6f15
to
d4b4e90
Compare
d4b4e90
to
f179f06
Compare
Currently seems it has two issues
If we can resolve the flake issue and reduce running time to 10mins, that would be great. |
f179f06
to
e28e3ab
Compare
The first one is a more urgent task. After the first one fixed, we will work on the second one. The root cause of the flaky test is due to slow ray cluster start in kind environment. we need to update the probe timeout parameters. |
8615eb0
to
612916a
Compare
From what i observed, when we killed the head node in a ray cluster in our github testbed, the worker readiness probe can fail for more than 30 seconds. This will lead to our default readiness probe getting involved and kill the worker node. I increased the readiness probe timeout to 40 seconds for now. FYI: @iycheng @brucez-anyscale @Jeffwan @scarlet25151 @DmitriGekhtman |
517b980
to
bb0ccc1
Compare
bb0ccc1
to
c82441d
Compare
@iycheng I think the worker node readiness or liveness do not depend on head node. Because if worker node cannot connect head node, it will return healthy. |
@wilsonwang371 Can you confirm the worker issue is caused by head node down? |
this is due to probe timeout is not big enough for kind cluster |
Why are these changes needed?
Related issue number
N/A
Checks