improve test stability

[wilsonwang371]

Why are these changes needed?

1. improve compatibility test stability by increasing the initial wait time in liveness/readiness probe.
2. remove unnecessary gofumpt run in Makefile.

Related issue number

N/A

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[Jeffwan]

What's the reason the test takes that long to finish?

[wilsonwang371]

@Jeffwan I did some investigation, it looks like the root cause for the failure and long waiting time is because in our test environment, the initial delay seconds is too small. I am working on this right now.

[Jeffwan]

Currently seems it has two issues

1. flaky and unstable issue - sometimes succeed and sometimes failed
2. takes long time to finish

If we can resolve the flake issue and reduce running time to 10mins, that would be great.

[wilsonwang371]

Currently seems it has two issues

1. flaky and unstable issue - sometimes succeed and sometimes failed
2. takes long time to finish

If we can resolve the flake issue and reduce running time to 10mins, that would be great.

The first one is a more urgent task. After the first one fixed, we will work on the second one.

The root cause of the flaky test is due to slow ray cluster start in kind environment. we need to update the probe timeout parameters.

[wilsonwang371]

From what i observed, when we killed the head node in a ray cluster in our github testbed, the worker readiness probe can fail for more than 30 seconds. This will lead to our default readiness probe getting involved and kill the worker node.

I increased the readiness probe timeout to 40 seconds for now.

FYI: @iycheng @brucez-anyscale @Jeffwan @scarlet25151 @DmitriGekhtman

[brucez-anyscale]

@iycheng I think the worker node readiness or liveness do not depend on head node. Because if worker node cannot connect head node, it will return healthy.
Can you confirm?

[fishbone]

Yes. Tested here https://github.com/ray-project/ray/blob/master/dashboard/modules/healthz/tests/test_healthz.py#L32

[brucez-anyscale]

@wilsonwang371 Can you confirm the worker issue is caused by head node down?

[wilsonwang371]

@wilsonwang371 Can you confirm the worker issue is caused by head node down?

this is due to probe timeout is not big enough for kind cluster