[Bug] Misuse of Docker API and misunderstanding of Ray HA cause test_ray_serve flaky

[kevin85421]

Search before asking

• I searched the issues and found no similar issues.

KubeRay Component

ci

What happened + What you expected to happen

See #620.

Reproduction script

See #620.

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[kevin85421]

cc @jasoonn

The following links may be useful.

• [Bug] Misuse of Docker API and misunderstanding of Ray HA cause test_detached_actor flaky #619
• [Bug] Misuse of Docker API and misunderstanding of Ray HA cause test_detached_actor flaky  #620

How does the docker container communicate with the k8s cluster?

• NodePort Service: head pod <--> local
• Docker (network_mode: host): container <--> local

You can try curl 127.0.0.1:10001 (valid port) in both your local laptop and the container. You will get curl: (56) Recv failure: Connection reset by peer. You can also run curl 127.0.0.1:10002 (invalid port), and you will get curl: (7) Failed to connect to 127.0.0.1 port 10002: Connection refused.

The following command enables you to run a single test in the test suite. In addition, both STDOUT (1) and STDERR (2) will be redirected to file log.

RAY_IMAGE=rayproject/ray:2.0.0 python3 tests/compatibility-test.py RayFTTestCase.test_ray_serve 2>&1 | tee log

[jasoonn]

@kevin85421
Thanks a lot for the information. I will work on it!