[Feature] Refactor test framework & test kuberay-operator chart with configuration framework

[kevin85421]

Why are these changes needed?

1. Test operator chart: This PR uses the kuberay-operator chart to install KubeRay operator. Hence, the operator chart is tested.

2. Refactor: class CONST and class KubernetesClusterManager should be singleton classes. However, the singleton design pattern is not encouraged, so we need to consider it thoroughly before we convert these two classes into singleton classes.

3. Refactor: Replace os with subprocess. The following paragraph is from Python's official documentation.

   The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function. See the Replacing Older Functions with the subprocess Module section in the subprocess documentation for some helpful recipes.

4. Skip test_kill_head due to

   • I observed that sometimes workers crash when the head pod is killed.

5. Refactor: Replace all existing k8s api clients with K8S_CLUSTER_MANAGER.

6. Refactor and relieve flakiness of test_ray_serve_work

   • working_dir is out-of-date (See this comment for more details), but the tests pass sometimes due to the error of the original test logic. => Solution: Update working_dir in ray-service.yaml.template.
   • To elaborate, the error of the test logic mentioned above is that it only checks the exit code rather than STDOUT.
   • When Pods are READY and RUNNING, RayService still needs tens of seconds to be ready for serving requests. The time.sleep(60) function is a workaround, and should be removed when [RayService] Track whether Serve app is ready before switching clusters #730 is merged.
   • Remove NodePort service in RayServiceTestCase. Use a curl Pod to communicate with Ray via ClusterIP service directly. Originally, using Docker container with network_mode='host' and NodePort service is very weird for me.

7. Refactor: remove useless RayService template ray-service-cluster-update.yaml.template and ray-service-serve-update.yaml.template. The original buggy test logic only checks the exit code rather than the STDOUT of the curl commands. Hence, the different templates are useless in RayServiceTestCase.

8. Refactor: Because APIServer is not tested by any test case, remove everything related to APIServer docker image in the compatibility test.

Follow up issues (@jasoonn and I will finish them):

• [Feature] Add a hook to check static Rules #767
• [Feature] Improve the observability of integration tests #766
• [Feature] Replace Docker container and NodePort with Kubernetes Pod #764
• [Feature] E2E Test plan for RayService #762
• [Feature] E2E Test plan for RayJob  #763
• [Feature] Make KubernetesClusterManager and CONST in test framework become singletons #771

Related issue number

Part of #184

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

Flakiness (no retry)

• 445b61f: Pass
• d599c71: Pass
• d977b2b: Fail
  • Ray client cannot connect to Ray cluster (test_detach_actor_1.py) => This is weird, and may be caused by the weird pattern of Docker container and Kubernetes NodePort service.
  • curl: (7) Failed to connect to rayservice-sample-serve-svc.default.svc.cluster.local port 8000: Connection refused command terminated with exit code 7 => This should be fixed by [RayService] Track whether Serve app is ready before switching clusters #730
• 829195b: Pass
• 78849d6: Fail
  • Exception: There was an exception during the execution of test_ray_serve_1.py. => May be caused by Docker container and NodePort service.
• 66107f9: Fail
  • Exception: There was an exception during the execution of test_ray_serve_1.py.
  • curl: (7) Failed to connect to rayservice-sample-serve-svc.default.svc.cluster.local port 8000: Connection refused command terminated with exit code 7 => This should be fixed by [RayService] Track whether Serve app is ready before switching clusters #730

[kevin85421]

cc @jasoonn Would you mind reviewing this PR? Thank you!

[jasoonn]

@kevin85421 Thanks for the contribution!

[DmitriGekhtman]

Skip test_kill_head.

Could you give context on that decision? Is the test complete nonsense or do we need to follow up -- do we need to fix it or write a test that covers similar logic?

[DmitriGekhtman]

Refactor: remove useless RayService template ray-service-cluster-update.yaml.template and ray-service-serve-update.yaml.template. The original buggy test logic only checks the exit code rather than the STDOUT of the curl commands. Hence, the different templates are useless in RayServiceTestCase.

Could you make a note in the code and on the RayService test issue that

• The previous logic for testing updates was problematic
• We need to test RayService updates

[DmitriGekhtman]

Some minor style complaints above.

Overall this looks great! The KubeRay CI should be much greener after these changes.

[DmitriGekhtman]

I think you can do .parents[3].
But .parent.parent.parent is fine too.

[kevin85421]

Skip test_kill_head.

Could you give context on that decision? Is the test complete nonsense or do we need to follow up -- do we need to fix it or write a test that covers similar logic?

• We need to rewrite the test (for example, remove kubectl wait).
• I skipped this test due to
  • [Bug] Head pod is deleted rather than restarted when gcs_server on head pod is killed. #638
  • [Bug] Worker pods crash unexpectedly when gcs_server on head pod is killed  #634

[DmitriGekhtman]

!!!

[kevin85421]

• I skipped this test due to

  • [Bug] Head pod is deleted rather than restarted when gcs_server on head pod is killed. #638
  • [Bug] Worker pods crash unexpectedly when gcs_server on head pod is killed  #634

@DmitriGekhtman These are wrong. These two issues are related to test_detached_actor and test_ray_serve. However, I observed that sometimes workers crash when the head pod is killed.