Allow E2E tests to run with arbitrary k8s cluster #1306

jiripetrlik · 2023-08-09T14:19:58Z

Why are these changes needed?

Currently E2E tests run only with Kind cluster. The goal of this PR is to make these tests more configurable and allow it to run with any Kubernetes cluster. It is now possible to login to arbitrary cluster using kubectl/oc command and run tests with arbitrary cluster. For example:

oc login ...
EXTERNAL_CLUSTER=true RAY_IMAGE=rayproject/ray:2.5.0 OPERATOR_IMAGE=kuberay/operator:nightly python3 tests/compatibility-test.py

Related issue number

Closes #1284

Checks

[] I've made sure the tests are passing.
Testing Strategy
- [] Unit tests
- [] Manual tests
- [*] This PR is not tested :(

jiripetrlik · 2023-08-09T14:26:17Z

@kevin85421
Hello, can you please provide initial feedback?

kevin85421 · 2023-08-10T00:37:35Z

tests/framework/utils.py


-    def create_kind_cluster(self, kind_config=None) -> None:
+    # def create_kind_cluster(self, kind_config=None) -> None:


forget to remove?

tests/framework/utils.py

kevin85421 · 2023-08-10T00:44:18Z

tests/framework/utils.py

+        return self.client_dict
+
+    def cleanup(self, namespace = "default") -> None:
+        config.load_kube_config()


Why do we need to load_kube_config again if we have already loaded the config in initialize_cluster?

kevin85421 · 2023-08-10T00:47:47Z

tests/framework/utils.py

+    def __init__(self) -> None:
+        self.client_dict = {}
+
+    def client_dict(self):


This function seems to be unused.

kevin85421 · 2023-08-10T00:51:29Z

tests/framework/utils.py

+
+    def cleanup(self, namespace = "default") -> None:
+        config.load_kube_config()
+        api_extensions = client.ApiextensionsV1Api()


Using multiple Kubernetes client instances will cause flakiness. Please reuse the client from self.client_dict and ensure there is only one Kubernetes client instance in this process.

kevin85421 · 2023-08-10T00:57:34Z

tests/framework/utils.py

+    def client_dict(self):
+        return self.client_dict
+
+    def cleanup(self, namespace = "default") -> None:


Why not delete the cluster directly?

kevin85421 · 2023-08-10T01:07:54Z

tests/framework/utils.py

+    def client_dict(self):
+        return self.client_dict
+
+    def cleanup(self, namespace = "default") -> None:


We should also close Kubernetes clients and cleanup self.client_dict.

kevin85421 · 2023-08-10T01:08:40Z

tests/framework/utils.py

+
+        time.sleep(15)
+
+        if "kuberay-operator" in [deployment.metadata.name for deployment in apps_v1.list_namespaced_deployment(namespace).items]:


Why do we need this if we will run helm uninstall below?

kevin85421 · 2023-08-10T01:10:14Z

tests/framework/utils.py

+        apps_v1 = client.AppsV1Api()
+        custom_objects_api = client.CustomObjectsApi()
+        crds = api_extensions.list_custom_resource_definition()
+        for crd in crds.items:


Custom resources should be cleaned up by the CREvent instead of the cluster manager.

kevin85421 · 2023-08-10T01:11:15Z

tests/framework/utils.py

+                    namespace = cr["metadata"]["namespace"]
+                    custom_objects_api.delete_namespaced_custom_object(group, version, namespace, plural, name)
+
+        time.sleep(15)


Can we not hardcode the sleep time?

jiripetrlik · 2023-08-30T21:42:37Z

Thank you @kevin85421 for your review! I tried to address all your comments. Except that I do not use CREvent for deleting Ray custom resources. I rather use clean up method which can be general for all tests. I'm sorry for late reply. It was due to my vacation.

kevin85421 · 2023-09-05T19:07:25Z

The sample YAML tests seem to fail consistently. Would you mind running the sample YAML tests locally? See https://github.com/ray-project/kuberay/blob/master/ray-operator/DEVELOPMENT.md#running-configuration-tests-locally for more details.

tests/framework/utils.py

kevin85421 · 2023-09-08T21:35:00Z

tests/framework/utils.py

+        """Check whether cluster exists or not"""
+        return (
+            shell_subprocess_run(
+                "kubectl cluster-info --context kind-kind", check=False


kind-kind is the default context for Kind.

kevin85421 · 2023-09-08T21:40:15Z

tests/framework/utils.py

+        )
+
+    def __delete_all_crs(self, group, version, namespace, plural):
+        custom_objects_api = self.k8s_client_dict[CONST.K8S_CR_CLIENT_KEY]


The custom resource lifecycles are better controlled by CREvent (link). In my opinion, the cleanup at the cluster level is not necessary. We can remove this function.

kevin85421 · 2023-09-08T21:41:44Z

tests/framework/utils.py

+        self.cleanup_timeout = 120
+
+    def cleanup(self, namespace = "default") -> None:
+        self.__delete_all_crs("ray.io", "v1alpha1", namespace, "rayservices")


The custom resource lifecycles are better controlled by CREvent (link). In my opinion, the cleanup at the cluster level is not necessary. We can remove this function.

kevin85421 · 2023-09-08T21:43:23Z

tests/framework/utils.py

+        )
+        self.cleanup_timeout = 120
+
+    def cleanup(self, namespace = "default") -> None:


In ClusterManager's definition, cleanup needs to delete the Kubernetes cluster, but this function does not do that.

jiripetrlik · 2023-09-11T13:11:16Z

Hello @kevin85421 thank you for the review. The point why I do not want to delete cluster itself in ExternalClusterManager -> cleanup method is that I want to run tests even with clusters which I can not easily provision or tear down. For example to provision and tear down some Openshift cluster may take half an hour or something, which is probably not doable with so many tests. Instead I've decided to implement full cluster wide clean up. This behavior is enabled by EXTERNAL_CLUSTER=true. If you do not enable it then the behavior of tests is the same as before. It creates Kind cluster and delete Kind cluster the same way as before. See cleanup method for KindClusterManager.

kevin85421 · 2023-09-13T21:21:26Z

cleanup method is that I want to run tests even with clusters which I can not easily provision or tear down. For example to provision and tear down some Openshift cluster may take half an hour or something, which is probably not doable with so many tests.

Thank you for the explanation! It makes sense to me. Some CR YAMLs not only define CR but also other resources, such as ConfigMap. Some tests may have side effects after the cluster-level cleanup, but it is fine to open follow-up PRs to fix that if any.

kevin85421

LGTM

kevin85421 · 2023-09-13T21:22:07Z

The RayService e2e tests are known to be flaky. It is not related to this PR.

Allow E2E tests to run with arbitrary k8s cluster

kevin85421 self-requested a review August 9, 2023 16:33

kevin85421 self-assigned this Aug 9, 2023

kevin85421 reviewed Aug 10, 2023

View reviewed changes

jiripetrlik force-pushed the 1284-arbitrary-cluster branch from a164211 to 4a8e49d Compare August 30, 2023 20:45

jiripetrlik force-pushed the 1284-arbitrary-cluster branch 3 times, most recently from 1cec918 to 60bbf1e Compare September 7, 2023 21:50

kevin85421 reviewed Sep 8, 2023

View reviewed changes

Allow E2E tests to run with arbitrary k8s cluster

a37388f

jiripetrlik force-pushed the 1284-arbitrary-cluster branch from 60bbf1e to a37388f Compare September 12, 2023 20:45

kevin85421 approved these changes Sep 13, 2023

View reviewed changes

kevin85421 merged commit 59d703f into ray-project:master Sep 13, 2023
20 of 21 checks passed

lowang-bh pushed a commit to lowang-bh/kuberay that referenced this pull request Sep 24, 2023

Allow E2E tests to run with arbitrary k8s cluster (ray-project#1306)

1007ca2

Allow E2E tests to run with arbitrary k8s cluster

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow E2E tests to run with arbitrary k8s cluster #1306

Allow E2E tests to run with arbitrary k8s cluster #1306

jiripetrlik commented Aug 9, 2023

jiripetrlik commented Aug 9, 2023

kevin85421 Aug 10, 2023

kevin85421 Aug 10, 2023

kevin85421 Aug 10, 2023

kevin85421 Aug 10, 2023

kevin85421 Aug 10, 2023

kevin85421 Aug 10, 2023

kevin85421 Aug 10, 2023

kevin85421 Aug 10, 2023

kevin85421 Aug 10, 2023

jiripetrlik commented Aug 30, 2023

kevin85421 commented Sep 5, 2023

kevin85421 Sep 8, 2023

kevin85421 Sep 8, 2023

kevin85421 Sep 8, 2023

kevin85421 Sep 8, 2023

jiripetrlik commented Sep 11, 2023

kevin85421 commented Sep 13, 2023

kevin85421 left a comment

kevin85421 commented Sep 13, 2023


		def create_kind_cluster(self, kind_config=None) -> None:
		# def create_kind_cluster(self, kind_config=None) -> None:


		time.sleep(15)

		if "kuberay-operator" in [deployment.metadata.name for deployment in apps_v1.list_namespaced_deployment(namespace).items]:

Allow E2E tests to run with arbitrary k8s cluster #1306

Allow E2E tests to run with arbitrary k8s cluster #1306

Conversation

jiripetrlik commented Aug 9, 2023

Why are these changes needed?

Related issue number

Checks

jiripetrlik commented Aug 9, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiripetrlik commented Aug 30, 2023

kevin85421 commented Sep 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiripetrlik commented Sep 11, 2023

kevin85421 commented Sep 13, 2023

kevin85421 left a comment

Choose a reason for hiding this comment

kevin85421 commented Sep 13, 2023