Add end to end tests to apiserver #1460

z103cb · 2023-09-29T11:24:53Z

Why are these changes needed?

The kuberay api server does not have an easy to execute set of test cases that would allow for the validation of the integration between it and the kuberay operator. This PR provides a basic set of such integration test cases.

This PR:

Adds the testcases
Adds the make targets to facilitate the test cases
Updates the documentation for development of kuberay api server.
Updates the github CI pipeline to execute only the unit tests for api server
Minor documentation updates (formatting)

Related issue number

Closes #1388

Checks

z103cb · 2023-09-29T11:31:25Z

.github/workflows/test-job.yaml

@@ -144,7 +144,7 @@ jobs:
        working-directory: ${{env.working-directory}}

      - name: Test
-        run: go test ./...
+        run: go test ./pkg/... ./cmd/... -race -parallel 4


@kevin85421 let me know if this change is acceptable. If not made the e2e test would execute, and fail as there's no kind cluster setup at this point of the pipeline.

Can we use make test instead?

Technically, I don't see a reason why make test cannot be here. I did not add it as it would have been different than what is happening for the other steps in the CI pipeline and I did not seem to be a changed related to the main scope of the PR.

I would prefer that we change the CI (Git hub actions) for the API Server in separate PR and perhaps integrate the execution of the e2e tests with build kite.

I would not change more in the GH action than I have already done.

Sounds good

apiserver/Makefile

z103cb · 2023-09-29T11:34:49Z

I still need to a a few more test cases and fixes, but I would like to get some feedback on what I have so far.

@tedhtchang, @blublinsky and @anishasthana can provide feedback ?
cc: @kevin85421

blublinsky · 2023-09-29T14:36:02Z

This looks great. Makes things so much simpler

apiserver/Makefile

apiserver/DEVELOPMENT.md

apiserver/Makefile

apiserver/hack/kind-cluster-config.yaml

tedhtchang

I had some small issues(see comment). Otherwise, this looks good. I was able to run make operator-image local-e2e-test -e OPERATOR_IMAGE_TAG=latest -e GO_TEST_FLAGS="-v" successfully.

z103cb · 2023-10-12T09:35:31Z

@astefanutti can you take a look ?

astefanutti

Looks great. Could a GH Actions workflow be created so it runs in CI?

apiserver/test/e2e/cluster_server_e2e_test.go

.github/workflows/test-job.yaml

blublinsky

Two suggestions, but overall looks good

apiserver/Volumes.md

apiserver/test/e2e/resources/counter_sample.py

blublinsky

LGTM

tedhtchang · 2023-10-28T01:11:13Z

I kept failing the same test for some reason:

              Error Trace:  /root/kuberay/apiserver/test/e2e/service_server_e2e_test.go:911
                                  /root/kuberay/apiserver/test/e2e/service_server_e2e_test.go:746
              Error:        Received unexpected error:
                            timed out waiting for the condition
              Test:         TestServiceServerV1Patch/Update_service_cluster_worker_group
              Messages:     No error expected when getting ray service: 'shepherd', err timed out waiting for the condition
FAIL
FAIL  github.com/ray-project/kuberay/apiserver/test/e2e 888.117s

apiserver/test/e2e/config_server_e2e_test.go

tedhtchang

/LGTM

z103cb · 2023-11-01T12:12:58Z

@kevin85421, let me know if you have any concerns with this PR, if not can you please merge it ?

Fixes ray-project#1388

kevin85421

Thank you for the contribution! Testing is always the most important part of the KubeRay community.

kevin85421 · 2023-11-15T16:47:00Z

Rerun the CI. There seem to be some changes in the Ray nightly build causing flakiness.

z103cb force-pushed the z103cb/issue1388 branch from 873b2a7 to 5b22dc3 Compare September 29, 2023 11:28

z103cb commented Sep 29, 2023

View reviewed changes

z103cb mentioned this pull request Oct 2, 2023

[apiserver] Mark required fields in .proto #1438

Merged

4 tasks

tedhtchang reviewed Oct 2, 2023

View reviewed changes

apiserver/Makefile Outdated Show resolved Hide resolved

z103cb force-pushed the z103cb/issue1388 branch 3 times, most recently from 68ec386 to 1de910a Compare October 5, 2023 08:16

tedhtchang reviewed Oct 6, 2023

View reviewed changes

apiserver/DEVELOPMENT.md Outdated Show resolved Hide resolved

tedhtchang reviewed Oct 6, 2023

View reviewed changes

apiserver/Makefile Outdated Show resolved Hide resolved

tedhtchang reviewed Oct 7, 2023

View reviewed changes

apiserver/hack/kind-cluster-config.yaml Outdated Show resolved Hide resolved

tedhtchang reviewed Oct 7, 2023

View reviewed changes

z103cb mentioned this pull request Oct 10, 2023

created HA example for API server #1461

Merged

1 task

z103cb force-pushed the z103cb/issue1388 branch 3 times, most recently from a4e36c6 to 89193e5 Compare October 12, 2023 08:51

z103cb force-pushed the z103cb/issue1388 branch from 89193e5 to b3c7848 Compare October 13, 2023 19:29

astefanutti reviewed Oct 16, 2023

View reviewed changes

apiserver/test/e2e/cluster_server_e2e_test.go Outdated Show resolved Hide resolved

apiserver/test/e2e/cluster_server_e2e_test.go Outdated Show resolved Hide resolved

.github/workflows/test-job.yaml Show resolved Hide resolved

z103cb force-pushed the z103cb/issue1388 branch 4 times, most recently from 1be8fd7 to a95cb9c Compare October 23, 2023 07:27

z103cb force-pushed the z103cb/issue1388 branch 2 times, most recently from 2336828 to 903f28b Compare October 27, 2023 09:49

z103cb marked this pull request as ready for review October 27, 2023 09:50

blublinsky approved these changes Oct 27, 2023

View reviewed changes

apiserver/Volumes.md Outdated Show resolved Hide resolved

apiserver/test/e2e/resources/counter_sample.py Show resolved Hide resolved

blublinsky approved these changes Oct 27, 2023

View reviewed changes

z103cb force-pushed the z103cb/issue1388 branch from 66ae42c to 6c8cc28 Compare October 31, 2023 13:18

tedhtchang reviewed Oct 31, 2023

View reviewed changes

apiserver/test/e2e/config_server_e2e_test.go Show resolved Hide resolved

z103cb mentioned this pull request Nov 1, 2023

[Bug] KubeRay API Server ListAllComputeTemplates is intermittently failing. #1591

Open

2 tasks

tedhtchang approved these changes Nov 1, 2023

View reviewed changes

kevin85421 self-assigned this Nov 13, 2023

z103cb added 2 commits November 15, 2023 09:08

[Feature] Add end to end tests to apiserver

cf17e46

Fixes ray-project#1388

[apiserver] e2e test updates

41934c0

z103cb force-pushed the z103cb/issue1388 branch from 045c364 to 41934c0 Compare November 15, 2023 07:09

kevin85421 approved these changes Nov 15, 2023

View reviewed changes

kevin85421 merged commit ff45923 into ray-project:master Nov 15, 2023
24 checks passed

blublinsky pushed a commit to blublinsky/kuberay that referenced this pull request Nov 16, 2023

Add end to end tests to apiserver (ray-project#1460)

9ec5ed5

z103cb deleted the z103cb/issue1388 branch February 5, 2024 14:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add end to end tests to apiserver #1460

Add end to end tests to apiserver #1460

z103cb commented Sep 29, 2023

z103cb Sep 29, 2023

kevin85421 Nov 13, 2023

z103cb Nov 15, 2023

kevin85421 Nov 15, 2023

z103cb commented Sep 29, 2023

blublinsky commented Sep 29, 2023

tedhtchang left a comment

z103cb commented Oct 12, 2023 •

edited

astefanutti left a comment

blublinsky left a comment

blublinsky left a comment

tedhtchang commented Oct 28, 2023

tedhtchang left a comment

z103cb commented Nov 1, 2023

kevin85421 left a comment

kevin85421 commented Nov 15, 2023

Add end to end tests to apiserver #1460

Add end to end tests to apiserver #1460

Conversation

z103cb commented Sep 29, 2023

Why are these changes needed?

Related issue number

Checks

z103cb Sep 29, 2023

Choose a reason for hiding this comment

kevin85421 Nov 13, 2023

Choose a reason for hiding this comment

z103cb Nov 15, 2023

Choose a reason for hiding this comment

kevin85421 Nov 15, 2023

Choose a reason for hiding this comment

z103cb commented Sep 29, 2023

blublinsky commented Sep 29, 2023

tedhtchang left a comment

Choose a reason for hiding this comment

z103cb commented Oct 12, 2023 • edited

astefanutti left a comment

Choose a reason for hiding this comment

blublinsky left a comment

Choose a reason for hiding this comment

blublinsky left a comment

Choose a reason for hiding this comment

tedhtchang commented Oct 28, 2023

tedhtchang left a comment

Choose a reason for hiding this comment

z103cb commented Nov 1, 2023

kevin85421 left a comment

Choose a reason for hiding this comment

kevin85421 commented Nov 15, 2023

z103cb commented Oct 12, 2023 •

edited