[Feature] Test sample RayService YAML to catch invalid or out of date one

[jasoonn]

Why are these changes needed?

Use #605 to test sample RayService YAML.
The test includes the following two checking rules.

1. It is able to submit a simple job to the created raycluster.
2. It is able to use curl in the pod to access the service.

Why we choose Subprocess instead of os.system

According to doc of os.system, the subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using subprocess is preferable to using this function.

• shell=True in subprocess.run(): The specified command will be executed through the shell.
• check=True in some of subprocess.run(): If the process exits with a non-zero exit code, a CalledProcessError exception will be raised.

Why removing the kubectl wait when creating kind cluster

kuberay/tests/framework/prototype.py

Lines 46 to 50 in 12225db

	def create_kind_cluster():
	"""Create a KinD cluster"""
	shell_subprocess_run("kind create cluster")
	shell_subprocess_run(
	"kubectl wait --for=condition=ready pod -n kube-system --all --timeout=900s")

Sometimes when calling "kubectl wait --for=condition=ready pod -n kube-system --all --timeout=900s", it will report an error error: no matching resources found(example log in line 1157), which I think because the namespace has not yet been created. When this situation happened, the test in the past still passed because we only use os.system() to execute the command and do not check the return code. In this pr, we updated os.system to subprocess.run(..., check=True) which will automatically raise an error when the return code of the exection is not 0. To solve this, we utilize --wait flag in kind to substitute the kubectl wait functionality.

Why we set tail=-1 in getting log?

kuberay/tests/framework/prototype.py

Line 128 in 48b0b05

shell_subprocess_run(f'kubectl logs -n={cr_namespace} -l ray.io/node-type=head --tail=-1')

The argument tail default is -1 with no selector, showing all log lines. However, we utilize a selector in this case. Hence, we need to specify --tail=-1 to display all of the log.

Related issue number

Closes #719

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

cd tests/framework
python3 test_sample_rayservice_yamls.py 2>&1 | tee log
# I have run more than ten times on my cluster, and the result is always pass.

[kevin85421]

Thank @jasoonn for your contribution!

[jasoonn]

@kevin85421 Thanks for the suggestions. I have updated the pr.

[kevin85421]

@shrekris-anyscale Can you help review the wait function of RayServiceAddCREvent? The function waits until RayService is ready to serve traffic. Thank you!

[jasoonn]

@kevin85421 Thanks for the suggestions. I have updated the pr.

[kevin85421]

There are too many shell = ..., check = .... Can we simplify it?

[jasoonn]

@kevin85421 Thanks for the suggestion. I have updated the pr.

[jasoonn]

@kevin85421 Thanks for the suggestion. I have updated the pr.

[kevin85421]

LGTM

[sihanwang41]

Hi @kevin85421 @jasoonn , is this test enabled in ci? (I don't see it is enabled.)

[kevin85421]

Hi @kevin85421 @jasoonn , is this test enabled in ci? (I don't see it is enabled.)

@sihanwang41 No, KubeRay CI (2 CPUs) does not have enough CPUs to run this test. I will deploy these tests in Ray CI (#695).