[Feature] Test sample RayCluster YAMLs to catch invalid or out of date ones

[kevin85421]

Why are these changes needed?

Use #605 to test sample RayCluster YAMLs. This PR found:

• Invalid YAML: Remove ray-cluster.without-block.yaml #675
• Outdated YAML: ray-cluster.getting-started.yaml
• Bug: [Bug] label rayNodeType is useless #679

Related issue number

Closes #642

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

cd tests/framework
python3 test_sample_raycluster_yamls.py 2>&1 | tee log

[DmitriGekhtman]

Will lgtm once you get the new test to pass consistently.

[kevin85421]

@DmitriGekhtman There are 4 tests hit the CPU constraint ("a standard Linux runner has 2-core CPU (x86_64), 7GB of RAM, and 14GB of SSD space.") of GitHub Actions as we discussed in the Design: KubeRay E2E Configuration tests.

I used os.system(f'kubectl describe pods -n={self.namespace}') to gather more information when RayClusterAddCREvent fails to converge. You can search "Insufficient cpu" in CI result.

Events:
  Type     Reason            Age               From               Message
  ----     ------            ----              ----               -------
  Warning  FailedScheduling  8s (x3 over 92s)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu.

[DmitriGekhtman]

Ok, it seems that to automate test execution, we will have to run these tests in the Ray CI.

Here's a proposed sequence of actions.

For this PR, make sure the tests are passing manually when you run them. You can remove the build step.

After merging this PR, the main priority is manual release testing:

The Ray 2.1.0 release is in progress and a more-or-less final rayproject/ray:2.1.0 image is available.
Run the tests with the 2.1.0 images manually and make sure they pass.
(It is possible that some minor commits will be picked in and the image will be modified before release... just ignore that for now. Or if you like repeat the tests when the Ray team is approaching the final 2.1.0 release -- should be within the next few days; you can check the relevant Anyscale-internal slack channel for details.)

Prior to the KubeRay 0.4.0 release, run the tests manually again with a KubeRay 0.4.0 candidate and Ray 2.1.0.

The next priority is establishing automated pipelines for running these tests in the Ray CI. It's important to track this and do it eventually, but it's fine if we don't put a deadline on it right now.

[DmitriGekhtman]

If there's a way to fit some subset of the test into CI, perhaps with modifications to the configs, that would also be good.

[kevin85421]

If there's a way to fit some subset of the test into CI, perhaps with modifications to the configs, that would also be good.

This PR runs 3 tests on KubeRay CI, and I will open issue #695 to track the progress of running tests on Ray CI.

Although each standard Linux runner has 2 cores, I always failed to schedule Pods when CPU usage is higher than 800m (0.8 CPU). Maybe there are some CPU fragmentation problems.

[DmitriGekhtman]

Looks good!

raycluster.complete.yaml and raycluster.autoscaler.yaml are important ones to test in the Ray CI.

[davidxia]

Thanks, just wondering if this would've caught the bug fixed by #501?

[DmitriGekhtman]

Probably not in its current form, but it wouldn't be too hard to extend the framework to validate log volume mounts.