[Feature] Running end-to-end tests on local machine

[kevin85421]

Why are these changes needed?

We need to download (docker pull) all necessary docker images before we load the images into a KinD cluster. Without this, the command kind load docker-image kuberay/apiserver:nightly will report an error message as follows.

ERROR: image: "kuberay/apiserver:nightly" not present locally

In addition, we also use kind load command to preload the rayproject/ray image into KinD cluster. Although the Kubernetes cluster will pull the ray image automatically when we create a custom resource RayCluster, pulling rayproject/ray:1.9.0 (2.44GB) may cause ImagePullBackOff (Link) due to timeout. In addition, the RayCluster startup time will vary based on network speed and make the tests flaky.

Related issue number

Closes #590

Checks

cd /path/to/your/kuberay
python3 tests/compatibility-test.py

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[wilsonwang371]

I just realized we may need to separate these functions into different files. that will be easier for our future updates. I posted another patch #591 and this should make our changes to ci tests more convenient.

[kevin85421]

I just realized we may need to separate these functions into different files. that will be easier for our future updates. I posted another patch #591 and this should make our changes to ci tests more convenient.

Thank you for this patch! Feel free to ping me for review when it passes CI/CD tests.

[kevin85421]

I found an interesting bug when I tried to use argparse to simplify the logic of parse_environments() (See 6751bc9 for more details.). The following simple scripts can reproduce the bugs.

• test1.py

   import argparse
   import unittest
  
   if __name__ == '__main__':
       parser = argparse.ArgumentParser()
       parser.add_argument('--input', '-i', dest='input',
                           required=False, default='Hello KubeRay')
       args = parser.parse_args()
       print(args.input)
       unittest.main()

• test2.py

   import argparse
   import unittest
  
   if __name__ == '__main__':
       parser = argparse.ArgumentParser()
       parser.add_argument('--input', '-i', dest='input',
                           required=False, default='Hello KubeRay')
       args = parser.parse_args()
       print(args.input)

• Experiment1: python3 test1.py --input haha

  $ python3 test.py --input haha
  haha
  usage: test.py [-h] [-v] [-q] [--locals] [-f] [-c] [-b] [-k TESTNAMEPATTERNS]
                [tests [tests ...]]
  test.py: error: unrecognized arguments: --input
  (base)

• Experiment2: python3 test2.py --input haha

  $ python3 test.py --input haha
  haha
  (base)

The error message "unrecognized arguments: --input" in Experiment1 is reported by unittest.main(). This is because unittest.main() implicitly read sys.argv (See Link for more details.)

[DmitriGekhtman]

Thanks, this is a great improvement.