refactor: Fix flaky tests by using RetryOnConflict

[Yicheng-Lu-llll]

Why are these changes needed?

In a previous experiment, it was shown that 86 runs succeed, and 14 runs failed with the following errors：

[Fail] Inside the default namespace When creating a raycluster [It] should update a raycluster object 
/home/lyc/Desktop/kuberay/ray-operator/controllers/ray/raycluster_controller_test.go:279

[Fail] Inside the default namespace When creating a raycluster [It] should have only 1 running worker 
/home/lyc/Desktop/kuberay/ray-operator/controllers/ray/raycluster_controller_test.go:286

[Fail] Inside the default namespace When creating a rayservice [It] should perform a zero-downtime update after a code change. 
/home/lyc/Desktop/kuberay/ray-operator/controllers/ray/rayservice_controller_test.go:354

The above errors are all due to 409 conflict:

• In previous PR Fix flaky tests by retrying 409 conflict error #73, retryOnOldRevision is used in certain places to retry updat call if facing a 409 conflict error. According to the previous experiment, it is necessary to apply the retry strategy to more k8sClient.Update calls especially those causing the above errors.
• Compare retryOnOldRevision with RetryOnConflict , RetryOnConflict gives more advantages. It provides exponential backoff to avoid exhausting the apiserver.

So, two changes are made in this PR:

• Replace retryOnOldRevision with RetryOnConflict
• Apply RetryOnConflict to more k8sClient.Update calls.

If you have concerns about using RetryOnConflict to handle 409 conflict errors in testing or need some background information, see the below links :

• k8s api-conventions document describe the client action in the case of a conflict
• The [ link link ] show how they implement the retry strategy.
• Don’t Retry on Conflict section in this article and comment suggests not to use retry strategy in controllers and explain why. (Note, there are concerns and shortages to using the retry strategy in controllers as described above, but they are not for testing)

Related issue number

Closes #902

Checks

In a 2-core CPU, 7 GB RAM VM (to simulate the github's standard Linux runner), I ran the test 100 times:

rootPath="/home/lyc/Desktop/lessFlasky/kuberay"
cd $rootPath/ray-operator
for i in {1..100};
        do echo "iteration ${i}";
        make test | tee /home/lyc/Desktop/tmp/log${i}
done

All pass with no error. The test is stable now.

[kevin85421]

Can you explain why we need to getResourceFunc here?

[Yicheng-Lu-llll]

Without getResourceFunc , if a 409 conflict happens, the resourceVersion we pass to API server during retry updating will never be updated and thus will never match the resourceVersion in the API server.

[Yicheng-Lu-llll]

Also update the usage of k8s.io/utils/pointer pkg. Some functions are deprecated and have new names. See here.

[kevin85421]

LGTM. Thank you for the fix!