[Feature][Hotfix] Add observedGeneration to the status of CRDs

[kevin85421]

Why are these changes needed?

observedGeneration is a common pattern in Kubernetes to determine whether the current status is out-of-date or not.

• Good article: Kubernetes operator best practices: Implementing observedGeneration

• StatefulSet uses ObservedGeneration to determine whether update status or not.

  // inconsistentStatus returns true if the ObservedGeneration of status is greater than set's
  // Generation or if any of the status's fields do not match those of set's status.
  func inconsistentStatus(set *apps.StatefulSet, status *apps.StatefulSetStatus) bool {
      return status.ObservedGeneration > set.Status.ObservedGeneration ||
    	  status.Replicas != set.Status.Replicas ||
    	  status.CurrentReplicas != set.Status.CurrentReplicas ||
    	  status.ReadyReplicas != set.Status.ReadyReplicas ||
    	  status.UpdatedReplicas != set.Status.UpdatedReplicas ||
    	  status.CurrentRevision != set.Status.CurrentRevision ||
    	  status.AvailableReplicas != set.Status.AvailableReplicas ||
    	  status.UpdateRevision != set.Status.UpdateRevision
  }

However, this PR does not use observedGeneration to do anything because:

• Currently, KubeRay lacks a well-defined strategy for updating the status of its resources. Furthermore, it's possible for multiple status updates to occur within a single reconciliation loop, which can lead to problematic patterns and outcomes.
• As a result, merging a pull request that includes any changes related to status updates shortly before the release of v0.5.0 is very risky.

The main motivation for this PR is to unblock Shopify. Other works will be included in the release v0.6.0.

ObservedGeneration definition at this moment

• ObservedGeneration in this PR only promises that ObservedGeneration <= Generation.
• I did not promise any other behavior in this PR because there are a lot of code paths that will update the status and I do not check every path.

Test

• We currently only have a test for RayCluster because we do not have tests with fake clients for RayJob and RayService at this moment. These should be done in the next release which we plan to focus on the stability of RayJob and RayService.

Related issue number

Hotfix for #916

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

• RayCluster

• RayJob

• RayService

[kevin85421]

cc @gariepyalex

[shrekris-anyscale]

Nice work, other than a few nits, this change looks good to me!

Out of curiosity, is the ObjectMeta.Generation set by the user or is it autogenerated by the controller?

[kevin85421]

Out of curiosity, is the ObjectMeta.Generation set by the user or is it autogenerated by the controller?

ObjectMeta.Generation is maintained by Kubernetes's APIServer. The field is a sequence number representing a specific generation of the desired state. Set by the system and monotonically increasing, per-resource. See this for more details.