[Hotfix][Bug] suspend is not a stateless operation

[kevin85421]

Why are these changes needed?

• Reproduce the error
  • Step 1: Set suspend to true.
  • Step 2: The status changes to suspended after all Pods are deleted.
  • Step 3: Set suspend back to false; this will trigger the creation of Pods.
  • Step 4: Set suspend back to true before all Pods are fully operational. In this case, the state remains suspended instead of becoming ready.
  • Step 5: At this moment, the KubeRay operator will skip the reconciliation because suspend: true and the status is suspended.

The function reconcilePods should only create Pods when suspend is false and the status is not suspended. However, RayCluster currently does not have a well-defined state machine. To avoid adding complexity to the codebase, I decided not to skip the reconciliation until we have a well-defined state machine.

Related issue number

#1711

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[kevin85421]

cc @andrewsykim This is the follow up for #1711 (comment).

[andrewsykim]

@kevin85421 thanks for fixing! LGTM