[Bug][GCS FT] Clean up the Redis key before the head Pod is deleted

[kevin85421]

Why are these changes needed?

#1785 checks numDeletedHeads to determine whether to create the Redis cleanup Kubernetes Job or not. However, numDeletedHeads does not always indicate the number of head Pods that have been deleted but rather those that are in the process of being deleted. Hence, since GCS is still running, the Redis key will not be fully deleted.

Related issue number

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

• Create a RayCluster with ray-cluster.external-redis-uri.yaml.
• Delete the RayCluster: kubectl delete rayclusters.ray.io raycluster-external-redis-uri
• Check the Redis after all Ray Pods are deleted.

  kubectl exec -it $YOUR_REDIS_POD -- redis-cli -a "5241590000000000"
  KEYS * # expected behavior: no key (without this PR), 1 key (with this PR)

[kevin85421]

cc @rueian

[jjyao]

Do we have tests?

[kevin85421]

Do we have tests?

We have a check here. However, it is in the tearDown hook, which will not cause the test to fail. In the screenshot below, the test passes although the check fails. We will have a follow up about it.

[kevin85421]

Open an issue to track the progress #1997.