[GCS FT] Improve GCS FT cleanup UX

[kevin85421]

Why are these changes needed?

• Mitigate the penalty if the Redis cleanup Job fails.

  • Minimize the computing resources for the K8s Job.
  • Set the backoffLimit to 0 to avoid 6 dead Pods to overload ResourceQuota's count/pods.

• With the default configuration redis_db_connect_retries = 600 and redis_db_connect_wait_milliseconds = 100, it takes 13 mins to fail. This PR configures the environment variables so that the Job takes 3 mins to fail. See [GCS FT] Update Redis connection configs ray#40860 for more details.

• Add loggings to improve the observability for troubleshooting.

• Note:

  • If we delete the Redis deployment, the Redis cleanup Job cannot connect to Redis, so it retries based on the environment variables mentioned above.
  • If we delete the key in Redis, the Redis cleanup Job can connect to Redis but fails to delete it due to non-existence. Hence, it fails immediately.

Related issue number

#1557

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[Vman909]

Need more suggestions like to answer this telephone with address