[Provisioner] Robustify the termiantion for provision failure to avoid leakage

[Michaelvll]

This is to fix the following case, inspired by #2975 (review):

1. sky launch -c test
2. The provision for the new VM fails with a VM actually created on the cloud (to simulate add `raise RuntimeError('test provisioning') at

   skypilot/sky/provision/provisioner.py

   Line 162 in eac8592

   return _bulk_provision(cloud, region, zones, cluster_name,

   )
3. The new provisioner tries to termiante the cluster and trigger failover.
4. If the termination fails (e.g., the cloud API fails to terminate a VM that is in a state of being created), our current implementation will ignore the error and go to the next region, causing the leakage of previous VM. (to simulate, add raise RuntimeError('test: failed teardown') at

   skypilot/sky/provision/provisioner.py

   Line 190 in eac8592

   if terminate:

   )

In this PR, we add additional retries for the termination, and if the failure still occurs after retrying for three times, we error out, instead of keeps failover through the following regions. Also, the failed cluster will remain in the cluster table, and the user should be able to check the cluster's status and terminate it manually.

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
  • Test described above.
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

[concretevitamin]

Good catch @Michaelvll! I assume provisioner V1 doesn't have this problem, since https://github.com/skypilot-org/skypilot/blob/master/sky/backends/cloud_vm_ray_backend.py#L1693-L1694 seems to throw?

[concretevitamin]

LGTM

[concretevitamin]

Should we make L173 as e and raise this error from e?

[Michaelvll]

We probably should raise from the teardown exception? I make it from the e raised by the teardown_cluster