[Serve] Fix orphaned actors on controller crash during shutdown

[vaishdho1]

Description

When the Serve controller crashes mid-shutdown, replica actors are orphaned and their resources are leaked. This happens because KV checkpoints are deleted at the very start of the shutdown process. If the controller crashes and restarts after checkpoint deletion but before actor teardown completes, the restarted controller has no record of the apps/deployments it needs to clean up.
This PR fixes the issue by:

• Persisting a SHUTDOWN_IN_PROGRESS_KEY in the KV store at the start of graceful_shutdown(). On restart, the controller checks this key and automatically re-enters the shutdown path.
• Deferring checkpoint deletion to the very end, after all the resources are released and before the controller kills itself.

Tests

• test_shutdown.py - controller crash mid-shutdown recovers and continues shutdown
• test_application_state.py - app checkpoint survives shutdown lifecycle
• test_deployment_state.py - deployment checkpoint survives shutdown lifecycle

Related issues

Fixes #62729

[gemini-code-assist]

Code Review

This pull request introduces a mechanism to persist the shutdown state of the Ray Serve controller in the KV store, ensuring that if the controller crashes during a graceful shutdown, it resumes the shutdown process upon recovery rather than re-applying the previous configuration. Changes include adding a SHUTDOWN_IN_PROGRESS_KEY, updating the controller's recovery logic, and moving checkpoint deletion to the final stage of the shutdown process. Feedback was provided regarding the recovery logic: when the controller restarts in a shutdown-in-progress state, it should explicitly trigger shutdown on the state managers to ensure all entities are correctly marked for teardown and to prevent the controller from waiting indefinitely.

[abrarsheikh]

does this test fail w/o your PR?

[vaishdho1]

Yes this fails w/o the changes

[abrarsheikh]

gcs_failiure tests are failing, PTAL

[vaishdho1]

While fixing the gcs tests error found something interesting, there was a test where gcs is down and takes a while to start again whengraceful_shutdown() is called, so the write to kv store fails.
So, I wrapped the write in a try/except block inside graceful_shutdown(), with retries in the next clock tick inside shutdown(), if the first write fails.
I have summarized the different crash scenarios during shutdown and their recovery pattern and whether it is supported with the changes or not.

Scenario	Behavior
Normal shutdown, no crash	Flag persisted in graceful_shutdown(), controller completes teardown.
Controller crash mid shutdown	Flag was persisted before crash. Restarted controller reads it, re enters shutdown, managers recover state from their checkpoints, teardown resumes.
GCS down when graceful_shutdown() is called	kv_store.put fails, caught by try/except, _shutting_down is set in memory. Control loop retries persisting via shutdown() on each tick until GCS is back.
GCS down and controller crash simultaneously	Flag may not be persisted. On restart, controller doesn't know shutdown was in progress and resumes normal operation. This is a double-failure scenario: this is not supported.
Crash before first control loop tick after graceful_shutdown	Flag is persisted in graceful_shutdown() itself before returning to caller, so even an immediate ray.kill after the call is safe.

Is this good?

[abrarsheikh]

what if we remove this code block completely? would shutdown would get called on every tick of controller (because self._shutting_down is set to True) and do the right thing right?

[vaishdho1]

Yes. But the only thing is there will be a small window up till the next control loop tick during which a crash cannot be recovered. If that is okay, we can directly set it inside shutdown(). This was done to reduce that window as much as possible.

[abrarsheikh]

you make a good point, it maybe atleast 100ms before the next tick. and by commiting to KV store here are able to provide correct response status to user.

[abrarsheikh]

@akyang-anyscale PTAL