[RFC] Clarifying vLLM Shutdown Semantics

[markmc]

🚀 The feature, motivation and pitch

RFC: Clarifying vLLM Shutdown Semantics

Introduction

This RFC defines the expected behavior of vLLM during shutdown, addressing two distinct use cases:

1. Library API usage (primarily offline inference).
2. Online serving usage (long-running HTTP server deployments).

Documenting these expectations will reduce surprises for users and help us achieve a consistent implementation across all features. This RFC was motivated by #22295 and #22828.

---

Library API Shutdown

Library users expect deterministic, resource-safe cleanup when disposing of LLM instances.

Expectations

• Library users may create and destroy vllm.LLM instances.
• Deleting an instance should:
  • Release all associated resources (memory, file descriptors, temporary files, etc.).
  • Shut down any child processes gracefully.
• No library should define global signal handlers.
  • Signal handlers are global and may interfere with applications embedding vLLM.
  • Applications must configure their own signal-handling behavior.

---

Online Serving Shutdown

Online serving users expect graceful termination semantics, consistent with production environments like Kubernetes, where shutdown is routine due to auto-scaling or rolling upgrades.

Note - vLLM does not support any partial restart scenario, for example a "reload model" or "restart workers" capability.

Context

• vllm serve launches an HTTP server for long-running online inference.
• Each request is relatively expensive (100ms+).
• Deployments typically involve multiple replicas behind a load balancer (e.g. Kubernetes Service).

Expectations

• Signal Handling

  • Parent process receives a SIGTERM on shutdown.
  • SIGINT (e.g. Ctrl-C) is treated the same as SIGTERM.
  • Signals are sent only to the parent; child processes must not act on signals directly.

• Graceful Quiescence

  • Stop accepting new HTTP requests immediately.
  • Allow in-flight requests to complete, bounded by a configurable max wait time.
  • Afterward, instruct child processes to exit.

• Child Processes

  • Child processes must not make independent shutdown decisions affecting in-flight requests; the parent process determines whether requests are aborted or completed
  • Child processes may delay shutdown for other reasons (e.g. KV transfer).
  • Each process is responsible for reaping its own children via waitpid().

• Resource Cleanup

  • Close network sockets, release locks, delete temporary files, etc.

---

Kubernetes Integration

Kubernetes shutdown semantics should align naturally with vLLM’s shutdown design.

• When a Pod enters Terminating:
  • Kubernetes removes its endpoints from the Service LB → no new traffic.
  • The container runtime sends SIGTERM to the container’s parent process.
  • After terminationGracePeriodSeconds (default 30s), SIGKILL is sent if still running.
  • Optionally, a preStop hook can be used to delay SIGTERM (e.g. sleep 5s) to allow straggling requests before quiescence.

---

KV Transfer Considerations

In disaggregated deployments (e.g. NIXL prefill/decode):

• Prefill workers may need to delay shutdown until KV transfer is complete.
• The KVConnector.shutdown() method should allow the connector to defer worker process (and parents) shutdown until all pending KV transfers complete or time out.

---

Open Questions

• Online serving API: consider how shutdown guarantees are reflected in the library API for online serving.
• Ray deployments: do Ray users have specific shutdown expectations?
• Observability: should the "I'm shutting down" status of the API server be observable, e.g. via the /health API?

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[pandalee99]

Hi @markmc , I think this feature is quite necessary. Many bugs can cause "vLLM" to be in a "pseudo - dead" state. Even if the health check passes, vLLM actually can't handle the request.

I suggest this feature can be upgraded step by step:

For example, use --enable-unsafe-api to enable the: shutdown interface.

The shutdown interface will use: os._exit(0) to end the process.

And users can use a POST request to force the instance to exit.
Then this will trigger a health check failure, causing k8s to reschedule.

What do you think?

[markmc]

Hi @markmc , I think this feature is quite necessary. Many bugs can cause "vLLM" to be in a "pseudo - dead" state. Even if the health check passes, vLLM actually can't handle the request.

Take a look at #24491 - we should be able to provide an endpoint to use as a Kubernetes liveness probe

[pandalee99]

Hi @markmc , I think this feature is quite necessary. Many bugs can cause "vLLM" to be in a "pseudo - dead" state. Even if the health check passes, vLLM actually can't handle the request.

Take a look at #24491 - we should be able to provide an endpoint to use as a Kubernetes liveness probe

Thanks a lot, but this PR hasn't been merged either.

I've also checked this PR. Adding "generate optional in health api" might not be an elegant approach.

This PR also mentioned:
"If there are too many requests in the queue, the /health request will time out.
If the /health request is executed every 3 seconds, it may affect actual user requests."

If we want to "Clarify vLLM Shutdown Semantics", adding an API seems to be a more straightforward option.

Do you think so?

best

[markmc]

And users can use a POST request to force the instance to exit. Then this will trigger a health check failure, causing k8s to reschedule.

When running vLLM on Kubernetes, users should not need to intervene if vLLM is in a "pseudo-dead state" - remediation should be automated

Even now, users can force an instance to exit by deleting the pod through the Kubernetes API - we don't need a vLLM API to force an exit