[Feature]: Add /health/ready endpoint for GPU health verification

[anencore94]

🚀 The feature, motivation and pitch

Problem: vLLM's current /health endpoint only checks whether the engine process is alive (engine_dead boolean). When a GPU encounters a page fault or illegal memory access, the process remains alive but inference becomes impossible. The /health endpoint continues returning 200, causing Kubernetes to route traffic to a broken pod.

I hit this exact scenario in production — GPU page fault / illegal memory access made inference fail, but the vLLM process itself was still running. K8s liveness probe passed, so traffic kept flowing to a pod that couldn't actually serve requests.

Solution: Add a /health/ready endpoint that runs an actual GPU forward pass (1-token dummy batch) to verify the GPU can execute inference end-to-end.

This aligns with K8s liveness/readiness probe separation:

• /health → liveness (existing, process-level boolean check)
• /health/ready → readiness (new, GPU forward pass verification)

Key design decisions:

• Reuses existing execute_dummy_batch_async() which calls _dummy_run(1, uniform_decode=True) — no new GPU code needed
• Uses the call_utility_async() path (not the scheduler's request queue), so it doesn't compete with normal inference requests for scheduling
• Configurable timeout via VLLM_HEALTH_CHECK_GPU_TIMEOUT env var (default 20s)
• No breaking changes — existing /health is untouched

Prior art: SGLang has a similar /health_generate endpoint (issues sgl-project/sglang#853, PRs sgl-project/sglang#1154, sgl-project/sglang#8444, sgl-project/sglang#13320). Their implementation goes through the scheduler's generate pipeline which causes contention under load. vLLM's approach via call_utility_async() avoids this issue by using a separate utility call path.

Estimated scope: ~6 files, ~96 lines added, no breaking changes.

Alternatives

1. Periodic background health checks with cached results — Rejected because cached results represent stale health state, and K8s already controls check intervals via probe configuration.
2. Full generate request (like SGLang's initial approach) — Goes through the scheduler queue, competes with real requests, and can cause false timeouts under load. SGLang had to add multiple workarounds for this (PRs [Improvements] Merge health check route sgl-project/sglang#8444, [RL] fix skip_server_warmup and rl health_generate logic sgl-project/sglang#8757, [RL] Allow bypassing /health check sgl-project/sglang#13320).
3. CUDA error detection via signal handlers — More invasive, doesn't catch all failure modes (e.g., driver-level hangs).

Additional context

• K8s probe configuration example:

  livenessProbe:
    httpGet:
      path: /health
      port: 8000
  readinessProbe:
    httpGet:
      path: /health/ready
      port: 8000
    periodSeconds: 30
    timeoutSeconds: 25

• SGLang references: [Feature] The real health check API sgl-project/sglang#853, support /v1/health using a generation 1 token sgl-project/sglang#1154, [Improvements] Merge health check route sgl-project/sglang#8444, [RL] Allow bypassing /health check sgl-project/sglang#13320
• I have a draft PR ready for this feature.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[gaby]

@anencore94 it should be /ready not /health/ready.

[hexfusion]

this makes sense that we have a /ready endpoint, @anencore94 were you planning on working on this? if not I would be interested.

[markmc]

#24491 is a previous PR that tried to tackle this

#32420 also proposed making a distinction between readiness and liveness:

Readiness vs Liveness: The /health endpoint returns 503 during drain (good for readiness probes to remove the pod from service), while /live remains accessible (good for liveness probes to avoid restarts during drain).

but in that case, the goal was to ensure that we have an readiness endpoint to signal that we should be removed from the load-balancer (i.e. we would indicate 'not ready' during shutdown), and a liveness endpoint to indicate that we're still alive and shouldn't be force-terminated (i.e. we would indicate 'still alive' during shutdown)

[markmc]

From @robertgshaw2-redhat on Slack:

We should have EngineCore being responsible for health. EngineCore can check:

1. was there any request in the last 30s
2. if so, was there any token generated in the last 10s. If no, set unhealthy

[anencore94]

Thanks @markmc for surfacing the EngineCore-side proposal from @robertgshaw2-redhat. Before I push the draft PR, I'd like to align on direction since the two approaches solve overlapping but distinct failure modes.

Passive (token-rate stall detection inside EngineCore)

• Pros: zero per-probe overhead, reflects real workload, catches scheduler/worker hangs a dummy batch wouldn't necessarily expose.
• Cons: produces no signal during idle periods. A pod that takes a GPU page fault / illegal memory access while traffic is momentarily zero stays "healthy" indefinitely. This is exactly the production case I hit — K8s readiness probes are expected to keep verifying GPU liveness regardless of incoming traffic.

Active (dummy forward pass on probe request, via call_utility_async())

• Pros: deterministic, works under idle, exercises the full execute path end-to-end.
• Cons: extra GPU work per probe (mitigated by the utility path + sane periodSeconds); a healthy-but-saturated engine could time out the probe under heavy load.

In practice I think we likely want both, composed:

1. EngineCore-internal request/token-rate watchdog flips an unhealthy flag (cheap, covers active-traffic stalls).
2. /health/ready consults that flag, and on idle (no recent traffic) runs a guarded dummy batch (covers GPU death during a traffic lull).

A pure passive design leaves the original production failure mode undetected, so I don't think it fully replaces the active probe — but it would be a strict improvement over the current engine_dead boolean and the two compose cleanly.

Happy to defer to your preference:

• (a) land my current draft as the active-probe piece (naming /ready per @gaby), and treat the passive watchdog as a separate follow-up.
• (b) extend the draft to combine both signals in one PR.
• (c) pause this and let the passive watchdog land first as a separate piece.

@hexfusion thanks for the offer — happy to collaborate once we agree on direction.