[Feature] Add abort request endpoint to handle request cancellations

[jianzs]

Purpose

In scenarios with disaggregated prefill, if an error happens during decoding or the user manually cancels the process before the KV cache is retrieved, the KV cache stored on the prefill node can't be released. This occurs because the HTTP stream between the prefill node and the user has already closed. To solve this issue, this pull request adds an active abort interface that proactively frees KV cache for delay_free requests.

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request introduces a new /abort_request endpoint to allow for the cancellation of requests and the release of their associated resources, particularly for delay_free requests in disaggregated prefill scenarios. The changes are well-implemented across the API server, protocol, scheduler, and output processor. However, I've identified a critical security vulnerability: the new endpoint is not authenticated. My review includes a comment with a suggested fix for this issue.

[gemini-code-assist]

The new /abort_request endpoint is not protected by authentication. The AuthenticationMiddleware in this file only protects routes that start with /v1. This could allow unauthorized users to abort arbitrary requests if they can guess or obtain a request_id.

To fix this, the endpoint should be moved under the /v1 path.

Suggested change

	@router.post("/abort_request", dependencies=[Depends(validate_json_request)])
	@router.post("/v1/abort_request", dependencies=[Depends(validate_json_request)])

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

[chatgpt-codex-connector]

Handle finished requests before early return

The new block that frees KV blocks for already finished requests can never run because the loop above still treats request.is_finished() as an invalid request and immediately continues. When the abort endpoint is invoked for a request that has already finished, this method skips the request entirely and no cleanup is performed, leaving the KV cache allocated. Move the finished-request logic ahead of the early return or drop the request.is_finished() check in the guard so the cleanup path can execute.

Useful? React with 👍 / 👎.

[markmc]

What component would call this endpoint? How can we authenticate these requests? Who should have permission to abort a given request?

In this thread we decided to rely on (at least in the NIXL case) the P worker timing out stranded KV blocks, even in the case of the client disconnecting.

/cc @njhill @NickLucche

[jianzs]

What component would call this endpoint? How can we authenticate these requests? Who should have permission to abort a given request?

In this thread we decided to rely on (at least in the NIXL case) the P worker timing out stranded KV blocks, even in the case of the client disconnecting.

/cc @njhill @NickLucche

The endpoint should be called by the proxy component that handles the prefill and decode request forwarding. When this proxy disconnects from the user, it can actively call this endpoint on the prefill node to ensure timely release of the KV cache associated with that request. While the timeout mechanism can achieve similar functionality, it may introduce unnecessary delays. Reducing the timeout duration could potentially lead to functional issues. Therefore, I believe the timeout mechanism is better suited as a fallback solution rather than the primary method for actively releasing KV cache.

Regarding authentication, the current vLLM implementation doesn't actually implement endpoint authorization. While the /v1 series of endpoints can be restricted using --api-key or the VLLM_API_KEY environment variable, this is rarely configured in practice. If authentication is a concern, the abort_request endpoint could also be secured using the same configuration.

[njhill]

@jianzs there are various downsides to having an out-of-band abort endpoint. It breaks the kv connector abstraction a bit since in this situation it's the connector which "owns" the blocks and it should therefore be the one to release them.

In theory, the window that we have to rely on the timeout fallback for should be very small:

1. If the proxy cancels the prefill request before it returns, then it should not reserve blocks.
2. If the proxy cancels the in-flight decode request it should still notify the prefill worker to release the blocks, even if the request is in the waiting queue.

So the timeout should only end up being used if the cancellation happens between the proxy receiving the prefill response and sending the decode request (in practice this should be immediate), or if the decode worker dies before it's generated the first token for the request. I think these cases should be rare enough to leave to the fallback.

However @markmc found that (1.) above is actually not working as intended, so if you observe that these timeout aborts are happening more often than expected (and affecting kvcache usage), that's probably the reason. It should be fairly straightforward to fix this though, see #26400.