fix(sglang): rebuild prefill/decode requests on every Proxy call

[Sanchit2662]

What type of PR is this?

/kind bug

What this PR does / why we need it:

The SGLang connector was caching the prefill and decode *http.Request objects on the struct between Proxy calls, keyed by lastPrefillAddr / lastDecodeAddr. These requests wrap a bytes.Buffer body in io.NopCloser ; once the HTTP transport reads it, it's gone. Can't reuse it.

The router's retry loop in proxyToPDDisaggregated calls connector.Proxy(...) multiple times on the same connector instance. In a typical PD setup with 1 prefill pool shared across multiple decode pods, the scheduler scores all decode pods against the same prefill pod set and picks the same best prefill pod for each iteration. So you end up with PrefillPods = [P0, P0] and DecodePods = [D0, D1]. When D0 fails and the router retries with D1, the prefill addr is still P0 , cache hit , and the drained
request gets reused. Transport sends Content-Length N with 0 bytes. Prefill server rejects it, or the SGLang bootstrap handshake aborts with KVTransferError: Aborted by AbortReq. Worse, the decode side already wrote headers to the response stream, so the client can get HTTP 200 with an empty/truncated SSE response ; totally silent failure.

The fix is to rebuild both requests on every Proxy call and drop the dead caching fields from the struct. Same approach used in 067dfb5 for NIXL and d9f656a for HTTP.

Special notes for your reviewer:

This is the same body-reuse regression class that was just fixed for NIXL and HTTP connectors , the SGLang connector was added after those fixes landed and reintroduced the same pattern. The most common topology that triggers it is a single prefill pool with 2+ decode pods, which is pretty standard for SGLang PD deployments.

Added two regression tests in sglang_test.go that mirror the NIXL ones , one that calls Proxy twice with the same prefill addr on the same connector instance and checks both backends got a non-empty body each time, and one that checks the caller's reqBody map isn't mutated across calls.

Does this PR introduce a user-facing change?:

Fixed a bug in the SGLang PD connector where prefill/decode request bodies were cached and reused across retry attempts. In a typical PD setup with a shared prefill pool,
retries after a decode pod failure would silently send an empty request body to the prefill backend, causing the entire request to fail or return a truncated response to the
 client.

[gemini-code-assist]

Code Review

This pull request refactors the SGLangConnector to rebuild prefill and decode requests on every Proxy call instead of caching them. This change ensures that request bodies, which are one-shot buffers, are not drained during retries. Additionally, unit tests were added to verify that request bodies are not reused and that the input request map is not mutated. Feedback suggests improving retry isolation by generating unique room IDs per call, handling errors in test handlers, and strengthening the mutation test with deep comparisons.

[gemini-code-assist]

The bootstrapRoom ID is currently stored in the SGLangConnector struct and initialized once when the connector is created. Since the connector instance is reused across retries for the same request (as noted in the PR description), the same room ID is reused for every attempt. If a retry occurs against the same prefill pod, this could lead to collisions or stale state in the prefill pod's bootstrap server if it hasn't yet cleaned up the previous attempt. For better isolation between retries, consider generating a unique bootstrapRoom ID inside the Proxy method for each call, which would also make the connector struct stateless.

[gemini-code-assist]

The error from io.ReadAll(r.Body) is ignored here and in the decode server handler. It is better to check this error and fail the test explicitly (e.g., using t.Errorf) if the body cannot be read, ensuring the test doesn't proceed with an empty body and potentially mask a failure in request construction.

[gemini-code-assist]

This mutation test is currently insufficient as it only checks if new keys were added to the reqBody map. It does not verify if existing values were modified (e.g., max_tokens being capped by preparePrefillBody) or if keys were removed. To properly ensure the caller's map is not mutated across retries, the test should perform a deep comparison of the map before and after the Proxy call.

[Copilot]

Pull request overview

Fixes a retry-path bug in the SGLang PD connector where *http.Request objects (with one-shot bodies) were cached on the connector instance and incorrectly reused across Proxy() calls, causing drained/empty request bodies during router retries.

Changes:

• Remove prefill/decode request caching fields from SGLangConnector and rebuild both upstream requests on every Proxy() call.
• Add regression tests to ensure request bodies are non-empty across repeated Proxy() calls and that caller request maps are not mutated.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
pkg/kthena-router/connectors/sglang.go	Rebuild prefill/decode http.Requests each Proxy() call to avoid drained body reuse during retries.
pkg/kthena-router/connectors/sglang_test.go	Adds regression coverage for body-drain retry behavior and reqBody immutability expectations.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[hzxuzhonghu]

/lgtm

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hzxuzhonghu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• pkg/OWNERS [hzxuzhonghu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment