[serve.llm] Set up KV-connector backend in mock engine for PD direct-streaming test#64266
Merged
eicherseiji merged 2 commits intoJun 23, 2026
Conversation
…streaming test test_pd_direct_streaming uses the mock vLLM engine, which skips the real engine init that runs LLMConfig.setup_engine_backend(). So _kv_connector_backend stays None and PDDecodeServer._get_connector_backend() asserts on every request, 500ing the test deterministically. Install a DefaultConnectorBackend in the mock when the config requests KV transfer, mirroring the real engine and the test_prefill_decode_disagg unit-test stub. The default backend avoids the real Nixl/LMCache setup() that needs hardware unavailable in CPU CI. Signed-off-by: Seiji Eicher <seiji@anyscale.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request updates the mock vLLM engine (MockVLLMEngine) to mirror the real engine's backend setup. Specifically, if kv_transfer_config is present in the configuration, it initializes a DefaultConnectorBackend on llm_config. This ensures the P/D orchestrator can find a KV-connector backend during testing without requiring the specialized hardware needed for the real setup. There are no review comments, and I have no additional feedback to provide.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Merged
2 tasks
…mock engine Signed-off-by: Seiji Eicher <seiji@anyscale.com>
eicherseiji
marked this pull request as ready for review
June 23, 2026 02:18
jeffreywang88
approved these changes
Jun 23, 2026
eicherseiji
added a commit
that referenced
this pull request
Jun 24, 2026
…oxy instead of 503 (#64218) ## Why are these changes needed? With HAProxy ingress and direct streaming, the ingress request router pins a replica by name and HAProxy routes to it through its statically reloaded server map. Right after the replica set changes, the router's in-process view runs ahead of HAProxy's config reload, so it can name a replica HAProxy has not loaded yet. HAProxy returned a 503 `unknown_replica_id`. This is an intermittent failure in the direct-streaming session-affinity tests and a real client-facing error during the gap. This routes `unknown_replica_id` to the fallback Serve proxy instead of 503ing. The fallback re-pins through its own router on the same consistent-hash ring, so affinity is preserved. A router app with no fallback still fails loud rather than silently using the primary backend, and other router failures still 503. Reproduced and validated with `test_dp_direct_streaming` against HAProxy 2.8. The deterministic `test_pd_direct_streaming` failure is a separate KV-connector gap fixed in #64266. ## Related issue number Closes #64265. ## Checks - [x] I've signed off every commit (DCO). - [x] Added `test_pin_miss_falls_back_to_fallback_server`. Lint clean. --------- Signed-off-by: Seiji Eicher <seiji@anyscale.com>
limarkdcunha
pushed a commit
to limarkdcunha/ray
that referenced
this pull request
Jun 30, 2026
limarkdcunha
pushed a commit
to limarkdcunha/ray
that referenced
this pull request
Jun 30, 2026
…oxy instead of 503 (ray-project#64218) ## Why are these changes needed? With HAProxy ingress and direct streaming, the ingress request router pins a replica by name and HAProxy routes to it through its statically reloaded server map. Right after the replica set changes, the router's in-process view runs ahead of HAProxy's config reload, so it can name a replica HAProxy has not loaded yet. HAProxy returned a 503 `unknown_replica_id`. This is an intermittent failure in the direct-streaming session-affinity tests and a real client-facing error during the gap. This routes `unknown_replica_id` to the fallback Serve proxy instead of 503ing. The fallback re-pins through its own router on the same consistent-hash ring, so affinity is preserved. A router app with no fallback still fails loud rather than silently using the primary backend, and other router failures still 503. Reproduced and validated with `test_dp_direct_streaming` against HAProxy 2.8. The deterministic `test_pd_direct_streaming` failure is a separate KV-connector gap fixed in ray-project#64266. ## Related issue number Closes ray-project#64265. ## Checks - [x] I've signed off every commit (DCO). - [x] Added `test_pin_miss_falls_back_to_fallback_server`. Lint clean. --------- Signed-off-by: Seiji Eicher <seiji@anyscale.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
test_pd_direct_streamingis broken deterministically on master: every request returns 500.PDDecodeServer._get_connector_backend()asserts a KV-connector backend on itsLLMConfig, and onlyLLMConfig.setup_engine_backend()populates it, which runs only during real vLLM engine init. The end-to-end test uses the mock engine, which skips that init, so_kv_connector_backendstaysNoneand the decode server raises on every request.This makes the mock engine install a default KV-connector backend when the config requests KV transfer, mirroring the real engine's
setup_engine_backend(). The default backend is used because the real Nixl/LMCachesetup()needs hardware the CPU test environment lacks. This matches the workaround the sibling unit testtest_prefill_decode_disagg.pyalready applies by hand.This is a test-infrastructure change only, in the mock engine under
ray/llm/tests.Related issue number
Closes #64264.
After this fix,
test_pd_direct_streamingreaches the request path. The only residual failure is the separate intermittentunknown_replica_idingress-router race, tracked in #64265 and fixed by #64218.Checks
unknown_replica_idfailure is the separate race in [CI]linux://python/ray/llm/tests:serve/cpu/deployments/data_parallel/test_dp_direct_streamingis failing/flaky on master. #64265.