Draft mode autodetection + fixes for speculative decoding - #4490
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR makes several small adjustments around speculative decoding and text-generation deployment configuration, spanning both runtime initialization and user-facing export/graph tooling.
Changes:
- Default the draft model device to the main model device when
draft_deviceis not provided (legacy + continuous batching initializers). - Reduce verbosity in the LLM HTTP calculator by moving per-iteration logs from DEBUG to TRACE.
- Extend graph/export tooling and demos to support auto-detected device behavior and updated OpenAI-compatible endpoint examples.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/llm/language_model/legacy/servable_initializer.cpp | Falls back to main device when draft_device is empty for draft pipeline init. |
| src/llm/language_model/continuous_batching/servable_initializer.cpp | Same draft-device fallback for continuous batching draft pipeline init. |
| src/llm/http_llm_calculator.cc | Downgrades several high-frequency DEBUG logs to TRACE in Process(). |
| src/graph_export/graph_export.cpp | Emits draft_device in generated graph template when provided. |
| src/graph_export/graph_cli_parser.cpp | Parses draft_device into graph settings (but CLI option wiring is incomplete). |
| src/capi_frontend/server_settings.hpp | Adds draftDevice to text-generation graph settings. |
| demos/continuous_batching/speculative_decoding/README.md | Updates Docker/devices guidance and pip install instructions; needs clarification for portability. |
| demos/continuous_batching/accuracy/README.md | Updates example base URLs from /v3 to /v1. |
| demos/common/export_models/README.md | Documents auto-detected device behavior when --target_device is omitted. |
| demos/common/export_models/export_model.py | Makes --target_device optional and omits device/draft_device fields from generated graph when not specified. |
Suppressed comments (1)
demos/continuous_batching/speculative_decoding/README.md:289
- This paragraph suggests OVMS will auto-detect the best available device, but the Docker command above does not expose any accelerator devices to the container (so only CPU will be visible). Clarify that auto-detection is limited to devices visible inside the container and that accelerator pass-through flags are required to actually use GPU/NPU.
OVMS auto-detects the best available device at startup. To target a specific device explicitly, pass `--target_device GPU` (or `NPU`, `HETERO:GPU,CPU`, etc.) to `export_model.py` and follow the [AI accelerators guide](../../../docs/accelerators.md) for additionally required docker parameters.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+158
to
+160
| if (result->count("draft_device")) { | ||
| graphSettings.draftDevice = result->operator[]("draft_device").as<std::string>(); | ||
| } |
Comment on lines
92
to
+96
| ```bash | ||
| docker run -d --rm -p 8000:8000 -v $(pwd)/models:/workspace:ro openvino/model_server:weekly --rest_port 8000 --config_path /workspace/config.json | ||
| docker run -d --rm --device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1) -p 8000:8000 -v $(pwd)/models:/workspace:ro openvino/model_server:weekly --rest_port 8000 --config_path /workspace/config.json | ||
| ``` | ||
|
|
||
| Running above command starts the container with no accelerators support. | ||
| To deploy on devices other than CPU, change `target_device` parameter in `export_model.py` call and follow [AI accelerators guide](../../../docs/accelerators.md) for additionally required docker parameters. | ||
| OVMS auto-detects the best available device at startup. To target a specific device explicitly, pass `--target_device GPU` (or `NPU`, `HETERO:GPU,CPU`, etc.) to `export_model.py` and follow the [AI accelerators guide](../../../docs/accelerators.md) for additionally required docker parameters. |
mzegla
force-pushed
the
eagle3_continuation
branch
from
August 31, 2026 09:24
fcf945a to
99472d4
Compare
mzegla
force-pushed
the
eagle3_continuation
branch
from
September 1, 2026 12:53
d2276c7 to
7497d63
Compare
mzegla
changed the title
dtrawins
approved these changes
Sep 1, 2026
Comment on lines
+72
to
+76
| // Detects draft model strategy from model artifacts without a full model load. | ||
| // Reads last 32KB of the XML (rt_info is at the end of OV IR format) for eagle3/dflash markers. | ||
| // DFlash takes priority over EAGLE3 when both markers are present (matches GenAI's strategy selection). | ||
| // Throws std::runtime_error if the XML cannot be opened. | ||
| GenAiServableProperties::DraftModelStrategy detectDraftModelStrategy(const std::string& draftPath); |
Comment on lines
3158
to
3160
| TEST_F(OvmsInferredTaskTest, positiveConfigureModeInfersTaskFromModel) { | ||
| const std::string modelPath = resolveTestModelPath("llama"); | ||
| const std::filesystem::path configJson = std::filesystem::path(modelPath) / "config.json"; |
Comment on lines
+382
to
+388
| docker run -d --rm $(test -d /dev/dri && echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)") \ | ||
| -p 8000:8000 -v ${HOME}/models:/models:rw openvino/model_server:weekly \ | ||
| --rest_port 8000 \ | ||
| --model_repository_path /models \ | ||
| --source_model OpenVINO/Qwen3.8-27B-int8-ov \ | ||
| --draft_model_path . | ||
| --enable_prefix_caching false |
Comment on lines
+394
to
+399
| ovms --rest_port 8000 \ | ||
| --model_repository_path c:\models \ | ||
| --source_model OpenVINO/Qwen3.8-27B-int8-ov \ | ||
| --draft_model_path . | ||
| --enable_prefix_caching false | ||
| ``` |
michalkulakowski
approved these changes
Sep 1, 2026
mzegla
commented
Sep 1, 2026
michalkulakowski
pushed a commit
that referenced
this pull request
Sep 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.