[data/llm] Fix PrepareMultimodalStage GPU detection on CPU-only nodes - #64048
[data/llm] Fix PrepareMultimodalStage GPU detection on CPU-only nodes#64048Vedika-Sd wants to merge 2 commits into
Conversation
- Check if GPU is available before creating vLLM ModelConfig - On CPU-only nodes, set gpu_memory_utilization=0.0 to prevent GPU detection crash - Fixes ray-project#64004 Signed-off-by: Vedika Sardeshmukh <vedikasardeshmukh7@gmail.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request aims to prevent GPU detection crashes on CPU-only nodes during the multimodal preparation stage by setting gpu_memory_utilization to 0.0 when no GPU is detected. The reviewer suggested a simplification: since this stage is CPU-only and does not run the actual engine, we can safely default gpu_memory_utilization to 0.0 if not specified, completely avoiding the need to import torch and check for GPU availability.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Vedika Sardeshmukh <vedikasardeshmukh7@gmail.com>
ray-gardener
Bot
added
data
|
Hi @Vedika-Sd , Is the problem that when you instantiate ModelConfig object from vllm on cpu machine it fails? If so, shouldn't the fix live on vllm side? |
|
This pull request has been automatically marked as stale because it has not had You can always ask for help on our discussion forum or Ray's public slack channel. If you'd like to keep this open, just leave any comment, and the stale label will be removed. |
github-actions
Bot
added
the
stale
|
Thanks @kouroshHakha You’re right that the immediate failure is tied to vLLM initialization. In this path, though Ray is the caller during multimodal stage preparation and Ray can hit that path on CPU-only nodes even when GPU execution is not required. I agree there may also be a complementary upstream improvement in vLLM, but Ray should still handle this integration path defensively for heterogeneous clusters. |
github-actions
Bot
added
unstale
| if "gpu_memory_utilization" not in model_config_kwargs: | ||
| model_config_kwargs = { | ||
| **model_config_kwargs, | ||
| "gpu_memory_utilization": 0.0, | ||
| } |
There was a problem hiding this comment.
Agree with @kouroshHakha -- this seems like a band-aid fix to me. Let's fix it more fundamentally in vLLM or seek for a different approach here.
|
Closing this PR as #64015 addresses the same issue. Thanks for taking a stab anyways! |
3 participants
Description
This PR fixes the
PrepareMultimodalStagecrash on CPU-only nodes by checking GPU availability before creating vLLM'sModelConfig.Changes
torch.cuda.is_available()before creating ModelConfiggpu_memory_utilization=0.0to prevent GPU detection crashFixes
Fixes #64004