Skip to content

[Fix] Fall back to Gloo when NCCL backend is unavailable #19641

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 17, 2025

Conversation

conroy-cheers
Copy link
Contributor

@conroy-cheers conroy-cheers commented Jun 14, 2025

Purpose

vLLM refuses to launch with CUDA device on systems without NCCL support (i.e. Jetson):

RuntimeError: Distributed package doesn't have NCCL built in

Previously I have been running vLLM on Jetson by doing a find-replace for nccl -> gloo in vllm/distributed/parallel_state.py, but submitting a proper patch seems more appropriate.

When Torch's NCCL distributed backend isn't available, this will simply print a warning and fall back to using Gloo.

Test Plan

  1. Ensure installed Torch doesn't have NCCL support
  2. Run vLLM
  3. Observe warning: Distributed backend nccl is not available; falling back to gloo.
  4. vLLM works as expected

Test Result

OK

Copy link

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of Changes

Hello @conroy-cheers, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses an issue where vLLM fails to start on systems where the default distributed backend (NCCL) is not available in the installed PyTorch distribution. It introduces a fallback mechanism to use the Gloo backend instead, ensuring broader compatibility and providing a warning message when the fallback occurs.

Highlights

  • Backend Availability Check: I've added a check in the init_distributed_environment function to verify if the specified distributed backend (like NCCL) is available using torch.distributed.is_backend_available.
  • Gloo Fallback: If the requested backend is not available, the code now automatically falls back to using the "gloo" backend for torch.distributed.init_process_group.
  • Warning Log: A warning message is logged to inform the user when the requested backend is unavailable and the system is falling back to Gloo.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in issue comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configureGemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This PR introduces a helpful fallback mechanism to the 'gloo' backend when the specified distributed backend (typically 'nccl') is unavailable. This is particularly useful for environments like Jetson that may lack NCCL support. The core change is simple and effective.

I've added one comment to refine the logging in an edge case to enhance clarity and debuggability. Otherwise, the change looks good and directly addresses the stated problem.

Comment on lines 941 to 947
if not torch.distributed.is_backend_available(backend):
logger.warning("Distributed backend '%s' is not available; falling back to gloo.")
backend = "gloo"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The current implementation for falling back to the 'gloo' backend is a good step. However, the warning message can be a bit confusing if the backend variable is already 'gloo' and 'gloo' itself is unavailable. In that scenario, the log would state "Distributed backend 'gloo' is not available; falling back to gloo.", which is redundant and might mislead someone debugging an issue where 'gloo' is the root problem.

To improve clarity, I suggest refining the conditional logic for the warning and the backend assignment. This ensures the log message accurately reflects the situation, especially when 'gloo' is the initially specified (and unavailable) backend. Additionally, using an f-string for the log message would align with the prevalent style in this file.

The proposed change below:

  1. Stores the original backend name for accurate logging.
  2. If the original backend was 'gloo' and it's unavailable, it logs that 'gloo' is unavailable without mentioning a fallback (as backend remains 'gloo').
  3. If the original backend was something else (e.g., 'nccl') and it's unavailable, it logs the fallback to 'gloo' and updates the backend variable.
  4. Uses f-strings for log messages for consistency.
Suggested change
if not torch.distributed.is_backend_available(backend):
logger.warning("Distributed backend '%s' is not available; falling back to gloo.")
backend = "gloo"
_original_backend_name = backend
if not torch.distributed.is_backend_available(_original_backend_name):
if _original_backend_name == "gloo":
# If 'gloo' was the requested backend and it's not available,
# log this fact. The subsequent init_process_group call will
# attempt to use 'gloo' and fail, which is the expected behavior.
logger.warning(
f"Requested distributed backend '{_original_backend_name}' is not available."
)
# `backend` variable remains 'gloo' as it was the initial request.
else:
# The originally requested backend (e.g., 'nccl') is not available.
# Log the situation and fall back to 'gloo'.
logger.warning(
f"Distributed backend '{_original_backend_name}' is not available. "
f"Falling back to 'gloo' backend."
)
backend = "gloo"
# If 'gloo' itself is also unavailable, init_process_group will handle the error.

@conroy-cheers conroy-cheers force-pushed the fix-when-missing-nccl branch 3 times, most recently from a10f943 to e8a421b Compare June 14, 2025 08:59
logger.warning(
"Distributed backend %s is not available; "
"falling back to gloo.", backend)
backend = "gloo"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better to check "gloo" is available in torch.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gloo is kind of the most basic distributed backend we have, right? We can check if Gloo is available before trying to fall back to it, but if it's also not available, all we can do is throw an error and exit, which is exactly what will already happen if we just try to use gloo anyway.

Happy to put in a check if you think it's worth it for the better error message, though.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, just assert torch.distributed.is_gloo_available().

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Copy link
Collaborator

@houseroad houseroad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, and one comment to address.

Signed-off-by: conroy-cheers <conroy@corncheese.org>
@conroy-cheers conroy-cheers force-pushed the fix-when-missing-nccl branch from e8a421b to aa131a9 Compare June 16, 2025 00:29
@houseroad houseroad added the ready ONLY add when PR is ready to merge/full CI is needed label Jun 16, 2025
@houseroad houseroad merged commit 0860087 into vllm-project:main Jun 17, 2025
78 checks passed
Copy link
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

yeqcharlotte pushed a commit to yeqcharlotte/vllm that referenced this pull request Jun 22, 2025
minpeter pushed a commit to minpeter/vllm that referenced this pull request Jun 24, 2025
…t#19641)

Signed-off-by: conroy-cheers <conroy@corncheese.org>
Signed-off-by: minpeter <kali2005611@gmail.com>
yangw-dev pushed a commit to yangw-dev/vllm that referenced this pull request Jun 24, 2025
…t#19641)

Signed-off-by: conroy-cheers <conroy@corncheese.org>
Signed-off-by: Yang Wang <elainewy@meta.com>
xjpang pushed a commit to xjpang/vllm that referenced this pull request Jun 30, 2025
…t#19641)

Signed-off-by: conroy-cheers <conroy@corncheese.org>
wseaton pushed a commit to wseaton/vllm that referenced this pull request Jun 30, 2025
…t#19641)

Signed-off-by: conroy-cheers <conroy@corncheese.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants