Skip to content

fix: NCCL group auto-resolve crash on load + Infinity SymInt in extract_var_range_info - #4428

Merged
apbose merged 1 commit into
release/2.13from
abose/torchTRT_export_example_load_fix
Jul 23, 2026
Merged

fix: NCCL group auto-resolve crash on load + Infinity SymInt in extract_var_range_info#4428
apbose merged 1 commit into
release/2.13from
abose/torchTRT_export_example_load_fix

Conversation

@apbose

@apbose apbose commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes # (issue)

Type of change

Please delete options that are not relevant and/or add your own.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Checklist:

  • My code follows the style guidelines of this project (You can use the linters)
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas and hacks
  • I have made corresponding changes to the documentation
  • I have added tests to verify my fix or my feature
  • New and existing unit tests pass locally with my changes
  • I have added the relevant labels to my PR in so that relevant reviewers are notified

@meta-cla meta-cla Bot added the cla signed label Jul 22, 2026
@apbose
apbose force-pushed the abose/torchTRT_export_example_load_fix branch from d3b3f84 to 4c90ef6 Compare July 23, 2026 00:48
@github-actions github-actions Bot added documentation Improvements or additions to documentation component: tests Issues re: Tests component: lowering Issues re: The lowering / preprocessing passes component: conversion Issues re: Conversion stage component: core Issues re: The core compiler component: converters Issues re: Specific op converters component: build system Issues re: Build system component: api [Python] Issues re: Python API component: runtime component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths labels Jul 23, 2026
@apbose
apbose changed the base branch from main to release/2.13 July 23, 2026 00:49
@apbose
apbose force-pushed the abose/torchTRT_export_example_load_fix branch from 4c90ef6 to 09956c0 Compare July 23, 2026 00:52

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some changes that do not conform to C++ style guidelines:

diff --git a/home/runner/work/TensorRT/TensorRT/core/runtime/TRTEngine.cpp b/tmp/changes.txt
index 3b1b52a..4270d84 100644
--- a/home/runner/work/TensorRT/TensorRT/core/runtime/TRTEngine.cpp
+++ b/tmp/changes.txt
@@ -592,7 +592,8 @@ bool TRTEngine::bind_nccl_comm() {
    std::vector<std::string> nccl_groups;
    for (int i = 0; i < 20; ++i) {
      auto pg = try_resolve(std::to_string(i));
-      if (!pg) break;
+      if (!pg)
+        break;
      if (pg->getBackendType() == c10d::ProcessGroup::BackendType::NCCL) {
        nccl_groups.push_back(std::to_string(i));
      }
ERROR: Some files do not conform to style guidelines

@apbose apbose changed the title set the process group in load "fix: NCCL group auto-resolve crash on load + Infinity SymInt in extract_var_range_info" Jul 23, 2026
@apbose
apbose force-pushed the abose/torchTRT_export_example_load_fix branch from 09956c0 to ca1cb51 Compare July 23, 2026 00:57

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some changes that do not conform to C++ style guidelines:

diff --git a/home/runner/work/TensorRT/TensorRT/core/runtime/TRTEngine.cpp b/tmp/changes.txt
index 7216be6..4f44c36 100644
--- a/home/runner/work/TensorRT/TensorRT/core/runtime/TRTEngine.cpp
+++ b/tmp/changes.txt
@@ -592,7 +592,8 @@ bool TRTEngine::bind_nccl_comm() {
    std::vector<std::string> nccl_groups;
    for (int i = 0; i < 20; ++i) {
      auto pg = try_resolve(std::to_string(i));
-      if (!pg) break;
+      if (!pg)
+        break;
      if (pg->getBackendType() == c10d::ProcessGroup::BackendType::NCCL) {
        nccl_groups.push_back(std::to_string(i));
      }
ERROR: Some files do not conform to style guidelines

@apbose apbose changed the title "fix: NCCL group auto-resolve crash on load + Infinity SymInt in extract_var_range_info" fix: NCCL group auto-resolve crash on load + Infinity SymInt in extract_var_range_info Jul 23, 2026
@apbose
apbose force-pushed the abose/torchTRT_export_example_load_fix branch from ca1cb51 to 283f7bd Compare July 23, 2026 01:03
…ct_var_range_info

TRTEngine.cpp — bind_nccl_comm auto-resolve:
  PyTorch >= 2.14 throws c10::Error when resolve_process_group() is called
  with a name that doesn't exist, instead of returning nullptr. The sequential
  probe loop (0..19) would crash at i=1 on a fresh load-only process that only
  has WORLD registered as "0". Replace the bare call with a try_resolve lambda
  that normalises the throw to nullptr so the scan loop stays exception-free.

dynamo/utils.py — extract_var_range_info:
  bound_sympy() returns sympy.oo for unbounded composite exprs (e.g. -s56),
  which is distinct from int_oo. int(sympy.oo) raises AttributeError
  ('Infinity' object has no attribute '_mpf_'). Add _bound_to_int_or_none
  helper that guards both int_oo and sympy.oo cases and catches
  TypeError/OverflowError/AttributeError so unbounded dims fall back to 1.
@apbose
apbose merged commit 68bbf46 into release/2.13 Jul 23, 2026
171 of 176 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed component: api [Python] Issues re: Python API component: build system Issues re: Build system component: conversion Issues re: Conversion stage component: converters Issues re: Specific op converters component: core Issues re: The core compiler component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths component: lowering Issues re: The lowering / preprocessing passes component: runtime component: tests Issues re: Tests documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant