[Core] Report CUDA versions when NVRTC compilation fails

[timmoon10]

Description

NVRTC compilation involves three CUDA versions:

• Compile-time CUDA: used to compile Transformer Engine
• CUDA Runtime: linked during runtime and visible to libnvrtc.so
• Run-time CUDA headers: included in run-time NVRTC compilations

If the user's system is misconfigured, these CUDA versions may be inconsistent and cause strange errors (e.g. #1018). This PR reports each of the CUDA versions to help with debugging.

Closes #1018.

Type of change

• Documentation change (change only to the documentation, either a fix or a new content)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Infra/Build change
• Code refactoring

Changes

• Report CUDA versions whe NVRTC compilation fails

Checklist:

• I have read and followed the contributing guidelines
• The functionality is complete
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[greptile-apps]

Greptile Summary

This PR improves NVRTC compilation failure diagnostics by reporting three CUDA versions — compile-time (CUDA_VERSION), run-time NVRTC (nvrtcVersion), and run-time CUDA headers (include_directory_version) — when a compilation error occurs, along with a mismatch warning when the NVRTC and header versions differ.

• Adds include_directory_version() to cuda_runtime.cpp/h, which parses CUDART_VERSION from cuda_runtime_api.h at the configured include path.
• In rtc.cpp, the NVRTC failure handler now prepends version info and an optional mismatch warning to the diagnostic log before surfacing the compile error.
• Version integers are decoded via a local version_string lambda that returns \"<not found>\" for -1 (undetected) and \"major.minor\" otherwise.

Confidence Score: 5/5

Safe to merge — the new code only runs on the NVRTC compilation failure path, so it cannot affect the happy path or introduce regressions.

All changes are confined to the diagnostic/error-reporting path triggered only when NVRTC compilation already fails. The version encoding arithmetic (major1000 + minor10) is consistent with the standard CUDA convention and with CUDA_VERSION. Error cases (missing headers, unparseable version, failed nvrtcVersion call) are handled gracefully with -1 / not-found rather than crashing. No new memory safety issues, no changes to the happy-path kernel compilation flow.

No files require special attention.

Important Files Changed

Filename	Overview
transformer_engine/common/util/cuda_runtime.cpp	Adds include_directory_version() that reads cuda_runtime_api.h and parses CUDART_VERSION; correct error handling and edge cases for missing/unparseable headers.
transformer_engine/common/util/cuda_runtime.h	Declaration of include_directory_version() with accurate doc-comment; no issues.
transformer_engine/common/util/rtc.cpp	NVRTC failure handler extended to prepend version diagnostics and a conditional mismatch warning; version encoding arithmetic matches the standard CUDA convention.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[nvrtcCompileProgram fails] --> B[Collect CUDA versions]
    B --> C1["build_version = CUDA_VERSION (compile-time)"]
    B --> C2["nvrtcVersion() → nvrtc_version"]
    B --> C3["include_directory_version() → header_version"]
    C3 --> D{include_dir empty?}
    D -- yes --> E[return -1]
    D -- no --> F[Open cuda_runtime_api.h]
    F --> G{Parse CUDART_VERSION?}
    G -- success --> H[return version int]
    G -- failure --> I[return -1]
    C1 & C2 & C3 --> J[Log all three versions via version_string]
    J --> K{nvrtc_version != header_version?}
    K -- yes --> L[Append mismatch warning with include_directory path]
    K -- no --> M[Skip warning]
    L & M --> N[Append NVRTC compile log]
    N --> O[Print to stderr and throw]

Loading

_{Reviews (4): Last reviewed commit: "Merge branch 'main' into tmoon/nvrtc-ver..." | Re-trigger Greptile}

[timmoon10]

I don't really like how we parse the header as a text file. However, when I tried compiling a test program with NVRTC it would override the header's CUDART_VERSION macro with the CUDA Runtime version.

[timmoon10]

/te-ci

[timmoon10]

/te-ci

[ptrendx]

/te-ci