Update documentation guide for ONNX INT4 PTQ on Windows cuda13 host - #2022
Conversation
Signed-off-by: vipandya <vipandya@nvidia.com>
📝 WalkthroughWalkthroughChangesThe Windows installation guide now documents CUDA 12 and CUDA 13 quantization setup and verification. The GenAI LLM example adds CUDA package guidance and logs the configured CUDA path with torch runtime details. Windows CUDA guidance
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 6✅ Passed checks (6 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2022 +/- ##
==========================================
- Coverage 66.83% 66.64% -0.19%
==========================================
Files 519 519
Lines 58916 58916
==========================================
- Hits 39376 39266 -110
- Misses 19540 19650 +110
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Warning
CodeRabbit couldn't request changes on this pull request because it doesn't have sufficient GitHub permissions.
Please grant CodeRabbit Pull requests: Read and write permission and re-run the review.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/source/getting_started/windows/_installation_standalone.rst`:
- Around line 87-89: Remove the unmatched closing parenthesis at the end of the
sentence describing CUDA environment variables, so it ends with “installation.”
while preserving the rest of the installation guidance.
- Around line 108-112: Update the ONNX Runtime verification command in the ONNX
and ONNX Runtime installation section to explicitly validate that
CUDAExecutionProvider is present in ort.get_available_providers(), causing the
command to fail when CUDA support is unavailable while still reporting the
version and providers.
In `@examples/windows/onnx_ptq/genai_llm/quantize.py`:
- Around line 315-320: Update the CUDA diagnostic variable resolution in the
quantization script to also check the versioned CUDA environment variables
supported by the installation guide, including CUDA_PATH_V12_x and
CUDA_PATH_V13_x. Preserve the existing CUDA_PATH and CUDA_HOME fallback order,
and report the first available configured toolkit path instead of “N/A”.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 43729e0d-dab6-4ef7-97b6-7b6994a091b3
📒 Files selected for processing (3)
docs/source/getting_started/windows/_installation_standalone.rstexamples/windows/onnx_ptq/genai_llm/README.mdexamples/windows/onnx_ptq/genai_llm/quantize.py
| Make sure a cuDNN 9 build for CUDA 13 is available on the host (for example, via the | ||
| NVIDIA Windows installer/zip package, or the ``nvidia-cudnn-cu13`` Python wheel), and | ||
| that the relevant environment variables like ``CUDA_PATH``, ``CUDA_HOME``, and ``PATH`` point to the CUDA 13.x installation). |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Remove the unmatched closing parenthesis.
The sentence ends with installation). without a matching opening parenthesis.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/source/getting_started/windows/_installation_standalone.rst` around
lines 87 - 89, Remove the unmatched closing parenthesis at the end of the
sentence describing CUDA environment variables, so it ends with “installation.”
while preserving the rest of the installation guidance.
| - **ONNX and ONNX Runtime**: Ensure that imports succeed and that CUDA EP is available for CUDA workflows: | ||
|
|
||
| .. code-block:: python | ||
|
|
||
| python -c "import onnx; import onnxruntime as ort; print(ort.__version__, ort.get_available_providers())" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Make the CUDA EP check fail when CUDA is unavailable.
The command only prints providers, so it exits successfully even when CUDAExecutionProvider is missing. Add an assertion or explicit check for CUDA workflows. ONNX Runtime documents CUDAExecutionProvider as the CUDA backend. (onnxruntime.ai)
Proposed fix
- python -c "import onnx; import onnxruntime as ort; print(ort.__version__, ort.get_available_providers())"
+ python -c "import onnx; import onnxruntime as ort; providers=ort.get_available_providers(); assert 'CUDAExecutionProvider' in providers, providers; print(ort.__version__, providers)"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - **ONNX and ONNX Runtime**: Ensure that imports succeed and that CUDA EP is available for CUDA workflows: | |
| .. code-block:: python | |
| python -c "import onnx; import onnxruntime as ort; print(ort.__version__, ort.get_available_providers())" | |
| - **ONNX and ONNX Runtime**: Ensure that imports succeed and that CUDA EP is available for CUDA workflows: | |
| .. code-block:: python | |
| python -c "import onnx; import onnxruntime as ort; providers=ort.get_available_providers(); assert 'CUDAExecutionProvider' in providers, providers; print(ort.__version__, providers)" |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/source/getting_started/windows/_installation_standalone.rst` around
lines 108 - 112, Update the ONNX Runtime verification command in the ONNX and
ONNX Runtime installation section to explicitly validate that
CUDAExecutionProvider is present in ort.get_available_providers(), causing the
command to fail when CUDA support is unavailable while still reporting the
version and providers.
| cuda_path = os.environ.get("CUDA_PATH") or os.environ.get("CUDA_HOME") or "N/A" | ||
| print( | ||
| f"\n--Quantize-Script-- torch={torch.__version__}, " | ||
| f"torch.version.cuda={torch.version.cuda}, " | ||
| f"torch.cuda.is_available={torch.cuda.is_available()}, " | ||
| f"CUDA_PATH={cuda_path}\n" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Resolve the same CUDA variables documented by the installation guide.
The guide permits CUDA_PATH_V12_x/CUDA_PATH_V13_x, but this diagnostic only reads CUDA_PATH and CUDA_HOME. On hosts where only the versioned variable is set, it reports N/A and misrepresents the selected toolkit.
Proposed fix
- cuda_path = os.environ.get("CUDA_PATH") or os.environ.get("CUDA_HOME") or "N/A"
+ cuda_path = (
+ os.environ.get("CUDA_PATH")
+ or os.environ.get("CUDA_HOME")
+ or next(
+ (
+ value
+ for key, value in sorted(os.environ.items())
+ if key.startswith(("CUDA_PATH_V12_", "CUDA_PATH_V13_")) and value
+ ),
+ "N/A",
+ )
+ )📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| cuda_path = os.environ.get("CUDA_PATH") or os.environ.get("CUDA_HOME") or "N/A" | |
| print( | |
| f"\n--Quantize-Script-- torch={torch.__version__}, " | |
| f"torch.version.cuda={torch.version.cuda}, " | |
| f"torch.cuda.is_available={torch.cuda.is_available()}, " | |
| f"CUDA_PATH={cuda_path}\n" | |
| cuda_path = ( | |
| os.environ.get("CUDA_PATH") | |
| or os.environ.get("CUDA_HOME") | |
| or next( | |
| ( | |
| value | |
| for key, value in sorted(os.environ.items()) | |
| if key.startswith(("CUDA_PATH_V12_", "CUDA_PATH_V13_")) and value | |
| ), | |
| "N/A", | |
| ) | |
| ) | |
| print( | |
| f"\n--Quantize-Script-- torch={torch.__version__}, " | |
| f"torch.version.cuda={torch.version.cuda}, " | |
| f"torch.cuda.is_available={torch.cuda.is_available()}, " | |
| f"CUDA_PATH={cuda_path}\n" |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/windows/onnx_ptq/genai_llm/quantize.py` around lines 315 - 320,
Update the CUDA diagnostic variable resolution in the quantization script to
also check the versioned CUDA environment variables supported by the
installation guide, including CUDA_PATH_V12_x and CUDA_PATH_V13_x. Preserve the
existing CUDA_PATH and CUDA_HOME fallback order, and report the first available
configured toolkit path instead of “N/A”.
2 participants
What does this PR do?
Type of change: Documentation update
Testing
Before your PR is "Ready for review"
Make sure you read and follow Contributor guidelines and your commits are signed (
git commit -s -S).Make sure you read and follow the Security Best Practices (e.g. avoiding hardcoded
trust_remote_code=True,torch.load(..., weights_only=False),pickle, etc.).CONTRIBUTING.md: ✅ / ❌ / N/AAdditional Information
Summary by CodeRabbit
Documentation
Enhancements