fix(recipes): sync INT4 blockwise recipe defaults#1836
Conversation
Signed-off-by: realAsma <akuriparambi@nvidia.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe INT4 blockwise weight-only PTQ recipe now assembles its quantization config from imported fragments instead of an inline disable list. Loader tests add the recipe, allow mappings without a kv config, and conditionally build the expected quant config. ChangesINT4 blockwise PTQ recipe
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 6✅ Passed checks (6 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1836 +/- ##
=======================================
Coverage 77.29% 77.29%
=======================================
Files 513 513
Lines 56920 56920
=======================================
Hits 43997 43997
Misses 12923 12923
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
Moved this diff into draft PR #1838 per owner request; closing this separate draft PR. |
|
## Summary - Skip pre-quant LayerNorm fusion when the representative input quantizer has no `_pre_quant_scale` buffer. - Add CPU unit coverage for the no-scale weight-only path and the existing fusion/removal behavior when pre-quant scales are present. - Compose the public `general/ptq/int4_blockwise_weight_only` recipe from the shared recipe units so it stays aligned with `INT4_BLOCKWISE_WEIGHT_ONLY_CFG`. ## Context NVBug: https://nvbugspro.nvidia.com/bug/6311597 NVBug 6311597 reports a deterministic HF export crash for `general/ptq/int4_blockwise_weight_only` on Llama-3.1-8B. That weight-only recipe disables input quantization and skips calibration, so `_pre_quant_scale` is not registered, but export still reaches `fuse_prequant_layernorm` through the AWQ-like format path. This PR also carries the recipe-sync diff that was previously in draft PR #1836; PR #1836 has been closed after moving that diff here. ## Tests - `pre-commit run --files modelopt/torch/export/quant_utils.py tests/unit/torch/export/test_unified_export_hf.py modelopt_recipes/general/ptq/int4_blockwise_weight_only.yaml tests/unit/recipe/test_loader.py` - `pytest_pwd tests/unit/torch/export/test_unified_export_hf.py::test_fuse_prequant_layernorm_skips_modules_without_pre_quant_scale tests/unit/recipe/test_loader.py::test_load_recipe_all_builtins tests/unit/recipe/test_loader.py::test_general_ptq_yaml_matches_config_dicts tests/unit/recipe/test_presets.py -q` -> 29 passed Full GB10/Llama repro was not run locally. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved handling of layer norm fusion so it safely skips cases where pre-quantization scale data is unavailable, avoiding unexpected errors. * Updated the layer norm fusion flow to correctly apply scaling when pre-quantization data is present. * **Tests** * Expanded coverage for export and recipe loading scenarios, including cases with and without pre-quantization scale data. * Added validation for the new int4 blockwise weight-only recipe. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: realAsma <akuriparambi@nvidia.com>
- Skip pre-quant LayerNorm fusion when the representative input quantizer has no `_pre_quant_scale` buffer. - Add CPU unit coverage for the no-scale weight-only path and the existing fusion/removal behavior when pre-quant scales are present. - Compose the public `general/ptq/int4_blockwise_weight_only` recipe from the shared recipe units so it stays aligned with `INT4_BLOCKWISE_WEIGHT_ONLY_CFG`. NVBug: https://nvbugspro.nvidia.com/bug/6311597 NVBug 6311597 reports a deterministic HF export crash for `general/ptq/int4_blockwise_weight_only` on Llama-3.1-8B. That weight-only recipe disables input quantization and skips calibration, so `_pre_quant_scale` is not registered, but export still reaches `fuse_prequant_layernorm` through the AWQ-like format path. This PR also carries the recipe-sync diff that was previously in draft PR #1836; PR #1836 has been closed after moving that diff here. - `pre-commit run --files modelopt/torch/export/quant_utils.py tests/unit/torch/export/test_unified_export_hf.py modelopt_recipes/general/ptq/int4_blockwise_weight_only.yaml tests/unit/recipe/test_loader.py` - `pytest_pwd tests/unit/torch/export/test_unified_export_hf.py::test_fuse_prequant_layernorm_skips_modules_without_pre_quant_scale tests/unit/recipe/test_loader.py::test_load_recipe_all_builtins tests/unit/recipe/test_loader.py::test_general_ptq_yaml_matches_config_dicts tests/unit/recipe/test_presets.py -q` -> 29 passed Full GB10/Llama repro was not run locally. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> * **Bug Fixes** * Improved handling of layer norm fusion so it safely skips cases where pre-quantization scale data is unavailable, avoiding unexpected errors. * Updated the layer norm fusion flow to correctly apply scaling when pre-quantization data is present. * **Tests** * Expanded coverage for export and recipe loading scenarios, including cases with and without pre-quantization scale data. * Added validation for the new int4 blockwise weight-only recipe. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: realAsma <akuriparambi@nvidia.com>
- Skip pre-quant LayerNorm fusion when the representative input quantizer has no `_pre_quant_scale` buffer. - Add CPU unit coverage for the no-scale weight-only path and the existing fusion/removal behavior when pre-quant scales are present. - Compose the public `general/ptq/int4_blockwise_weight_only` recipe from the shared recipe units so it stays aligned with `INT4_BLOCKWISE_WEIGHT_ONLY_CFG`. NVBug: https://nvbugspro.nvidia.com/bug/6311597 NVBug 6311597 reports a deterministic HF export crash for `general/ptq/int4_blockwise_weight_only` on Llama-3.1-8B. That weight-only recipe disables input quantization and skips calibration, so `_pre_quant_scale` is not registered, but export still reaches `fuse_prequant_layernorm` through the AWQ-like format path. This PR also carries the recipe-sync diff that was previously in draft PR #1836; PR #1836 has been closed after moving that diff here. - `pre-commit run --files modelopt/torch/export/quant_utils.py tests/unit/torch/export/test_unified_export_hf.py modelopt_recipes/general/ptq/int4_blockwise_weight_only.yaml tests/unit/recipe/test_loader.py` - `pytest_pwd tests/unit/torch/export/test_unified_export_hf.py::test_fuse_prequant_layernorm_skips_modules_without_pre_quant_scale tests/unit/recipe/test_loader.py::test_load_recipe_all_builtins tests/unit/recipe/test_loader.py::test_general_ptq_yaml_matches_config_dicts tests/unit/recipe/test_presets.py -q` -> 29 passed Full GB10/Llama repro was not run locally. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> * **Bug Fixes** * Improved handling of layer norm fusion so it safely skips cases where pre-quantization scale data is unavailable, avoiding unexpected errors. * Updated the layer norm fusion flow to correctly apply scaling when pre-quantization data is present. * **Tests** * Expanded coverage for export and recipe loading scenarios, including cases with and without pre-quantization scale data. * Added validation for the new int4 blockwise weight-only recipe. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: realAsma <akuriparambi@nvidia.com> Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
What does this PR do?
Fixes the public
general/ptq/int4_blockwise_weight_onlyrecipe added in PR #1172 by composing it from the shared recipe units instead of carrying a manually duplicated quantizer exclusion list.The duplicated list had drifted from
configs/ptq/units/default_disabled_quantizers, so the public INT4 recipe missed newer standard exclusions such as linear-attention projections, VLM vision/projector modules, andnn.Embedding. Recipe mode can apply directly to the broader model path, so those missing exclusions can quantize modules that the shared preset/qformat path avoids.NVBug: 6311597
Testing
python_pwd tools/precommit/check_modelopt_recipes.py modelopt_recipes/general/ptq/int4_blockwise_weight_only.yamlpytest_pwd tests/unit/recipe/test_loader.py::test_load_recipe_all_builtins tests/unit/recipe/test_loader.py::test_general_ptq_yaml_matches_config_dicts -qpytest_pwd tests/unit/recipe/test_presets.py -qgeneral/ptq/int4_blockwise_weight_onlyincludes current standard exclusions for*linear_attn.in_proj_a*,*linear_attn.in_proj_b*, VLM modules, andnn.Embedding.Notes
The NVBug page redirects to Microsoft sign-in from my shell, so this fix is based on the Slack-linked PR diff anchor resolving to
modelopt_recipes/general/ptq/int4_blockwise_weight_only.yamland local recipe/preset drift analysis.Summary by CodeRabbit
New Features
Bug Fixes