Use shared ModelOpt calibration loop on 0.45+ with 0.44 fallback fix#4881
Open
kevalmorabia97 wants to merge 1 commit into
Open
Use shared ModelOpt calibration loop on 0.45+ with 0.44 fallback fix#4881kevalmorabia97 wants to merge 1 commit into
kevalmorabia97 wants to merge 1 commit into
Conversation
Contributor
|
This PR has been automatically converted to draft because all PRs must start as drafts. When you are ready for review, click Ready for Review to begin the review process. This will:
See the contribution guide for more details. |
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
kevalmorabia97
changed the title
kevalmorabia97
changed the title
The new modelopt 0.45 shared util get_megatron_calibration_forward_loop unifies prune/quantize calibration with pack=True. Wrap both prune.py and quantize.py with try-import + _HAS_SHARED_CALIB so they continue to work on modelopt 0.44 (inline pack=True for prune, legacy local-JSONL / HF-dataset pad+truncate for quantize) and use the shared util on 0.45+. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
kevalmorabia97
force-pushed
the
kmorabia/modelopt-calib-loop-dual-compat
branch
from
676fa82 to
ab1d17c
Compare
kevalmorabia97
changed the title
Open
4 tasks
Contributor
Author
|
/ok to test ab1d17c |
kevalmorabia97
requested review from
AAnoosheh,
ChenhanYu and
jenchen13
and removed request for
jenchen13
kevalmorabia97
changed the title
kevalmorabia97
requested review from
yueshen2016
and removed request for
a team and
AAnoosheh
kevalmorabia97
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related: NVIDIA/Model-Optimizer#1501
Summary
prune.pyandquantize.pynow use ModelOpt 0.45's sharedget_megatron_calibration_forward_loop(one sample per row + per-row trim + EOS-at-row-end), guarded by atry-import so 0.44 falls back to the prior inline path.--calib-dataset nemotron-post-training-dataset-v2,--calib-size 1024,--calib-max-sequence-length 4096,--calib-batch-size 1. Conservative defaults sized for MoE robustness (top-K routing → fewer tokens per expert → more samples × longer seq needed for stable amax / scoring statistics).Results
Qwen3-8B (TP=1 PP=2 prune; TP=2 PP=1 quantize). MMLU 5% fraction, 0-shot, bs=4. "Original" = pre-PR inline per-example calibration; "Shared" = ModelOpt 0.45 shared loop.
Minitron prune (Qwen3-8B → 30L/3584/11776 ≈ 6B params)
pack=TrueWAR)NVFP4 quantize (
NVFP4_DEFAULT_CFG)get_calib_dataloaderpad+truncate)Takeaway: the shared loop is ≥ original on every workload tested, with +2.5 to +3.0 pt MMLU wins where the original underperformed. Differences across (seq_len, calib_size, calib_batch_size) on the shared path are within MMLU noise floor for dense Qwen3-8B; defaults err on the conservative side for MoE pruning.
🤖 Generated with Claude Code