Update to using Model Optimizer (formerly AMMO) in PTQ workflow #9178

janekl · 2024-05-13T12:48:28Z

What does this PR do ?

Summary:

use Model Optimizer library (formerly AMMO) for LLM PTQ workflow
restore PTQ tests failing due to setting apply_rope_fusion = True by default
update documentation

Collection: NLP

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

kevalmorabia97 · 2024-05-13T13:38:06Z

nemo/export/quantize/quantizer.py

@@ -197,7 +202,7 @@ def forward_loop():
                    print(f"Calibrating batch {i}")
                model.predict_step(batch, i)

-        model = atq.quantize(model, self.atq_config, forward_loop)
+        model = mtq.quantize(model, self.quant_cfg, forward_loop)


Please note that def forward_loop(model): ... is recommended over def forward_loop(): ... and the latter one will be deprecated soon

kevalmorabia97 · 2024-05-13T13:39:22Z

.github/workflows/cicd-main.yml

-            # AMMO installation
-            pip install nvidia-ammo~=0.9.0 --extra-index-url https://pypi.nvidia.com --no-cache-dir
+            # ModelOpt installation
+            pip install nvidia-modelopt[all]~=0.11.0 --extra-index-url https://pypi.nvidia.com --no-cache-dir


I believe you dont need onnx or deployment dependencies here

Suggested change

pip install nvidia-modelopt[all]~=0.11.0 --extra-index-url https://pypi.nvidia.com --no-cache-dir

pip install nvidia-modelopt[torch]~=0.11.0 --extra-index-url https://pypi.nvidia.com --no-cache-dir

nemo/export/quantize/quantizer.py

kevalmorabia97 · 2024-05-14T13:03:15Z

examples/nlp/language_modeling/megatron_quantization.py

megatron_gpt_quantization may be a more appropriate and consistent name

Maybe.. WDYT @ericharper ?

Background is that the script is supposed to support multiple models -- gpt, llama, gemma etc. by specifying decoder_type param -- so I decided to rename it megatron_llama_quantization.py -> megatron_quantization.py.

Let's postpone final decision after merge as this name also is tied to the one used in NeMo-Launcher

pablo-garay · 2024-05-14T20:16:05Z

Can we look to merge this PR soon? @janekl @kevalmorabia97

kevalmorabia97 · 2024-05-15T06:00:20Z

@pablo-garay no objection from my side. I don't have permissions to approve but this LGTM

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

Signed-off-by: janekl <janekl@users.noreply.github.com>

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

Signed-off-by: janekl <janekl@users.noreply.github.com>

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

oyilmaz-nvidia

LGTM.

* Update PTQ to use nvidia-modelopt Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Restore PTQ tests Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Update docs Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Comment on apply_rope_fusion Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Support for calibration PP > 1 Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Apply isort and black reformatting Signed-off-by: janekl <janekl@users.noreply.github.com> * Fix cicd-main.yml indent Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Set data/tensor parallel groups Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Install only torch dependecies Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Follow up on recent modelopt changes Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Model support matrix Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Apply isort and black reformatting Signed-off-by: janekl <janekl@users.noreply.github.com> * Rename PTQ script as it should be model-agnostic Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Remove unused import Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Update setup instructions Signed-off-by: Jan Lasek <janek.lasek@gmail.com> --------- Signed-off-by: Jan Lasek <janek.lasek@gmail.com> Signed-off-by: janekl <janekl@users.noreply.github.com> Co-authored-by: janekl <janekl@users.noreply.github.com>

…IA#9178) * Update PTQ to use nvidia-modelopt Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Restore PTQ tests Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Update docs Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Comment on apply_rope_fusion Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Support for calibration PP > 1 Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Apply isort and black reformatting Signed-off-by: janekl <janekl@users.noreply.github.com> * Fix cicd-main.yml indent Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Set data/tensor parallel groups Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Install only torch dependecies Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Follow up on recent modelopt changes Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Model support matrix Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Apply isort and black reformatting Signed-off-by: janekl <janekl@users.noreply.github.com> * Rename PTQ script as it should be model-agnostic Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Remove unused import Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Update setup instructions Signed-off-by: Jan Lasek <janek.lasek@gmail.com> --------- Signed-off-by: Jan Lasek <janek.lasek@gmail.com> Signed-off-by: janekl <janekl@users.noreply.github.com> Co-authored-by: janekl <janekl@users.noreply.github.com> Signed-off-by: Boxiang Wang <boxiangw@nvidia.com>

…IA#9178) * Update PTQ to use nvidia-modelopt Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Restore PTQ tests Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Update docs Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Comment on apply_rope_fusion Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Support for calibration PP > 1 Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Apply isort and black reformatting Signed-off-by: janekl <janekl@users.noreply.github.com> * Fix cicd-main.yml indent Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Set data/tensor parallel groups Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Install only torch dependecies Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Follow up on recent modelopt changes Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Model support matrix Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Apply isort and black reformatting Signed-off-by: janekl <janekl@users.noreply.github.com> * Rename PTQ script as it should be model-agnostic Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Remove unused import Signed-off-by: Jan Lasek <janek.lasek@gmail.com> * Update setup instructions Signed-off-by: Jan Lasek <janek.lasek@gmail.com> --------- Signed-off-by: Jan Lasek <janek.lasek@gmail.com> Signed-off-by: janekl <janekl@users.noreply.github.com> Co-authored-by: janekl <janekl@users.noreply.github.com>

github-actions bot added NLP CI labels May 13, 2024

janekl added Run CICD and removed Run CICD labels May 13, 2024

janekl force-pushed the jlasek/ammo2modelopt branch from 1a7fa1f to ad3dc28 Compare May 13, 2024 12:51

janekl changed the title ~~Jlasek/ammo2modelopt~~ Update to nvidia-modelopt in PTQ workflow May 13, 2024

janekl changed the title ~~Update to nvidia-modelopt in PTQ workflow~~ Update to using Model Optimizer (formerly AMMO) in PTQ workflow May 13, 2024

janekl added the Run CICD label May 13, 2024

kevalmorabia97 reviewed May 13, 2024

View reviewed changes

janekl removed the Run CICD label May 13, 2024

pablo-garay previously approved these changes May 14, 2024

View reviewed changes

janekl dismissed pablo-garay’s stale review via 9618f85 May 14, 2024 11:01

janekl force-pushed the jlasek/ammo2modelopt branch from 4f4080e to d54836b Compare May 14, 2024 11:08

github-advanced-security bot found potential problems May 14, 2024

View reviewed changes

nemo/export/quantize/quantizer.py Fixed Show fixed Hide fixed

kevalmorabia97 reviewed May 14, 2024

View reviewed changes

janekl added the Run CICD label May 14, 2024

janekl and others added 11 commits May 15, 2024 13:15

Update PTQ to use nvidia-modelopt

ba771c8

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

Restore PTQ tests

3ee4a11

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

Update docs

1de7d1b

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

Comment on apply_rope_fusion

7e07f54

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

Support for calibration PP > 1

b76f954

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

Apply isort and black reformatting

8d94348

Signed-off-by: janekl <janekl@users.noreply.github.com>

Fix cicd-main.yml indent

e9577b2

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

Set data/tensor parallel groups

5cd63aa

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

Install only torch dependecies

6fddcf5

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

Follow up on recent modelopt changes

65dacbd

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

Model support matrix

a6ef378

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

janekl and others added 3 commits May 15, 2024 13:15

Apply isort and black reformatting

8300844

Signed-off-by: janekl <janekl@users.noreply.github.com>

Rename PTQ script as it should be model-agnostic

c46ac4d

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

Remove unused import

4813f10

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

janekl force-pushed the jlasek/ammo2modelopt branch from 07faf83 to 4813f10 Compare May 15, 2024 11:15

Update setup instructions

291be6a

Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

janekl added Run CICD and removed Run CICD labels May 15, 2024

oyilmaz-nvidia approved these changes May 15, 2024

View reviewed changes

oyilmaz-nvidia merged commit 6cb618a into main May 15, 2024
132 of 133 checks passed

oyilmaz-nvidia deleted the jlasek/ammo2modelopt branch May 15, 2024 13:01

ko3n1g mentioned this pull request Jul 18, 2024

Release 2.0.0rc1 #9786

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to using Model Optimizer (formerly AMMO) in PTQ workflow #9178

Update to using Model Optimizer (formerly AMMO) in PTQ workflow #9178

janekl commented May 13, 2024

kevalmorabia97 May 13, 2024

janekl May 14, 2024

kevalmorabia97 May 13, 2024

janekl May 14, 2024

kevalmorabia97 May 14, 2024

janekl May 14, 2024

janekl May 14, 2024

janekl May 15, 2024

pablo-garay commented May 14, 2024

kevalmorabia97 commented May 15, 2024

oyilmaz-nvidia left a comment

	pip install nvidia-modelopt[all]~=0.11.0 --extra-index-url https://pypi.nvidia.com --no-cache-dir
	pip install nvidia-modelopt[torch]~=0.11.0 --extra-index-url https://pypi.nvidia.com --no-cache-dir

Update to using Model Optimizer (formerly AMMO) in PTQ workflow #9178

Update to using Model Optimizer (formerly AMMO) in PTQ workflow #9178

Conversation

janekl commented May 13, 2024

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Additional Information

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pablo-garay commented May 14, 2024

kevalmorabia97 commented May 15, 2024

oyilmaz-nvidia left a comment

Choose a reason for hiding this comment