Skip to content

[WIP] Make AWQ more general #2400

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[WIP] Make AWQ more general #2400

wants to merge 1 commit into from
+809 −41

Conversation

jerryzh168
Copy link
Contributor

@jerryzh168 jerryzh168 commented Jun 18, 2025 •
edited
Loading

Summary:

  • Added AWQConfig that takes a base config and made corresponding changes in other parts of the flow

Test Plan:

# Produce model
# make sure to change the model_save_path
python torchao/prototype/awq/example2.py --repo "Qwen/Qwen3-4B" --quant awq-8da4w-128 --tasks bhh --model_save_hf_hub_path jerryzh168/Qwen3-4B-8da4w-awq


# eval
lm_eval --model hf --model_args pretrained=jerryzh168/Qwen3-4B-8da4w-awq --tasks bhh --device cuda:0 --batch_size auto

Reviewers:

Subscribers:

Tasks:

Tags:

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Jun 18, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/2400

Note: Links to docs will display an error until the docs builds have been completed.

❌ 11 New Failures

As of commit 8b1fca1 with merge base c561d26 (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 18, 2025
@jerryzh168 jerryzh168 mentioned this pull request Jun 18, 2025
Open
kimishpatel
kimishpatel reviewed Jun 18, 2025
View reviewed changes
torchao/prototype/awq/api.py
eps=eps,
)
else:
observer = AWQObserver2(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you not add kwargs to the AWQObserver and just check 'base_config' in kwargs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, this is temporary, I think we can deprecate the old one in the end

kimishpatel
kimishpatel reviewed Jun 18, 2025
View reviewed changes
torchao/prototype/awq/api.py


@dataclass
class AWQConfig(AOBaseConfig):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok this is consolidating with quantize_ api's config based design?

kimishpatel
kimishpatel reviewed Jun 18, 2025
View reviewed changes
torchao/prototype/awq/api.py
Comment on lines +242 to +282
dummy_mod = DummyModule(observed_linear.weight * equalization_scale)
quant_mod = base_config_handler(dummy_mod, config.base_config)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure whats happening here?. Isnt module already nn.Module?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is just trying to quantize the weight with the quantization type specified by config.base_config

kimishpatel
kimishpatel reviewed Jun 18, 2025
View reviewed changes
torchao/prototype/awq/api.py Outdated
if config.set_inductor_config:
torchao.quantization.utils.recommended_inductor_config_setter()

observed_linear = module
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is for linear only should you not assert that this nn.Linear? Plus how to you make sure this function is called only on nn.Linear?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah that's true, will add an assert, we rely on user to use quantize_ correctly (it's through specifying the filter_fn arg in quantize_ API)

ao/torchao/quantization/quant_api.py

Line 578 in 4e3d019

filter_fn: Optional[Callable[[torch.nn.Module, str], bool]] = None,

@jerryzh168
[WIP] Make AWQ more general
8b1fca1 
Summary:
* Added AWQConfig that takes a base config and made corresponding changes
in other parts of the flow

Test Plan:
TODO

Reviewers:

Subscribers:

Tasks:

Tags:
@jerryzh168 jerryzh168 added the topic: improvement Use this tag if this PR is an improvement (doesn't fit into any of the other categories) label Jun 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kimishpatel kimishpatel kimishpatel left review comments

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. topic: improvement Use this tag if this PR is an improvement (doesn't fit into any of the other categories)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jerryzh168 @kimishpatel @facebook-github-bot