Make HQQ default PTQ quantization in ExecuTorch

[metascroy]

On MMLU, we see a 1% to 10% improvement on MMLU accuracy by using HQQ algorithm for quantization, depending on the model.

We make this the default quantization algorithm in the export LLM export script:

MODEL			INT8-INT4		INT8-INT4-HQQ
---------------------------------------------
Gemma3-4B		0.50			0.55
SmolLM3-3B		0.5397			0.5546
Phi4-mini-4B	0.6050			0.6156
Qwen3-4B		0.6526			0.6585			
Llama1B			0.2892			0.3153	

Evaulations were done using lm_eval with the following command:

HF_HUB_DISABLE_XET=1 lm_eval --model hf --model_args pretrained=quantized_model_id --tasks mmlu --device cuda:0 --batch_size 8

Here quantized_model_id corresponding to one of the above model/quant-scheme combinations uploaded to my huggingface account.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/14834

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 6 New Failures, 23 Cancelled Jobs, 2 Unrelated Failures

As of commit 55a9588 with merge base d4129b7 ():

NEW FAILURES - The following jobs have failed:

• pull / unittest / linux / linux-job (gh)
  examples/models/test/test_export.py::ExportTest::test_ic3_export_to_executorch
• trunk / test-arm-cortex-m-size-test (zephyr-preset) / linux-job (gh)
  RuntimeError: Command docker exec -t 078242bad2a8be684d662c9065a9170549e8e04a1c573bc6e42926479e8b01e3 /exec failed with exit code 1
• trunk / test-models-macos-coreml (efficient_sam) / macos-job (gh)
  Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/Users/ec2-user/runner/_work/executorch/executorch/test-infra/.github/actions/check-disk-space'. Did you forget to run actions/checkout before running your local action?
• trunk / test-models-macos-coreml (emformer_transcribe) / macos-job (gh)
  Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/Users/ec2-user/runner/_work/executorch/executorch/test-infra/.github/actions/check-disk-space'. Did you forget to run actions/checkout before running your local action?
• trunk / test-models-macos-coreml (resnet50) / macos-job (gh)
  Missing file at path: /Users/ec2-user/runner/_work/_temp/_runner_file_commands/set_output_d366cf5e-a00f-4137-9b93-247d3c575d6d
• trunk / test-selective-build-macos (cmake) / macos-job (gh)
  Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/Users/ec2-user/runner/_work/executorch/executorch/test-infra/.github/actions/check-disk-space'. Did you forget to run actions/checkout before running your local action?

CANCELLED JOBS - The following jobs were cancelled. Please retry:

• pull / unittest / macos / macos-job (gh)
  ##[error]The operation was canceled.
• pull / unittest-buck / macos / macos-job (gh)
• pull / unittest-editable / macos / macos-job (gh)
• trunk / test-custom-ops-macos (cmake) / macos-job (gh)
  ##[error]The operation was canceled.
• trunk / test-llama-runner-mac (fp32, coreml) / macos-job (gh)
  ##[error]The operation was canceled.
• trunk / test-llama-runner-mac (fp32, mps) / macos-job (gh)
  ##[error]The operation was canceled.
• trunk / test-models-macos-coreml (dl3) / macos-job (gh)
  ##[error]The operation was canceled.
• trunk / test-models-macos-coreml (edsr) / macos-job (gh)
  ##[error]The operation was canceled.
• trunk / test-models-macos-coreml (emformer_join) / macos-job (gh)
• trunk / test-models-macos-coreml (ic4) / macos-job (gh)
  ##[error]The operation was canceled.
• trunk / test-models-macos-coreml (mobilebert) / macos-job (gh)
  ##[error]The operation was canceled.
• trunk / test-models-macos-coreml (mv2) / macos-job (gh)
  ##[error]The operation was canceled.
• trunk / test-models-macos-coreml (mv3) / macos-job (gh)
  ##[error]The operation was canceled.
• trunk / test-models-macos-cpu (efficient_sam, portable) / macos-job (gh)
  ##[error]The operation was canceled.
• trunk / test-models-macos-cpu (ic4, xnnpack-quantization-delegation) / macos-job (gh)
  ##[error]The operation was canceled.
• trunk / test-models-macos-cpu (llama, portable) / macos-job (gh)
  ##[error]The operation was canceled.
• trunk / test-models-macos-cpu (llama2, xnnpack-quantization-delegation) / macos-job (gh)
  ##[error]The operation was canceled.
• trunk / test-models-macos-cpu (mobilebert, xnnpack-quantization-delegation) / macos-job (gh)
  ##[error]The operation was canceled.
• trunk / test-models-macos-cpu (mv3, xnnpack-quantization-delegation) / macos-job (gh)
  ##[error]The operation was canceled.
• trunk / test-models-macos-cpu (resnet50, xnnpack-quantization-delegation) / macos-job (gh)
  ##[error]The operation was canceled.
• trunk / test-models-macos-cpu (vit, xnnpack-quantization-delegation) / macos-job (gh)
  ##[error]The operation was canceled.
• trunk / test-models-macos-cpu (w2l, xnnpack-quantization-delegation) / macos-job (gh)
  ##[error]The operation was canceled.
• trunk / unittest-release / macos / macos-job (gh)
  ##[error]The operation was canceled.

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• pull / unittest-arm-backend-with-no-fvp (test_pytest_models) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Apple / build-demo-ios / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 74

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[meta-codesync]

@metascroy has imported this pull request. If you are a Meta employee, you can view this in D84020605.

[mergennachin]

This is awesome!! Thanks for working on this

[mergennachin]

Can we make this part of the release/1.0?

[jackzhxng]

How did you test gemma3 with etllm? (from your pr description)

[metascroy]

How did you test gemma3 with etllm? (from your pr description)

I published quantized checkpoints for all of these models on my personal huggingface page: https://huggingface.co/metascroy

So you can do something like:

HF_HUB_DISABLE_XET=1 lm_eval --model hf --model_args pretrained=metascroy/gemma-3-4b-it-INT8-INT4-HQQ --tasks mmlu --device cuda:0 --batch_size 8

to re-create the number.

[metascroy]

Can we make this part of the release/1.0?

I'll cherry pick, but I need to land the AO diff train before this so we don't break internal.

[meta-codesync]

@metascroy has exported this pull request. If you are a Meta employee, you can view the originating Diff in D84020605.

[metascroy]

@pytorchbot cherry-pick --onto release/1.0 -c critical

[metascroy]

@pytorchbot cherry-pick --onto release/1.0 -c "critical"

[pytorchbot]

Cherry picking #14834

Command git -C /home/runner/work/executorch/executorch cherry-pick -x -X theirs d39992f6d971e3548ee3ffe943d9224f63979126 returned non-zero exit code 1

hint: Recursive merging with submodules currently only supports trivial cases.
hint: Please manually handle the merging of each conflicted submodule.
hint: This can be accomplished with the following steps:
hint:  - go to submodule (third-party/ao), and either merge commit 01849b2b1
hint:    or update to an existing commit which has merged those changes
hint:  - come back to superproject and run:
hint:
hint:       git add third-party/ao
hint:
hint:    to record the above merge or update
hint:  - resolve any other conflicts in the superproject
hint:  - commit the resulting index in the superproject
hint:
hint: Disable this message with "git config set advice.submoduleMergeConflict false"
Auto-merging examples/models/llama/export_llama_lib.py
Failed to merge submodule third-party/ao (commits don't follow merge-base)
CONFLICT (submodule): Merge conflict in third-party/ao
error: could not apply d39992f6d9... Make HQQ default PTQ quantization in ExecuTorch
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config set advice.mergeConflict false"

Details for Dev Infra team

Raised by workflow job