Quant in checkpoint dtype

[metascroy]

Switches order in etLLM so we quantize in checkpoint dtype and then cast to dtype-override. This can prevent underflowing on scales.

Also exposes ability to turn HQQ on/off.

Export:

python -m extension.llm.export.export_llm \   
  base.model_class=phi_4_mini \
  base.params=examples/models/phi_4_mini/config/config.json \  model.use_kv_cache=true \     
  model.use_sdpa_with_kv_cache=true \
  model.dtype_override=fp32 \  export.output_dir=/tmp/phi_4_mini_no_hqq \  export.output_name=model.pte \  export.max_seq_length=2048 \  export.max_context_length=2048 \  quantization.qmode=8da4w \  quantization.group_size=32 "quantization.embedding_quantize='8,0'" quantization.use_hqq=False \                           
  backend.xnnpack.enabled=true \
  backend.xnnpack.extended_ops=true

Phi4 output:

<|im_start|>system
You are a highly capable, helpful, and honest AI assistant designed to provide clear, accurate, and thoughtful responses to a wide range of questions. Your primary goal is to assist users by offering information, explanations, and guidance in a manner that is respectful, unbiased, and safe. Always strive to be as helpful as possible, but never provide content that is harmful, unethical, offensive, or illegal. If a question is unclear, nonsensical, or based on incorrect premises, politely explain the issue rather than attempting to answer inaccurately. If you do not know the answer to a question, it is better to admit uncertainty than to provide false or misleading information. When appropriate, include examples, analogies, or step-by-step reasoning to enhance understanding. Your responses should be positive, inclusive, and supportive, fostering a constructive and informative interaction.<|im_end|>
<|im_start|>user
Please answer the following question in detail and provide relevant context, examples, and explanations where possible: What are some of the most important considerations when designing a machine learning system for real-world applications? Discuss potential challenges, best practices, and how to ensure ethical and responsible use.<|im_end|>
<|im_start|>assistant
Designing a machine learning system for real-world applications involves various considerations to ensure the system is effective, fair, and secure. Some of the most important considerations include data quality and sourcing, model choice and design, evaluation and validation, interpretability and transparency, and ensuring fairness and avoiding biases.

Data quality and sourcing involve ensuring data is of high quality, representative of the target application, and properly curated and preprocessed to remove noise and biases.

Model choice and design involve selecting an appropriate model for the application, understanding the strengths and limitations of different models, and understanding the application domain and data.

Model evaluation and validation involve properly training and tuning the model on a training set and properly validating and testing the model on a separate validation set to avoid data leakage and

Related work: improvement in torchao's HQQ algorithm that helps with Phi4's model distribution: pytorch/ao#4259

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18781

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• Rolling out OSDC (ARC) runners on pull workflow for PyTorch trunk commits

❌ 2 New Failures, 1 Pending, 2 Unrelated Failures

As of commit 6e67701 with merge base 36e8ed9 ():

NEW FAILURES - The following jobs have failed:

• pull / test-samsung-models-linux / linux-job (gh)
  test_w2l_fp16
• pull / unittest-arm-backend-with-no-deps (test_pytest_ops_tosa) / linux-job (gh)
  RuntimeError: Command docker exec -t 263cde0d7d4b5d00ba721d6a1db583886415d3a426555a864c2e01d0a4b54fcd /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / windows / windows-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• pull / unittest-editable / windows / windows-job (gh) (trunk failure)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[meta-codesync]

@metascroy has imported this pull request. If you are a Meta employee, you can view this in D100066455.

[SS-JIA]

LGTM!!