Skip to content

Update QCOM llama hardware support #15965

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Dec 1, 2025
Merged

Update QCOM llama hardware support #15965

merged 4 commits into from
Dec 1, 2025

Conversation

@cccclai
Copy link
Contributor

@cccclai cccclai commented Nov 24, 2025

Added hardware support details and memory limit error handling instructions.

It seems like lots of users try to use the llama flow on other hardware other than phones. Let's try to document it first

@pytorch-bot
Copy link

pytorch-bot bot commented Nov 24, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15965

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

✅ No Failures

As of commit 8fe731e with merge base 12d17ef (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 24, 2025
@cccclai cccclai mentioned this pull request Nov 24, 2025
Closed
@github-actions
Copy link

github-actions bot commented Nov 24, 2025

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

shewu-quic
shewu-quic reviewed Nov 25, 2025
View reviewed changes
Copy link
Collaborator

@shewu-quic shewu-quic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks to make document better!

examples/qualcomm/oss_scripts/llama/README.md Outdated

We’ve validated this flow on the **Samsung Galaxy S23**, **Samsung Galaxy S24**, and **OnePlus 12**.
Support on other hardware depends on the **HTP architecture (HtpArch)**.
The **16a4w_block** format and **weight sharing between prefill and decode** are supported on **V73 and newer**.
Copy link
Collaborator

@shewu-quic shewu-quic Nov 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minimum requirement of LPBQ is v69.
Minimum requirement of weight sharing is v73
Minimum requirement of 16bit activation and 16 bit weight for matmul is V73. (16 bit kv)

Copy link
Contributor Author

@cccclai cccclai Nov 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated, what do you think?

examples/qualcomm/oss_scripts/llama/README.md Outdated
For older devices, you may need to **retune the quantization recipe**. A good starting point is:

- Use **16a4w**
- Optionally apply **SpinQuant** for better stability and accuracy
Copy link
Collaborator

@shewu-quic shewu-quic Nov 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially layers apply 16a8w to get better accuracy.

Copy link
Contributor Author

@cccclai cccclai Nov 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

shewu-quic
shewu-quic approved these changes Nov 26, 2025
View reviewed changes
Copy link
Collaborator

@shewu-quic shewu-quic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@cccclai
Update QCOM llama hardware support
18d9c69 
Added hardware support details and memory limit error handling instructions.
@cccclai
Update README.md
66859c7
@cccclai
Update README with correct LPBQ version requirement
6773d3b
@cccclai
Add Samsung Galaxy S25 to hardware support list
8fe731e 
Updated hardware support section to include Samsung Galaxy S25.
@cccclai cccclai merged commit 60a2bd6 into main Dec 1, 2025
165 checks passed
@cccclai cccclai deleted the cccclai-patch-12 branch December 1, 2025 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Gasoonjia Gasoonjia Gasoonjia approved these changes

@haowhsu-quic haowhsu-quic Awaiting requested review from haowhsu-quic

@winskuo-quic winskuo-quic Awaiting requested review from winskuo-quic

@chenweng-quic chenweng-quic Awaiting requested review from chenweng-quic

+1 more reviewer

@shewu-quic shewu-quic shewu-quic approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@cccclai @Gasoonjia @shewu-quic