Qualcomm AI Engine Direct - Fixed the order of the transforms for llama #5221
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/5221
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit dfd80b0 with merge base 657789e ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
|
Hi @cccclai, |
|
I have a small request. I would like to add a document that explains how to export llama with Qualcomm AI Engine Direct, including steps like downloading spin quant and setting num_sharding. However, I’m not sure where the best place to save this file would be. Could you please advise? |
shewu-quic
force-pushed
the
dev1/hutton/fixed_spin_quant_r1_r2
branch
from
999a992 to
0e23b45
Compare
| if self._generate_full_logits: | ||
| return torch.cat(result_logits, dim=1) | ||
| else: | ||
| return torch.stack(result_logits, dim=1) |
There was a problem hiding this comment.
hmm what's the difference between these?
There was a problem hiding this comment.
Because the shape of the function output should be (batch, seq, vocab_size).
If _generate_full_logits, the shape of each result logit in result_logits are (batch, seq, vocab_size)
We could just use cat by dim=1.
If not _generate_full_logits, the shape of each result logit in result_logits are (batch, vocab_size).
We will need use stack to get one more dimension (batch, seq, vocab_size)
|
CI needs to be fixed. |
shewu-quic
force-pushed
the
dev1/hutton/fixed_spin_quant_r1_r2
branch
from
807d763 to
dfd80b0
Compare
|
@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
3 participants
We need to apply r1 r2 before converting linear to conv.