Qualcomm AI Engine Direct - GA Static Smollm3 3B #14149
Conversation
DannyYuyang-quic
requested review from
cccclai,
lucylq and
jackzhxng
as code owners
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/14149
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (9 Unrelated Failures)As of commit cd3b2f4 with merge base 6ed10e5 ( FLAKY - The following jobs failed but were likely due to flakiness present on trunk:
BROKEN TRUNK - The following jobs failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
bot
added
the
CLA Signed
|
Hi @cccclai, this is the PR for SmolLM3-3B in static version, please have a look! cc: @haowhsu-quic |
DannyYuyang-quic
force-pushed
the
dev1/danny/GA_static_Smollm3
branch
from
ac9394f to
953112e
Compare
|
@pytorchbot label "release notes: qualcomm" |
pytorch-bot
bot
added
the
release notes: qualcomm
|
There is still lint error and conflict |
|
|
||
| from torchtune.models.convert_weights import get_mapped_key | ||
|
|
||
| from torchtune.training import FullModelHFCheckpointer |
There was a problem hiding this comment.
Can we use this instead? I'd like to move away from TorchTune if possible
There was a problem hiding this comment.
Sounds good! I’ll make the changes, thanks!!
There was a problem hiding this comment.
Hi @jackzhxng, I’ve made the changes. Could you please review them again and let me know if they now meet your expectations? Thanks!
cc: @cccclai
DannyYuyang-quic
force-pushed
the
dev1/danny/GA_static_Smollm3
branch
2 times, most recently
from
3a71f1f to
cd3b2f4
Compare
| if args.decoder_model == "smollm3-3b": | ||
| from transformers import AutoConfig | ||
|
|
||
| kv_config.apply_rope_layers = AutoConfig.from_pretrained( |
There was a problem hiding this comment.
I don't feel too strongly about this but if you want to avoid having a transformers dep, what if we just did this instead - add to model_args:
And in your 3b_config.json
"no_rope_layer_interval": 4
There was a problem hiding this comment.
Thanks for the suggestion, I like this change!
I’ve updated the config to include "no_rope_layer_interval": 4.
Appreciate the tip!
|
I guess there is merge conflict again. Can you resolve it? |
|
Since #12700 is merged, is it the last PR we need to merge? |
Summary: - e2e script for GA Static SmolLM3-3B - perf: 16a4w block quant token rate in kv mode: ~= 30 tokens/sec(SM8750) - acc: PPL ~= (fp: 8.345 -> htp:8.976) in wikitext dataset - add model params file & model weight converter
DannyYuyang-quic
force-pushed
the
dev1/danny/GA_static_Smollm3
branch
from
cd3b2f4 to
4f3d12e
Compare
I've rebased the PR! Thanks! |
@haowhsu-quic |
okay ping me when you have it out |
|
or add milestones 1.0.0 so we can track them |
Summary:
- e2e script for GA Static SmolLm3-3B
- perf: 16a4w block quant token rate in kv mode: ~= 30
tokens/sec(SM8750)
- acc: PPL ~= (fp: 8.345 -> htp:8.976) in wikitext dataset
- add model params file & model weight converter
### Test plan
``` bash
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s ${SERIAL_NUM} -m ${SOC_MODEL} --decoder_model smollm3-3b --model_mode kv --max_seq_len 1024 --prompt "I would like to learn python, could you teach me with a simple example?" --tasks wikitext --limit 1
```
Summary:
Test plan