Repository navigation
Marco-Nano 1- to 3-bit GGUFs and a Pal: 25 to 37 tokens/s on a Pixel 10a #1031
regalinvader-byte
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
We have released 1-, 2- and 3-bit GGUF builds of Marco-Nano-Instruct for phones, tested in PocketPal 1.21.2. It is a multilingual mixture-of-experts model with 8B parameters, 232 experts and 29 languages, but only 0.6B parameters are active per token, so it generates quickly on a phone CPU.
Rather than simply rounding the weights, we trained the experts in their low-bit formats by distilling from the original model. This keeps far more of its quality than llama.cpp's own quantisations at the same size.
Speed on a Pixel 10a
PocketPal 1.21.2 benchmark on a Pixel 10a (Tensor G4, 8 GB) with 4 CPU threads, f16 KV cache and 512/128:
There is also a 2-bit IQ2_S file (3.46 GB), which we did not time on the phone.
Which file to use
Instruction following and code are the model's weak spots, and this is true of the original as well. It also has no thinking mode.
Settings that matter in PocketPal
For the maintainers: the thread count made the biggest difference to speed in our tests, so a default based on the number of fast cores might help on other Tensor phones as well.
Feedback is welcome, in particular speeds from other phones.
All reactions