Support for n-gram/prompt-lookup speculative decoding (draftless, no separate model needed) #8653
joshuaneill2121-del
started this conversation in
Feature Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
llama.cpp's server supports several draftless speculative decoding methods via --spec-type — ngram-simple, ngram-map-k, ngram-map-k4v, and ngram-mod — that speed up generation by matching repeated token sequences already in the prompt/context, with no separate draft model required at all. I checked through Jan's Model Settings (MTP, speculative decoding via draft model, sampling params) and couldn't find any of these exposed anywhere in the UI.
This would be genuinely useful for people who can't get a compatible draft model pairing working — I've been troubleshooting draft/target compatibility with Gemma 4 12B + E2B, and an n-gram-based method would give a similar speedup without needing a second model loaded or a compatible draft pairing at all. It's also apparently able to be combined with an existing draft model setup per llama.cpp's own docs, so it's not necessarily an either/or with what Jan already has.
Is this something already planned, or worth raising as a proper request?
All reactions