You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have implemented speculative drafting via model over TCP/IP using existing llamaserver API /completion that allows to send context as tokenids between models with the same token library. Speedup can be more than 50% in good situations (see test results below).
Good thing was that it was easier than I tought, and patch is fairly small. I am not especially familiar with github and doing PR and things. Can this be of interest?
How do I proceed?
BR
Description
CLI options for remote draft:
• --spec-type draft-remote (added draft-remote to existing --spec-type)
• --spec-draft-model-url (new)
• --spec-draft-model-name (new, optional, default empty)
• --spec-draft-api-key (new, optional, default empty)
• --spec-draft-context-limit (new, optional, default: 8192)
• --spec-draft-n-max (reused, optional - now works for both local and remote draft, default 3)
Test setup
Two computers with direct connection using a 10Gbit/s ethernet point to point cable. Main model is running on an RTX 6000 Pro Blackwell and draft model is running on a RTX 5090.
Main model: Devstral-2-123B-Instruct-2512-Q4_K_M (KV: q8_0, ctx_length: 128k), standard 15-20 tokens/s slowing to below 10 on large contexts
Draft model: Ministral-3-3B-Instruct-2512-Q4_K_M (KV: q8_0, ctx_length: 8k), 300-400 tokens/second
Result
Programming task writing a python program:
draft acceptance = 0.81300 ( 2013 accepted / 2476 generated), mean len = 4.5
Speedup: 25-30 tokens per second, increasing speed 50%+. When context grows, speedup increases since main model slows a lot.
For natural language tasks, acceptance drops to 0.3-0.5 depending on question and speed up is between 0-20% only.
My test commands
draft model start:
/home/1/llama.cpp/build/bin/llama-server
--model /home/mir/services/models/Ministral-3-3B-Instruct-2512-Q4_K_M.gguf
-fa on
-ctk q8_0
-ctv q8_0
-c 8192
-ngl 99
-b 1024
-ub 1024
--jinja
--kv-unified
--no-mmap
--parallel 4
--threads 8
--temp 0.0
--top-k 1
--host 0.0.0.0
--port 3535
main model start:
/home/2/llama.cpp/build/bin/llama-server
--model /home/mir/services/models/Devstral-2-123B-Instruct-2512-Q4_K_M-00001-of-00002.gguf
-fa on
-ctk q8_0
-ctv q8_0
-c 131072
--no-mmap
--parallel 4
--jinja
--threads 16
--spec-type draft-remote
--spec-draft-model-url http://x.x.x.x:3535/v1
--spec-draft-n-max 5
--port 3536
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
I have implemented speculative drafting via model over TCP/IP using existing llamaserver API /completion that allows to send context as tokenids between models with the same token library. Speedup can be more than 50% in good situations (see test results below).
Good thing was that it was easier than I tought, and patch is fairly small. I am not especially familiar with github and doing PR and things. Can this be of interest?
How do I proceed?
BR
Description
CLI options for remote draft:
• --spec-type draft-remote (added draft-remote to existing --spec-type)
• --spec-draft-model-url (new)
• --spec-draft-model-name (new, optional, default empty)
• --spec-draft-api-key (new, optional, default empty)
• --spec-draft-context-limit (new, optional, default: 8192)
• --spec-draft-n-max (reused, optional - now works for both local and remote draft, default 3)
Test setup
Two computers with direct connection using a 10Gbit/s ethernet point to point cable. Main model is running on an RTX 6000 Pro Blackwell and draft model is running on a RTX 5090.
Main model: Devstral-2-123B-Instruct-2512-Q4_K_M (KV: q8_0, ctx_length: 128k), standard 15-20 tokens/s slowing to below 10 on large contexts
Draft model: Ministral-3-3B-Instruct-2512-Q4_K_M (KV: q8_0, ctx_length: 8k), 300-400 tokens/second
Result
Programming task writing a python program:
draft acceptance = 0.81300 ( 2013 accepted / 2476 generated), mean len = 4.5
Speedup: 25-30 tokens per second, increasing speed 50%+. When context grows, speedup increases since main model slows a lot.
For natural language tasks, acceptance drops to 0.3-0.5 depending on question and speed up is between 0-20% only.
My test commands
draft model start:
/home/1/llama.cpp/build/bin/llama-server
--model /home/mir/services/models/Ministral-3-3B-Instruct-2512-Q4_K_M.gguf
-fa on
-ctk q8_0
-ctv q8_0
-c 8192
-ngl 99
-b 1024
-ub 1024
--jinja
--kv-unified
--no-mmap
--parallel 4
--threads 8
--temp 0.0
--top-k 1
--host 0.0.0.0
--port 3535
main model start:
/home/2/llama.cpp/build/bin/llama-server
--model /home/mir/services/models/Devstral-2-123B-Instruct-2512-Q4_K_M-00001-of-00002.gguf
-fa on
-ctk q8_0
-ctv q8_0
-c 131072
--no-mmap
--parallel 4
--jinja
--threads 16
--spec-type draft-remote
--spec-draft-model-url http://x.x.x.x:3535/v1
--spec-draft-n-max 5
--port 3536
diff.patch
Beta Was this translation helpful? Give feedback.
All reactions