Repository navigation
Replies: 1 comment
|
Hi guys! I am sorry I wrote this like a report or something. Because I've getten the all data and tested needed. So, I just gave the detailed data in order to show the performance. If anybody has problem, feeling free @me to discuss! |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Ouro runs the same layers 4 times per token. I noticed that the next token after only 1 loop is the same as after all 4 loops about 94% of the time. So the model can draft for itself: draft a few tokens with 1 loop, then verify them in one batch with all 4 loops. There is no second model, just one context and one copy of the weights.
The Result was tested on an RTX 4080 Laptop (CUDA), Ouro-2.6B and 2.6B Thinking, on the top of #29823, drafting length for 7 tokens per round.
Greedy has the best performance as shown as:
Details
with physical batch size 1 the output is token-identical to normal decoding. With the default batch size F16 was identical on 50/50 prompts and Q4_K_M on 44/50 (near-ties can go the other way)
But Sampling's performance would be lower due to the temp going up (F16, top-k 40, top-p 0.95, 20 GSM8K prompts):
Thinking Model (Q4_K_M, only small samples of 5 prompts):
The change in the engine is small, about 30 lines: an experimental call in llama-ext.h to run only the first N loops for the next decode, one extra check in graph reuse, and ouro.cpp running only the first N loops. The rest is a standalone example program. The numbers above also include a 1-line fix in the CUDA graph cache, which I would send separately.
Limitations:
This is different from #31732 and #18471. Here the draft comes from the early loops of a looped model.
Something Need Answering:
@BrandeisPatrick if you have time, I would be curious how this does on your M4.
AI usage: the code was written with AI assistance (Claude Fable 5.2). I tested it locally and will review every line before any PR produced.
All reactions