Can someone explain what are the meanings of these timings

[Tommy787576]

What does load / sample / prompt eval / eval / total time mean? What does 0.58 ms per run mean? Why can't I see the timings report like what readme shows?

This is much clearer! What is the mapping between them? I want to know the predict: token per ms.

[akx]

The timing print format was changed in #370, if that helps.

[Tommy787576]

Thank you

[IdoAmit198]

I tried to follow the Change mentioned above and couldn't find the answer there..
If any of you can explain how to understand the time per token from the new print format, I'd really appreciate it.

[SlyEcho]

llama_print_timings:        load time =   134.84 ms
llama_print_timings:      sample time =    12.25 ms /    32 runs   (    0.38 ms per token)
llama_print_timings: prompt eval time =   925.78 ms /    24 tokens (   38.57 ms per token)
llama_print_timings:        eval time =  4178.32 ms /    31 runs   (  134.78 ms per token)
llama_print_timings:       total time =  5211.06 ms

• load time: loading model file
• sample time: generating tokens from the prompt/file choosing the next likely token.
• prompt eval time: how long it took to process the prompt/file by LLaMa before generating new text.
• eval time: how long it took to generate the output (until [end of text] or the user set limit).
• total: all together

When it says 38.57 ms per token, it means it evaluated 24 tokens in 925.78 ms. Lower is better.

Another common metric is tokens per second which is used in other ML projects than llama.cpp. You can get that by calculating 1000/38.57 = 25.93 t/s. Higher is better.

Why is eval time slower than prompt eval time?

Because we can only predict the single next token, that means that after the sampler chooses the next token the model has to be run again but with a batch size of 1. In the prompt eval phase, the model can evaluate large batches (512 max) meaning less overhead and more efficiency, especially with BLAS or GPU.

EDIT: "sample time" is not tokenization time, it is actually the "sampling" time, that means running the RNG, sorting and filtering candidates, etc.

[IdoAmit198]

@SlyEcho Thanks a lot, it is very informative :)

[isaac-chung]

Summing all timings together gave 5251.19ms instead of the printed 5211.06ms. I consistently see this in my trials as well, where the actual summed time > printed total time.
Any idea where the difference is coming from?

[JayKayNJIT]

same here, I am manually summing all the times and they never matches bottom line, any idea??

[joyyang1215]

Many thanks