How to get prompt & generation time from api & cli replies?

[nnbyte]

As in the LocalAI webUI, after each (e.g openai) api or cli response, together with the tokens, I would expect to also receive the time it took for the prompt/generation. So, I could eventually calculate the tokens/second.

If I understand it correctly, In the code this is mentioned as:
tokenUsage.TimingTokenGeneration = reply.TimingTokenGeneration
tokenUsage.TimingPromptProcessing = reply.TimingPromptProcessing

Do I need to enable a flag? I am not seeing this anywhere!

[gitsult4n]

It is a request header, not a config flag. Send Extra-Usage with any non-empty value and the two timing fields appear inside usage.

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Extra-Usage: true" \
  -d '{"model":"your-model","messages":[{"role":"user","content":"hi"}]}'

"usage": {
  "prompt_tokens": 12,
  "completion_tokens": 40,
  "total_tokens": 52,
  "timing_prompt_processing": 84.2,
  "timing_token_generation": 512.7
}

The gate is extraUsage := c.Request().Header.Get("Extra-Usage") != "" at core/http/endpoints/openai/chat.go:143, and the same line exists in completion.go:90 and edit.go:36. Without it the fields are dropped, because they carry omitempty and are left at zero, core/schema/openai.go:36:

// Extra timing data, disabled by default as is't not a part of OpenAI specification
TimingPromptProcessing float64 `json:"timing_prompt_processing,omitempty"`
TimingTokenGeneration  float64 `json:"timing_token_generation,omitempty"`

Both are milliseconds, straight from llama.cpp's own timings, backend/cpp/llama-cpp/grpc-server.cpp:2214 reads timings.predicted_ms into timing_token_generation and the prompt one alongside it. So tokens per second is completion_tokens / (timing_token_generation / 1000), and prompt throughput is prompt_tokens / (timing_prompt_processing / 1000).

For streaming you need both the header and an opt in to the usage chunk, since usage only rides in the trailer, chat.go:674:

{"model":"...","stream":true,"stream_options":{"include_usage":true},"messages":[...]}

It is documented, just easy to miss, docs/content/advanced/advanced-usage.md:227 under Request headers.

Read at commit 0a89fdb. I did not run a model to capture a live response, the header plumbing above is from source and the docs.

[nnbyte]

Thanks, I found out the hard way (browsing though core/http/endpoints/openai/chat.go:143), but thanks a lot for the clarifications!

And I can confirm it works!