[Bug]: Status-bar tokens/sec is inaccurate for streaming / OpenAI-compatible (local) providers

[Elevatormusic]

Bug Description

Status-bar tokens/sec is inaccurate for streaming / OpenAI-compatible (local) providers

Summary

The status-bar tokens/sec reads well below the model's real generation speed when using a
local / OpenAI-compatible provider (e.g. LM Studio). tokenSpeedLabel correctly prefers a
provider-measured rate, but that value is never available for these providers, so it falls
back to a turn-duration estimate that includes prompt submission, prefill,
time-to-first-token, and tool-call gaps — heavily diluting the decode rate.

Environment

• Hermes desktop 0.18.0 (packaged build, sourceMode: false)
• OS: Windows 11
• Provider: custom OpenAI-compatible endpoint "Local (localhost:1234)" = LM Studio
• Model: qwen3.6-35b-a3b-mtp (35B-A3B MoE + MTP speculative decoding)

Steps to reproduce

1. Add a local OpenAI-compatible provider (LM Studio at http://localhost:1234/v1).
2. Load a model with a non-trivial context; send a turn with a few thousand tokens of prompt
   and/or a tool call or two.
3. Compare the status-bar tokens/sec against the model's true generation speed.

Expected vs actual

• Expected: ≈ the model's decode rate. LM Studio's server log reports
  eval time … N tokens ( X tokens per second ) at 76–123 tok/s for this model.
• Actual: the status bar shows materially less, and swings with prompt size / tool usage
  rather than tracking the real decode rate.

Root cause

apps/desktop/src/app/shell/statusbar-controls.tsx

Call site (~line 465):
const speed = tokenSpeedLabel(usage.output, turnStartedAt, now, usage.tokens_per_second)

tokenSpeedLabel (~line 242) prefers measured (4th arg); otherwise
output / max(1, (now - since)/1000) with since = turnStartedAt.

Two facts make the fallback fire and be wrong:

1. usage.tokens_per_second is never populated for OpenAI-compatible / local providers.
   Verified against LM Studio /v1/chat/completions: response has usage
   {prompt_tokens, completion_tokens, total_tokens} and a stats object holding only
   draft-token counts (total/accepted/rejected_draft_tokens_count) — no tokens_per_second.
   So it always falls through to the self-timed branch.
2. The fallback anchors to turnStartedAt (stamped at turn start in
   use-message-stream/gateway-event.ts, setTurnStartedAt(Date.now())). That window includes
   prompt submission + prefill + time-to-first-token + tool-call idle. There is no first-token
   timestamp anywhere in the session code, so decode-only timing isn't possible today.

For a local model this is large: a ~14k-token prompt prefills at ~2000 tok/s (~7 s) before the
first output token, and that 7 s is folded into now − turnStartedAt, dragging the shown rate
far below the true ~80–120 tok/s decode speed.

Suggested fix (provider-agnostic, preferred)

Measure decode-only speed:

1. In the streaming handler, record firstTokenAt = Date.now() on the first assistant
   content delta of the turn (once per turn).
2. In the fallback, divide by the decode window:
   const seconds = Math.max(0.001, (now - firstTokenAt) / 1000)
   const speed = output / seconds
3. Optional: subtract tool-call idle spans so multi-tool turns don't dilute the rate.

No change to tokenSpeedLabel's display logic — only pass firstTokenAt as since instead of
turnStartedAt.

Secondary (when a provider exposes a real rate)

Keep the existing measured-first path. For LM Studio the rate isn't in the OpenAI /v1 body
but is available via its native REST API (/api/v0/...) and server logs; wiring that is optional.

Note

tokenSpeedLabel's "prefer measured, else compute" structure is correct and should stay. The
defect is purely (a) the fallback's timing anchor and (b) the missing first-token timestamp.

Steps to Reproduce

Steps to reproduce

1. Add a local OpenAI-compatible provider (LM Studio at http://localhost:1234/v1).
2. Load a model with a non-trivial context; send a turn with a few thousand tokens of prompt
   and/or a tool call or two.
3. Compare the status-bar tokens/sec against the model's true generation speed.

Expected Behavior

Expected vs actual

• Expected: ≈ the model's decode rate. LM Studio's server log reports
  eval time … N tokens ( X tokens per second ) at 76–123 tok/s for this model.

Actual Behavior

• Actual: the status bar shows materially less, and swings with prompt size / tool usage
  rather than tracking the real decode rate.

Affected Component

CLI (interactive chat), Agent Core (conversation loop, context compression, memory)

Messaging Platform (if gateway-related)

N/A (CLI only)

Debug Report

n/a

Operating System

windows 11

Python Version

3.14

Hermes Version

0.18.0

Additional Logs / Traceback (optional)

Root Cause Analysis (optional)

Root cause

apps/desktop/src/app/shell/statusbar-controls.tsx

Call site (~line 465):
const speed = tokenSpeedLabel(usage.output, turnStartedAt, now, usage.tokens_per_second)

tokenSpeedLabel (~line 242) prefers measured (4th arg); otherwise
output / max(1, (now - since)/1000) with since = turnStartedAt.

Proposed Fix (optional)

Suggested fix (provider-agnostic, preferred)

Measure decode-only speed:

1. In the streaming handler, record firstTokenAt = Date.now() on the first assistant
   content delta of the turn (once per turn).
2. In the fallback, divide by the decode window:
   const seconds = Math.max(0.001, (now - firstTokenAt) / 1000)
   const speed = output / seconds
3. Optional: subtract tool-call idle spans so multi-tool turns don't dilute the rate.

Are you willing to submit a PR for this?

• I'd like to fix this myself and submit a PR

[Elevatormusic]

Closing — after digging further, this report doesn't apply to stock Hermes. The quoted code (tokenSpeedLabel, the usage.tokens_per_second call site in statusbar-controls.tsx) comes from a status-bar customization pack installed on my build, not from upstream: stock apps/desktop/src/app/shell/statusbar-controls.tsx has no tokens/sec readout at all, so there's nothing upstream to be inaccurate.

The analysis itself held up — the fallback was dividing output tokens by the whole turn window (prompt upload + prefill + TTFT + tool gaps), which is why it under-read a local model's decode rate — and it's now fixed on the pack side by anchoring the rate to the first output token of the turn (decode-only window): Elevatormusic/hermes-classic-gold-pack@1223344.

Apologies for the noise; the underlying facts that ARE upstream (no first-token timestamp exists in the session stream, and OpenAI-compatible providers report no measured tok/s) might still interest anyone building a rate display, but there's no stock bug here.