Skip to content

Anubis OSS v3.5 — N-rep groups with confidence intervals, LM Studio fixes, accurate SoC power

Choose a tag to compare

@uncSoft uncSoft released this 23 May 00:00
· 91 commits to main since this release

A focused follow-up to v3.3 — this release combines what was staged as v3.4 (more accurate SoC power accounting + backend persistence) with substantial new work: N-rep benchmark groups with bootstrap confidence intervals, comprehensive LM Studio reliability fixes, and a server-side schema for group-aware leaderboard submissions.

New: N-rep benchmark groups

  • Repetitions stepper (1–20) in the benchmark toolbar. A group runs N reps of the same configuration sequentially. On completion the dashboard surfaces mean ± 95% bootstrap CI (1000 resamples) for tok/s, TTFT, J/Tok, system power, GPU power, and peak memory.
  • Seed strategy picker. Random (default) picks a fresh seed per rep — CI then captures both hardware AND sampler variance, the honest measurement of what an end user actually experiences. Fixed uses the same seed across reps — CI captures hardware variance only, useful for reproducibility runs.
  • Group view-mode toggle. Dashboard cards switch between "Full group" (mean across all reps) and "Last rep" with one click.
  • Per-rep streaming and per-rep charts preserved. Every rep collects full per-sample data; the group aggregate is computed at completion.

New: Group context attached to leaderboard submissions

  • Submissions from a group rep now carry their group's run_group_id, sample count, repetition index, seed strategy, and mean + 95% CI for the headline metrics (tok/s, TTFT, J/Tok).
  • Server schema (leaderboard_submissions v7) updated to persist all of this. The current leaderboard page still renders per-rep rows; the underlying data now supports a future "mean ± CI" view without re-uploading anything.
  • Also fixes a long-standing issue where methodology_version (added in code at v3.4) was being silently dropped by the server. Pre-v3.5 submissions stay NULL for that column; v3.5+ correctly tags as methodology 2.

Fixed: More accurate SoC power accounting

The CPU, ANE, and DRAM power channels reported by IOReport use different units (mJ) than the GPU channel (nJ). The prior implementation treated all four uniformly, which meant the CPU/ANE/DRAM contributions were effectively rounded out of `system_power` — leaving it dominated by GPU. The headline `J/Tok` metric was a corresponding undercount.

Per-channel scaling is now correct, so `system_power` reflects the whole SoC, and `J/Tok` reflects true SoC energy per token. Migration v7 in the local app DB retroactively rescales the affected columns in your historical sessions so older reports re-aggregate cleanly. The methodology version tag bumps from 1 → 2, and the leaderboard server now records this so cross-version comparisons stay honest — pre-v3.5 submissions stay tagged methodology 1 (or `NULL` for pre-v3.4); v3.5+ submissions are methodology 2.

Fixed: LM Studio reliability

  • Alternating-failure pattern in N>1 groups. Every other rep was failing in ~100 ms with NULL tokens. Root cause: the `Connection: close` header (added for Ollama chunk pacing) caused LM Studio to reject the immediately-following request while still in TCP teardown. Header is now Ollama-only; OpenAI-compat backends keep the connection alive. 250 ms inter-rep delay added in run groups as belt-and-suspenders. Failed sessions also now record the error description in their `response` field for easy diagnosis.
  • Process auto-detection grabbed the wrong worker on the first run. The actual model holder lives at `~/.lmstudio/.internal/utils/node`, outside the `.app` bundle and sometimes reparented to launchd — and on first run the model isn't loaded yet so the worker has tiny memory. Detection now recognizes that path AND a self-healing soft pin re-evaluates every 2 s, converging to the real worker within ~2 s of model load completing. No more "first rep wrong, second rep right."
  • Mis-attribution to Ollama after switching LM Studio models. When LM Studio's port briefly stopped listening, the metrics loop silently fell through to a priority default that picked Ollama if it happened to be running. Closed that fall-through.
  • tok/s = 0 for reasoning-only LM Studio output — falls back to total throughput.
  • mlx-server-style buffered streams reporting 10,000+ tok/s — warning triangle + tooltip when timing is suspicious instead of inflating the headline.

Fixed: M5 Max GPU frequency

Pinned at 1084 MHz on M5 Max due to inverted Hz/KHz/MHz scale heuristic. Now picks unit scale from the table maximum + saturation fallback; reports the full clock range.

Fixed: OpenAI-compat sessions reporting promptTokens = 0

Added `stream_options.include_usage: true` to force usage in the final SSE chunk.

Streaming + performance

  • Streaming text view migrated from TextKit 1 to TextKit 2. Fixes a long-running hang after ~25 reps caused by `CopyOfFontWithLigatureSetting` getting called on every layout pass. Font is now pre-configured with ligatures disabled at the CTFont level.
  • Stream consume task promoted to `.userInteractive` priority. Stop button responds immediately; group-mode chunks stream evenly instead of arriving in bursts.
  • Group streaming cascade storm fixed. Six `@Published` cascades demoted to plain vars and batched at 1 Hz during multi-rep runs. Sidebar toggles take <50 ms during runs (was up to 1 s).

Polish

  • Process picker enhancements. Live CPU% updates (1.2 s refresh) so the active inference worker stands out. Interpreter processes (node, python) are labelled by script — e.g. `node (llmworker.js) — 1.2 GB · 69% CPU` — so you don't have to guess between five identical "node" rows.
  • Last-used backend and model persist across launches. Ollama is no longer the always-default; first launch shows what's available, subsequent launches restore your last selection.
  • Friendly error for non-thinking models. When a model rejects Ollama's `think` parameter, the error message now explains the fix instead of a generic 400.