Skip to content

v2.8.0 — Streaming requests, first-byte watchdog, cross-process result cache

Choose a tag to compare

@RunhuaHuang RunhuaHuang released this 14 Aug 12:14
· 28 commits to main since this release

What's Changed

⚡ Streaming requests + first-byte watchdog

  • Provider requests are now sent streamed (stream: true) and aggregated internally into the complete answer — callers see no behavior change, and answers can arrive slightly sooner because the gateway no longer buffers the whole response before sending.
  • A new first-byte watchdog (firstByteTimeoutMs, default 15000 ms, capped at timeoutMs) aborts and retries early when an upstream accepts the request but stalls before emitting its first token (queue congestion, gateway hang) — worst-case waits drop from minutes to seconds instead of idling to the full request timeout.
  • Full gateway compatibility ladder: gateways that ignore the stream flag (plain JSON answers, even SSE-shaped ones) are parsed as before, and gateways that explicitly reject the parameter are retried once non-streamed — for both OpenAI-compatible and Anthropic protocols.
  • Streamed reads enforce the same 5MB raw-body safety cap as buffered reads.

💾 Cross-process result cache

  • The in-memory result cache now has an on-disk mirror (default ~/.visionpower/cache, override with VISIONPOWER_CACHE_DIR), so short-lived processes such as the standalone Skill script reuse a recent identical image+prompt answer instead of billing a second model call — repeated identical requests drop from seconds to milliseconds.
  • Owner-only files (0600), atomic writes, mtime-LRU eviction, same TTL/entry budget as the memory cache; any disk failure or an insecure (group/world-accessible or symlinked) cache directory degrades to a plain cache miss and never fails the image request.

🔢 Fewer truncated answers, faster focused questions

  • Default maxTokens raised 2048 → 4096: long OCR answers no longer get cut off mid-result (which previously forced agents to re-run the whole call). Configs saved by an older WebUI with the historical 2048 default still count as "unset" so provider-recommended budgets (e.g. Kimi's 32768) keep applying.
  • Tool and parameter descriptions, plus the Skill guide, now steer agents toward focused prompts ("Read the error message text") over open-ended "describe everything" requests — output length is the dominant per-call latency factor.

🖥️ WebUI

  • New First-Byte Timeout setting (synced with firstByteTimeoutMs / VISIONPOWER_FIRST_BYTE_TIMEOUT_MS).
  • The result-cache toggle copy now accurately describes the on-disk mirror instead of claiming "no disk usage".
  • GET /api/config no longer exposes the server-side cache directory.

Full Changelog: v2.7.2...v2.8.0