Repository navigation
v2.8.0 — Streaming requests, first-byte watchdog, cross-process result cache
What's Changed
⚡ Streaming requests + first-byte watchdog
- Provider requests are now sent streamed (
stream: true) and aggregated internally into the complete answer — callers see no behavior change, and answers can arrive slightly sooner because the gateway no longer buffers the whole response before sending. - A new first-byte watchdog (
firstByteTimeoutMs, default 15000 ms, capped attimeoutMs) aborts and retries early when an upstream accepts the request but stalls before emitting its first token (queue congestion, gateway hang) — worst-case waits drop from minutes to seconds instead of idling to the full request timeout. - Full gateway compatibility ladder: gateways that ignore the
streamflag (plain JSON answers, even SSE-shaped ones) are parsed as before, and gateways that explicitly reject the parameter are retried once non-streamed — for both OpenAI-compatible and Anthropic protocols. - Streamed reads enforce the same 5MB raw-body safety cap as buffered reads.
💾 Cross-process result cache
- The in-memory result cache now has an on-disk mirror (default
~/.visionpower/cache, override withVISIONPOWER_CACHE_DIR), so short-lived processes such as the standalone Skill script reuse a recent identical image+prompt answer instead of billing a second model call — repeated identical requests drop from seconds to milliseconds. - Owner-only files (0600), atomic writes, mtime-LRU eviction, same TTL/entry budget as the memory cache; any disk failure or an insecure (group/world-accessible or symlinked) cache directory degrades to a plain cache miss and never fails the image request.
🔢 Fewer truncated answers, faster focused questions
- Default
maxTokensraised 2048 → 4096: long OCR answers no longer get cut off mid-result (which previously forced agents to re-run the whole call). Configs saved by an older WebUI with the historical 2048 default still count as "unset" so provider-recommended budgets (e.g. Kimi's 32768) keep applying. - Tool and parameter descriptions, plus the Skill guide, now steer agents toward focused prompts ("Read the error message text") over open-ended "describe everything" requests — output length is the dominant per-call latency factor.
🖥️ WebUI
- New First-Byte Timeout setting (synced with
firstByteTimeoutMs/VISIONPOWER_FIRST_BYTE_TIMEOUT_MS). - The result-cache toggle copy now accurately describes the on-disk mirror instead of claiming "no disk usage".
GET /api/configno longer exposes the server-side cache directory.
Full Changelog: v2.7.2...v2.8.0