2.2.2

Changelog

[2.2.2] - 2026-10-12

Fixed

• One /api/engines request per node, not two. The launch form's engine layer cached only resolved responses, so concurrent callers never shared a request: updateEngineSelect() fired updateStrataModelsButton() without awaiting it and then fetched the same node itself, hitting /api/engines twice per engine change (and again at page load); the 15 s TTL can't help a race. fetchEnginesForNode now caches the in-flight promise per node and hands it to callers that arrive while a request is pending, cleared on settle (success or failure); force still bypasses both maps. #107
• Strata-models button only on the Strata engine. The button beside the Inference Engine dropdown showed whenever the target node was Strata-ready, even with llama.cpp selected. It now shows only while Strata is the selected engine and the node is Strata-ready, matching how the dropdown gates its Strata option. #107

2.2.1

[2.2.1] - 2026-10-11

Added

• 🆕 Decision models (System One) — decision models answer typed questions about an input with probabilities instead of generating text: the System One API introduced by TypeSafe's Jev, served by llama.cpp (images built after 2026-10-05) on POST /v1/systemone. Open GGUFs: Julia-1, Laya, Kev-4B, lev, Clef-Flash, Clef, OpenJev (OpenJev and Clef also take images, with their --mmproj). llama-server recognises them from <arch>.decision.type in the GGUF; no flag is needed. #104 #105
  • Routing. POST :42069/v1/systemone picks the model by "model" like /v1/chat/completions (auto-start, eviction rules, cluster forwarding, queue). The request is forwarded unchanged — no sampling overrides. On an instance's own port it goes through the per-instance proxy's queue when that proxy runs. Chat endpoints refuse decision models with HTTP 422 — they don't generate text.
  • Detection + launch presets. Picking one in the launch form says so under Model Path and fills the form without a preset: Exclude from Max Models on, Web UI off (its chat page can't drive the model), Parallel Slots 8 (each question takes a slot, sharing the input), Embedding Model off (llama.cpp turns embedding mode on itself where needed). An encoder (BERT-style) model also gets Context, Batch and Micro-batch Size set to its context length — its whole input must fit in one micro-batch. Clef (and Clef-Flash) asks all questions in one prompt and one pass, so it gets Parallel Slots 1 and Batch = Micro-batch = Context (16384, its trained default, when Context is blank).
  • Batch knobs. Batch Size (--batch-size, default 2048) and Micro-batch Size (--ubatch-size, default 512) added to the launch form; blank omits the flag. A question whose options don't fit in one batch is rejected by llama-server ("increase the physical batch size") — raise these.
  • Request log. System One answers are recorded, with input_tokens as the prompt tokens (output_tokens is always 0).
• Exclude from Max Models. New launch-form toggle: instances with it on are not counted toward LLAMAMAN_MAX_MODELS and are never evicted — embedding models, decision models, anything small that should stay up. Split out from Embedding Model, which now only sets the embedding flag; turning Embedding Model on still turns this one on, and presets saved before the toggle existed inherit it from their Embedding Model setting. Live-updated from preset (no relaunch). #105
• Clear Preset button. In the launch form (before Export): resets the form to the selected model's defaults — recommended decision-model settings included — without saving.

Fixed

• TTFT and tokens/s stamped on the first generated token. The request log treated the first bytes of a stream as the first token. Strata sends an empty role chunk and keep-alive comments while it reads the prompt, so its TTFT read ~0 and tokens/s was divided over prompt reading plus generation. SSEAccumulator.feed now reports whether a chunk carried generated output, and the first-token stamp waits for one. #103

2.2.0

[2.2.0] - 2026-10-09

Added

• 🆕 Strata engine (experimental, opt-in) — Strata as a second inference engine behind a new engine layer (core/engines/). Opt-in via STRATA_ENABLED=true + NVIDIA GPU; off by default, llama.cpp behaviour snapshot-tested byte-for-byte. Engine abstraction + registry; config["engine"] is additive — missing key = llama.cpp, no storage migration. Family × size catalogue (8 virtual models, /strata/<family>-<SIZE>), env-var-only launch, pinned-revision shard pre-download through llamaMan's downloader, load-stage parsing from container logs, settings-fingerprint-driven REINSTALL=1, per-model flags from Strata's own setup.py, Hang Watchdog + Slow-Storage Allowance. Images: Strata repo download + docker build from the UI (built for this node's GPU generations), pull-by-name for both engines, deletion of built images. ⚠️ Builds default to STRATA_REPO_REF=main (unpinned third-party source); pin a commit for reproducibility. UI: engine picker, Strata launch fields beside their llama.cpp counterparts, recommended-models modal, quant-name aliases. Docs: README section + docs/STRATA.md. #97
• llama.cpp Web UI toggle (default on; off passes --no-webui) and per-instance "Open llama-server" links. #97
• Model Archive (opt-in) — mount a second volume, set ARCHIVE_DIR=/archive; unset hides the feature entirely. Archive/Restore buttons in the model list; archived models stay listed, dimmed, not launchable, absent from /api/tags / /v1/models. Moves run as cancellable jobs in the Downloads panel; cross-disk moves are atomic (hidden partial + fsync + size check), cancel/failure/low-space leave the source untouched, partials cleaned at startup. Refused while the model is in use (incl. sleeping, draft, mmproj, Strata shards), mid-download, or on name collision. Restore returns files to the original path, so presets/favorites/notes/display names survive. #98

Fixed

• Download speed limit is now per node. "Global speed limit (Mbps)" lived in the shared settings blob, so one node's value capped every node. Now per-node (core/node_settings.py), falling back to the legacy top-level value — existing installs keep their limit until each node saves its own. #99

2.1.0

[2.1.0] - 2026-09-21

Added

• 🆕🆕🆕 Knowledge base + MCP server (opt-in, MariaDB 11.8+ only) 🆕🆕🆕 — store documents grouped by topic; llamaman chunks them (~1200 chars, paragraph-aligned, 200 overlap), embeds them through a llama.cpp instance launched with Embedding model enabled, and stores the vectors in a MariaDB VECTOR column with a DISTANCE=cosine index. One code path, three ways in:

  • Settings → MCP tab — enable toggles, embedding-model picker, topic list (filter by name, delete), test search, Re-embed everything, and (with Expose over MCP on) a Copy config.json button that copies a ready-to-paste mcpServers block for this endpoint, including an Authorization: Bearer <your-api-key> placeholder when Require Authentication is on.
  • REST under /api/kb/* — status, settings, topics (optionally created as a given API key's private topic), documents (list/get/add/delete), search, re-embed + status. Admin-only: UI sessions and cluster peers are accepted, requests authenticated with an API key get 403 (keys use the MCP endpoint).
  • MCP endpoint at POST /mcp/knowledge — JSON-RPC 2.0, protocol revisions 2025-06-18 / 2025-03-26 / 2024-11-05. Read-only tools: kb_list_topics, kb_list_documents, kb_search, kb_get_document. Mutating tools, each behind its own setting (off by default) and hidden from tools/list while off: kb_ingest_document (creates the topic if missing; 200 KB per call; one ingest at a time, others get "ingest busy, retry shortly") and kb_delete_document / kb_delete_topic (topic delete cascades to its documents; irreversible).

  How it behaves:

  • Adding documents goes through POST /api/kb/documents or the MCP kb_ingest_document tool — the MCP tab lists and deletes topics but has no add form. A document is keyed by topic + title; resubmitting identical content (sha256) is a no-op that skips embedding.
  • Search embeds the query with the same model and returns matched passages, best first, with a score clamped to 0–1 (max(0, 1 - cosine_distance/2)). Searching before anything has been embedded returns "knowledge base is empty".
  • Embedding model is chosen by model name, not instance: the dropdown lists every embedding-enabled instance across the cluster, deduplicated by name. An embed call uses a healthy matching instance on this node; failing that, wakes a sleeping/stopped one on this node, or waits for one that is already starting; failing that, forwards to a peer that has both Knowledge base and Expose over MCP turned on and has the model running there (or asleep, stopped or starting there, in which case that peer wakes it or waits for it); failing that, relaunches the model on the node that last served it (at most 3 attempts per model per 10 minutes — once that budget is spent and another attempt fails, the remembered node is forgotten). Otherwise it fails with "no live embedding instance".
  • Timeouts: embed calls use a 5 s connect / 30 s read timeout (60 s read for peer-forwarded calls), so a hung embedding server produces an error rather than a request that never returns. Local embed calls are recorded in the request log (kb-embed); peer-forwarded ones are logged on the peer.
  • Dimension lock: the first embed records the vector dimension. An embed or search that produces a different dimension is refused with a prompt to run Re-embed everything. Only the dimension is checked — switching to a different model with the same dimension is not detected, so re-embed manually when you change models.
  • Re-embed clears all chunks and re-embeds every document in a background job with progress reporting. One job per process; the table drop/recreate on a dimension change is guarded by a cluster-wide MariaDB GET_LOCK. Re-embedding an empty KB leaves the dimension lock untouched.

  Access modes (kb_mcp_access, MCP access in the tab):

  • Global (default) — every MCP client works on the shared pool: it sees all shared topics and, if ingest/delete are allowed, can change them. New topics go into the shared pool. Keys' private topics (created in Per API key mode) are not visible in Global mode.
  • Per API key — every MCP request needs an API key, even with Require Authentication off. Topics a key creates are private to that key. Each key also sees the shared pool (topics created in Global mode or through the UI/REST) read-only: it can search and read it, but ingest or delete there is refused. Another key's topics and documents come back as "not found", and responses carry a shared flag instead of key IDs. Switching modes never moves or deletes data. After switching Global → Per API key, everything already in the knowledge base stays visible to every key as the shared pool. After switching Per API key → Global, private topics stay private (hidden from all clients) and reappear for their keys when you switch back; the admin UI always shows everything. The MCP tab labels each topic shared or with its owning key's name.

  Settings (seven keys): kb_enabled, kb_mcp_enabled, kb_mcp_access (global | per_key), kb_mcp_ingest, kb_mcp_delete, kb_embedding_model, and kb_embedding_instance (selects an instance by ID on the local node; only used when no model name is set).

  Auth: in Global mode /mcp/knowledge follows the app-wide Require Authentication setting — on means a valid API-key bearer is required, off means the endpoint is open. In Per API key mode a valid key is always required, and cluster peers (which carry no key) are refused. API keys are still not restricted to the knowledge base: a key that can use MCP can also call the inference endpoints. The MCP tab shows the endpoint URL on port 42069 (the route is registered on the app, so it also answers on 5000).

  Off means off: with the feature disabled, /mcp/knowledge returns 404 in both access modes, and a strict-backend test verifies no request path touches any kb_* storage method. There is no migration: the kb_* tables are created lazily on first use and only on MariaDB. On the JSON backend or MariaDB older than 11.8 the tab shows a banner saying why, and enabling is refused. KB data is cluster-wide (not node-scoped); each node's heartbeat advertises its KB/MCP flags. #96

2.0.2

[2.0.2] - 2026-09-15

Added

• Queue sharing info-tip. The Share-queue toggle is renamed Queue sharing (label + aria-label; the share_queue field key is unchanged) and gains an info-tip matching the Fallback-only pattern: it shares one queue across all cluster instances running the same model by filename, and toggling on unlocks Queue group name. Stale references to the old label updated in comments across boot.js, cluster.py, index.html, style.css. #94

Changed

• MoE CPU Layers label trimmed. (0 off, -1 all, N = first N) → (0 off, -1 all, first N) — parallel clauses, no stray =; the adjacent info-tip already carries the full sentinel semantics. #92
• Queue group name tooltip wording. "…even if they're running different quants or files" → "…different quants or models" — pooled members may be entirely different models under one alias, which "files" muddied. #95
• Display Name label. Dropped the "(optional)" suffix on the Launch Instance field; the info-tip already says "Leave blank to use the filename".

Fixed

• Turning off Image & PDF Input no longer blocks the save. Image & PDF Input is the master gate for the whole group, but toggling it off only collapsed the reveal — Accept PDF uploads stayed checked underneath, hidden, and the next save tripped the pdf_input_enabled requires mmproj_enabled guard with no visible cause. updateMmprojState() now clears the two PDF sub-toggles on master-off, the same reveal-and-clear contract the Queue sharing toggle uses for its sub-fields. UI-only: the server-side 400 in parse_mmproj_config stays as the direct-API contract, pdf_dpi / pdf_max_pages keep their values across a toggle cycle, and saved presets are untouched until you deliberately save with the master off. #93

1.9.1

[1.9.1] - 2026-09-04

Added

• Async instance stop. DELETE /api/instances/<id> returns 202 immediately; a transient stopping state guards docker's ~10s SIGTERM grace. Orange "Stopping…" badge in the UI. Eviction→relaunch stays sync. 7 tests.
• Instant cluster visibility. Launch/stop/restart publish a heartbeat inline; peer cards refresh in one round-trip instead of ~10s.
• Load Mode dropdown. --load-mode (auto/none/mmap/mlock/mmap+mlock/dio), llama.cpp's replacement for --mlock/--mmap/--direct-io. Bad values fold to auto. 20 tests.
• Opt-in perf timing. LLAMAMAN_PERF_LOG=1 logs perf <name> <ms> for auth, heartbeat, proxy forwards, save_state, docker ops, idle-proxy shutdown. No-op when off.

Changed

• Idle reaper stops asynchronously — no longer holds the 5s poller tick through a SIGTERM grace.

Fixed

• Queue stats + sidecar proxy lost across a llamaman restart. Reattached-running instances came back without the "Queue N/M active · K queued" indicator and with their public port unbound. load_state's rebuild block was gated on sleeping only; now covers starting too. 8 tests.
• Image & PDF Input layout. Section stretched end-to-end on desktop and overflowed on tablets — hint <p>s inside a max-content toggle grid blew the columns out to their one-line width. Split into two mini-grids with hints as block siblings; a .mmproj-toggle-table modifier pins label cols so the Offload / Accept-PDF switches align. Phone 1fr 40px still wins ≤480px.
• Nested-repo model update check. 404'd on HuggingFace files in subfolders (every unsloth UD-* quant + multipart siblings) because the check probed resolve/main/<basename> while the subfolder was discarded. A 404 now retries once through resolve_filename; the apply-swap lands by basename instead of double-nesting. Flat root-level models unchanged. 15 tests.

1.8.5

[1.8.5] - 2026-08-28

Added

• Anti-Loop: two-tier defense against stuck model output. Real models still get stuck in stable output attractors — repeating a sentence, a paragraph, or a <think> fragment for hundreds of tokens — even with --repeat-penalty and DRY tuned. This release ships a two-tier defense that operators can turn on per model preset from a new Anti-Loop section in the launch form. Off by default because both tiers change generation behavior and detection thresholds have to be tuned per model class; existing launches produce byte-identical CLI to before. Every hook in the streaming path is wrapped in try/except Exception that logs and falls through, so a detector bug can never break a real user's stream.

  • DRY sampler (soft, sampling-time). Exposes llama.cpp's --dry-multiplier / --dry-base / --dry-allowed-length / --dry-penalty-last-n as UI fields. dry_enabled: true AND dry_multiplier > 0 are BOTH required to emit anything — llama.cpp treats multiplier=0 as disabled, so emitting the four flags with a zero multiplier would be noisy no-ops. dry_base < 1.0 is rejected at the preset boundary (llama.cpp silently ignores those, which would surface as a mystifying no-op at runtime instead of a clear error). dry_penalty_last_n empty means "flag omitted, llama.cpp default (ctx-size) applies"; 0 explicitly disables the penalty window per llama.cpp's docs; negative values are rejected because llama.cpp's arg parser throws on them and the container would die at startup with an opaque error. All four are baked into the container at launch, so preset edits take effect on relaunch.
  • Output Loop Detection (hard, post-hoc). llamaman's proxy watches the model's streamed assistant-visible output (both content AND reasoning_content — thinking-phase loops matter as much as content-phase) and, if a large chunk of text repeats often enough, closes the upstream connection (which stops llama-server generating) and injects a synthetic terminator into the client stream — OpenAI clients see a finish_reason=stop chunk with a [llamaman: output loop detected, ending turn] marker; Ollama clients see done_reason: "loop_detected" on a done: true line. Detection algorithm is deliberately simple: take the last min_chunk_chars of the buffer, count exact occurrences, flag when ≥ min_repetitions. Catches every runaway loop I've seen in the wild without false-positiving on code with for-loops / poetry choruses / markdown tables at the default 200-char × 3-rep thresholds. Doesn't catch drifting near-repeats ("The answer is: The answer is." with slight punctuation drift) — those get caught by DRY at sampling time. The five thresholds (min_chunk_chars, min_repetitions, max_buffer_chars, scan_every_n_tokens, scan_interval_s) are all preset-live: the next request picks up new thresholds without a relaunch (in-flight streams keep the thresholds captured at attach time).
  • Two-trigger scan cadence. Inline scans run every N chunks (default 64, amortized to hundreds-of-ms detection latency at typical generation speeds); a background worker on the existing 5s poller catches slow streams that emit tokens too slowly to hit the inline threshold (default scan_interval_s = 10). The worker is a cheap no-op when the registry is empty — no dedicated thread, and the check is a dict-emptiness test.
  • Where it watches. Compat routes on :42069 (Ollama /api/chat, /api/generate; OpenAI /v1/chat/completions, /v1/completions) — always, whenever the toggle is on for the target instance's preset, because llamaman is doing the streaming in-process either way. Per-instance ports (8000-8020) — only when a sidecar proxy is already present for the instance (Idle Timeout, Max Concurrent, or Proxy Sampling Overrides also on). Loop detection does not, by itself, spawn a sidecar proxy; the UI hint text points this out so operators know to combine with one of the other three if they want direct-port coverage. This is a deliberate design choice — the entire purpose of llamaman is that you route through its proxy; a direct-hits-only path isn't a supported topology to bolt features onto.
  • Per-request buffering. With max_concurrent > 1 an instance has N turns in flight, so the registry is dict[stream_id → TurnBuffer], not dict[inst_id → ...]. Each TurnBuffer holds a rolling list[str] capped at max_buffer_chars (default 8192, hard cap 64 KB), plus its captured thresholds and a threading.Event for the kill signal. Buffer growth is O(1) on feed (append, cap-check via truncation to the last N chars); scan runs outside the lock so a slow scan can't stall the feed side.
  • Boundary validation. parse_dry_config and parse_loop_detect_config in core/dry_sampling.py / core/loop_detect.py are the single source of truth — every launch route, preset save, and merge whitelist routes through them. Every threshold has floor/ceiling bounds and a "max_buffer_chars ≥ min_chunk_chars × min_repetitions" cross-check that rejects configs where detection would be silently impossible.
  • Coverage. 143 new tests across tests/test_dry_sampler.py, tests/test_loop_detect.py, tests/test_loop_detect_proxy.py, tests/test_loop_detect_compat.py: parser bounds, algorithm correctness on realistic content (code with for-loops, poetry choruses, markdown tables, numbered lists, drifting near-repeats), rolling-cap memory safety, thread-safety under 8-worker concurrent feed, SSE extractor edge cases ([DONE], mid-chunk boundaries, reasoning_content, llama.cpp native format), defensive "broken scanner must not break the stream" invariant, sidecar proxy fork end-to-end, and both Ollama-native + OpenAI compat generators.

• Reasoning Format dropdown. New Model Settings → Reasoning Format select maps to llama-server's --reasoning-format (values: auto / none / deepseek / deepseek-legacy, default auto). Controls how model thought tags are parsed out of streamed output — routed to reasoning_content vs left inline in content — which matters for DeepSeek-R1, Qwen-thinking, and any model that emits reasoning tokens. auto omits the flag entirely so llama.cpp's own default applies; the three explicit values are emitted verbatim, and unknown / hand-crafted values fold to auto at the boundary in normalize_reasoning_format() so a corrupt preset can never make llama-server refuse to start. Shared preset field (not per-node hardware), same tier as flash_attn and cache types. 17 tests in tests/test_reasoning_format.py.

• CPU Threads Batch field. New Container & Proxy Settings → CPU Threads Batch number field maps to llama-server's --threads-batch. Separate thread pool for batch / prompt processing (the prefill / evaluation phase), distinct from the generation threads set by CPU Threads. Useful on CPUs with a big/little split (e.g. Intel P+E cores) where prefill benefits from more threads than steady-state generation, or when you want prefill to use every core but keep generation on the performance cores only. Blank = flag omitted, llama-server falls back to the --threads value on its own — we never re-emit that value server-side. Per-node hardware key (CPU core counts vary per node), so it participates in the cluster preset's node_overrides block same as threads. 14 tests in tests/test_threads_batch.py.

• Auto-clipping for info-tip tooltips extended to respect the model library sidebar. The existing updateInfoTipClipping() fallback in static/js/boot.js only measured tooltip overflow against the viewport edges (0 … window.innerWidth), so a tooltip on a leftmost field of the launch panel — inside the viewport but partially overlapping the 260px sidebar's visual territory — stayed centered and rendered on top of the model rows. The updated function measures .sidebar via getBoundingClientRect() and, when the info-tip icon is to the right of the sidebar, uses the sidebar's right edge as the effective left-overflow threshold. When that check overflows, the existing clip-left class kicks in and the tooltip anchors its left edge at the icon, extending only rightward into the launch panel where there's room. Called on every open (not cached) so a collapsed / expanded sidebar Just Works, guarded defensively for the case the sidebar element is missing (future layouts), and only treats the sidebar as an obstacle for tips outside it — a tip inside the sidebar itself wouldn't get force-anchored based on its own container.

Changed

• GPU Layers label + tooltip corrected to match current llama.cpp value semantics. llama.cpp changed what --n-gpu-layers values mean without changing the flag name: -1 used to mean "all layers" and is now "auto" (a smart VRAM-aware picker that can, and does, leave layers on CPU), -2 is now "all" (force offload every layer), any positive integer is exact. The old llamaman UI still said -1 = all and defaulted to -1, so every launch was silently sending "auto" while the operator thought they were sending "all" — cases like "I set GPU Layers to -1 but not everything is on the GPU" were the observable symptom. The label now reads GPU Layers (-1 = auto, -2 = all, 0 = CPU only) and the tooltip spells out that -1, -2, and positive integers all mean different things, plus the aliases -ngl / --gpu-layers / --n-gpu-layers route to the same flag internally in llama.cpp's arg parser (verified against common/arg.cpp:2811). No behavior change to the field itself — llamaman still sends whatever the operator typed — this release only aligns the UI with what llama.cpp actually does with those values today.

• README.md and DOCKERHUB.md compressed to something scannable. README.md went from 847 → 596 lines (~30% shorter), docs/DOCKERHUB.md from 398 → 207 (~48% shorter). Consolidated the four near-identical GPU-vendor Quick Start bloc...

1.8.0

[1.8.0] - 2026-08-24

Added

• Image & PDF Input. llama-server has no PDF support at all — a PDF sent as an image_url data URL, an OpenAI file block with inline file_data, or an Ollama images[] entry simply returns a base64/decode error, and every OpenAI-compatible layer downstream of it inherits that gap. This release closes it. When an instance opts into Image & PDF Input (renamed from "Image Input") in the launch form and points MMPROJ Model at a multimodal projector GGUF, any inbound PDF is rewritten before llama-server sees the request — into an image_url block per rasterized page (default; needs the vision model + mmproj) or, when Try text layer first is on, into a single text block carrying the PDF's embedded text if it's substantive enough to trust (threshold: 50 chars/page averaged over the pages actually looked at). The text-layer path is a free win for born-digital PDFs (Word/LaTeX/browser print) and completes in milliseconds with no raster pass or model context spent on pixels; scanned PDFs fall through to rasterization automatically, so turning it on never breaks the OCR case. It is nevertheless default off, because "sometimes text, sometimes N images" is a surprising per-request shape shift for a caller who thought they were sending a picture — leaving it off gives predictable behavior (always rasterize) and operators opt in per instance when their traffic is text-heavy PDFs.
  • Both compat surfaces: /v1/chat/completions accepts either the OpenAI image_url block with an application/pdf data URL, or the OpenAI file block with inline file_data — MIME is header-sniffed (%PDF-) rather than trusted from the label. /api/chat and /api/generate accept PDFs inside the Ollama-native images[] array on any message; the translator splits real images from PDFs, lifts the images through the existing image_url path, and inlines the PDF's expanded blocks alongside. Broken PDFs surface as 400 with the specific decode error, not an opaque 500 from llama-server.
  • Per-request caps, not global: PDF DPI (72–600, default 200) and Max PDF Pages (1–200, default 20) are preset fields so operators bound the cost per instance. A PDF above max_pages is refused up front before any raster work — so a 500-page whitepaper cannot monopolize the raster pool for minutes. All four PDF keys are require-relaunch (unlike sampling knobs), because they participate in launch args and the mmproj path check.
  • LLAMAMAN_PDF_MAX_CONCURRENT env var (default 4) caps how many rasterizations can run process-wide at once. This is a process-level BoundedSemaphore, deliberately independent of any per-instance RequestGate: rasterization is CPU-heavy and spikes a few hundred MB of transient RAM per page render, and the RequestGate only limits inference concurrency downstream. A burst of PDF traffic without this cap would peg cores and thrash swap before the gate ever saw a request. Raise on beefy hosts with PDF-heavy traffic, lower on tight ones. Overflow blocks on the semaphore rather than piling on load.
  • Stack: poppler-utils (added to the Docker image) + pdf2image for rasterization, pypdf for the text-layer probe, Pillow for the PNG encode. All permissive licenses; PyMuPDF (AGPL) was considered and rejected to keep license semantics simple for downstream users.
  • UI: the launch form's Image & PDF Input section adds Accept PDF uploads, Try text layer first, PDF DPI, and Max PDF Pages underneath the existing MMPROJ Model control; the toggles gate the numeric inputs client-side, and the server re-validates bounds. Preset save/load, restart, and the auto-launch preset-merge whitelist all carry the four new keys.
  • Migration: none. All four preset keys default to safe values (pdf_input_enabled: false), so existing installs behave identically until an operator turns the feature on per instance. 23 new tests in tests/test_pdf_input.py cover the header sniff, text-layer heuristic, page-cap enforcement, OpenAI and Ollama block expansion, and the preset validator.

Fixed

• Sleeping models could jam LLAMAMAN_MAX_MODELS and refuse to wake on OpenAI-compat requests. With MAX=2 and two sleeping instances, an OpenAI request for one of them returned HTTP 503: model limit reached; the OpenAI API does not evict running models — instead of doing the obvious thing and waking it. The Ollama-compat path could hit the same 503 whenever Allow Ollama API to evict admin-launched models was off and the cap was full of admin-launched sleepers. Root cause was structural: _ensure_model_running ran the cap check before it knew whether the request would just wake an existing sleeper or actually launch a new one, and treated both identically. The fix computes would_wake_existing up front from the resolved existing record and skips both the cap check and the eviction pass when it's true. Waking a sleeping instance is using the slot that instance was already granted at launch — sleep pauses the container to free VRAM/RAM, but the launcher's authorization for that slot persists across the pause; only creating a genuinely new instance consumes a new slot.
  • Deliberately preserved: sleeping still counts against the cap for fresh launches, so an API request for a different model, when the cap is saturated by admin-launched sleepers, still returns 503 unless the "Allow API to evict admin-launched" toggle is on. That's the entire reason those toggles exist — the admin's slot claim has to survive a sleep/wake cycle or the toggles would be meaningless the moment idle timeout fires. The _count_running_* helpers therefore keep counting sleeping instances (six lines of docstring, no behavioral change to counting); only the wake path skips the check.
  • Applies to: both compat surfaces (/v1/chat/completions, /api/chat, /api/generate) — they share _ensure_model_running, so the fix reaches everything driven by the proxy. Direct hits to a per-instance public port were never affected because they don't cap-check; only the auto-launch/wake path did.
  • Docs: README.md §"Model eviction policy" gains a bullet spelling out the wake-vs-launch distinction explicitly, so a reader inferring behavior from the cap docs alone doesn't reach the old (buggy) expectation. Covered by 6 regression tests in tests/test_sleeping_not_counted.py.

1.7.0

[1.7.0] - 2026-08-21

Added

• Speculative decoding: three more draft types and an Advanced knobs subsection. The Draft Type dropdown grows to all five draft-model-family values llama.cpp accepts: draft-simple (any smaller model with the same tokenizer), draft-mtp (default; the only value where the drafter is optional and falls back to built-in MTP heads), draft-dflash, draft-dspark, draft-eagle3. The four newly listed types require a Draft Model; the preset save rejects a config that would fail at launch, and unknown values are rejected up front. The n-gram / lookup speculative family stays reachable via Extra Args. A new native <details> Advanced subsection exposes --spec-draft-n-min / --spec-draft-p-split / --spec-draft-p-min — blank omits the flag so llama-server uses its own default (which drifts across versions).
• Flash Attention + KV cache quantization. Model Settings gains a Flash Attention select (--flash-attn [on|off|auto], default Auto) and per-side K Cache Type / V Cache Type dropdowns (--cache-type-k / --cache-type-v; accepts f16 default plus f32, bf16, q8_0, q5_1, q5_0, iq4_nl, q4_1, q4_0). Flash Attention maps 1:1 to llama.cpp's tri-state: Auto lets llama.cpp decide per backend (and omits the flag entirely, matching its own default); On forces the fused kernel; Off disables it. --flash-attn on|off is emitted explicitly rather than as a bare flag, because llama.cpp's tri-state parser now consumes the next arg as the value. Non-default cache types emit their flag only when set to a value in the whitelist, so leaving all defaults reproduces pre-feature behaviour byte-for-byte. llama-server refuses quantized V cache without Flash Attention actually enabled; the UI enforces this by greying the V dropdown unless Flash Attention is explicitly On (Auto is not a guarantee - llama.cpp may resolve it to off on some backends) and auto-snapping a stale quantized V back to f16 the moment Flash Attention leaves On, but server-side we deliberately pass the flags through unchanged so an API caller bypassing the UI sees llama-server's real error rather than a silently dropped flag. All three keys are shared cluster-wide preset fields (matching ctx_size).
• Multi-GPU placement: Split Mode and Tensor Split. The launch form's new GPU Settings section exposes --split-mode (none / layer default / row) and --tensor-split as first-class controls. Tensor Split takes comma-separated relative weights per container-visible GPU (llama.cpp normalizes, so 24,16 == 3,2 == 0.6,0.4); blank auto-fills at launch time from each visible GPU's total VRAM, with a live preview under the field. Both fields are per-node hardware in the preset (topology varies per cluster node), and an empty split_mode from an old preset backfills to "layer" at emit time so existing configs behave identically.
• Grey-out matrix in GPU Settings, honest to what actually happens in-container. Intel vendor greys GPU Devices + Split Mode + Tensor Split (per-instance SYCL device filtering isn't wired up; gpu_devices is currently a silent no-op on Intel); GPU Layers = 0 greys the same trio; fewer than 2 visible GPUs greys Split Mode + Tensor Split; Split Mode = None greys Tensor Split alone. GPU Layers stays editable in every case because -ngl IS honoured on Intel. /api/gpu-info now returns the detected vendor alongside gpus[] so the client can decide the Intel case without a second roundtrip.

Changed

• The launch form is grouped into bordered sections. The compact top grid had grown to ~9 standalone fields with no visual grouping. Runtime fields now live in titled bordered boxes — Model Settings, Container & Proxy Settings, GPU Settings — using a new .launch-section-grid that mirrors the outer .form-grid exactly, so fields inside a section look identical to the top-level ones above them. Display Name gains an (optional) suffix; it stays in the launch-pair with Model Path + Note. No functional changes: field ids, preset serialization, and launch bodies are unchanged.
• Info-tip tooltips: wider variant + honest disabled-field opacity. A new .info-tip-wide variant bumps the max-width from 240px to 360px for tooltips whose text needs the room; the viewport-edge clip helper knows about the wider variant. Greyed fields' tooltips no longer render half-transparent — .gpu-field-disabled previously set opacity: 0.55 on the whole form-group, which cascades to every descendant via CSS opacity's stacking-context rules including the tooltip pseudo-element. Now only the label's colour is muted; the disabled input/select is already visually distinct via the browser's own :disabled styling.

1.6.2-1

[1.6.2-1] - 2026-08-10

Fixed

• The local database mirror still could not be switched on from the UI. 1.6.2 fixed a genuine defect on that path, but not the one operators were hitting, so the symptom was unchanged: the checkbox unticked itself a few seconds after being ticked, DATA_DIR/db_mirror/ never grew past the owner.json marker written at startup, and nothing appeared in the app log or the browser console. The toggle had no change listener. Every other control in App settings registers addEventListener('change', saveAppSettings), and saveAppSettings() did read s-db-mirror-enabled into its payload — but nothing ever called it when that checkbox changed, so ticking the box issued no request at all. loadStorageStatus() then reset the box from the effective state on its next poll, which looks identical to a save being rejected. Nothing was ever persisted, which is also why restarting the container did not help, and why GET /api/storage-status reported last_sync_at: null — not a failed sync, but no attempt ever made.
  • LLAMAMAN_DB_MIRROR=1 was the only way to turn the mirror on in 1.6.1 and 1.6.2. The env var is read directly by resolve_enabled() at startup and never went through the UI path.
  • 1.6.2's merge_settings fix remains necessary: with the listener restored but that fix absent, ticking the box would persist the setting to the database and leave it inert until the next restart. Two independent defects sat on the same path, so neither fix alone made the feature work — which is why 1.6.2 changed nothing observable.