Arc Llama 0.9.1.2

Arc Llama 0.9.1.2 makes the path from finding a model to chatting on Intel Arc easier to follow, and adds stronger cleanup and recovery around runtime compatibility checks.

Models and interface

• Hugging Face discovery sits beside your local library, with publisher avatars, exact quant choices, estimated memory fit, background downloads, and automatic model registration.
• Review a model and its settings before opening chat. Setup and recovery guidance explain the next action.
• Check a local GGUF or a selected Hugging Face file against your installed llama.cpp runtime. Explicit architecture rejections are shown separately from memory-fit warnings. Recognition alone does not guarantee every encoding or model will load or run.
• Chat adds Stop with partial-answer preservation, regenerate/edit, context estimates, per-model prompt and temperature presets, and image attachments for vision models.
• The README includes a six-screen screenshot walkthrough. An offline gallery provides thumbnails, keyboard navigation, and mobile swipe.

Runtime, integrations, and tuning

• Canary-checked runtime updates install beside the existing runtime before switching; rollback restores the previous installation.
• Vision projectors are paired during discovery and download, included in memory estimates, and passed to the runtime. Split GGUFs register as one model and fetch all shards. Text-only models reject image requests before loading.
• Versioned plugin API 1.0 provides authenticated admin access, GPU leases, lifecycle events, status, and plugin tooling. Incompatible plugins are reported instead of loaded.
• Reranker models support POST /v1/rerank; /v1/models reports capabilities and context size.
• API keys and LAN mode support remote clients while native model backends remain bound to loopback.
• Depth benchmarks, automatic speculative-draft comparisons, driver-aware recipe invalidation, and hourly performance history improve tuning visibility.
• Multi-GPU tensor split is available as an experimental option.

Reliability

• Runtime compatibility checks have bounded network reads, deadlines, and limited concurrency. Disconnects, runtime changes, and shutdown cancel pending work, close streams, and reap probe subprocesses.
• Abandoned or stale browser requests cannot overwrite a newer result. Failed checks remain retryable with clearer access, file, and runtime messages.
• Light-theme settings fields remain readable. Browser regressions are split into focused modules, with added cancellation coverage.
• Native inference smoke tests close subprocess pipes after cleanup so successful inference passes strict warning checks. File-replacement regression tests no longer depend on filesystem timestamp resolution.

Validation and coverage

• Linux and Windows CI on Python 3.10, 3.12, and 3.14, browser regressions, and bare-wheel installs pass.
• Local Linux suites pass 1,300 tests with warnings treated as errors on each supported Python version. Live inference is validated separately; 33 browser regressions pass. Windows CI passes 1,278 tests with 22 platform-specific skips and the opt-in inference smoke excluded; Linux CI skips only that smoke.
• A real Intel Arc Pro B60 (24 GiB), Linux, SYCL journey passed: fresh setup, real Hugging Face Qwen2.5-0.5B Q4_K_M download and registration, settings, streamed and non-streamed GPU inference, history, Stop/recovery, architecture rejection, and server restart. The marked native CLI smoke also passes with strict warnings.
• Wheel/source metadata and package contents were checked; a bare Python 3.10 installation and its packaged UI checks pass.
• Physical Windows hardware, other Arc GPUs, Vulkan inference, multi-GPU inference, and the separate vision companion's image-rendering workflow were not rerun for this release. Windows is covered by CI, not a new hardware certification. Vision and multi-GPU support depend on the installed runtime and model.

The requested four-component version 0.9.1.2 is a valid Python package version under PEP 440.

Install or upgrade

python -m pip install --upgrade arc-llama==0.9.1.2

See the full changelog for details. Updating Arc Llama does not itself replace your installed llama.cpp runtime; use the runtime update command when you want to update it.

Arc Llama 0.9.1

Arc Llama 0.9.1 is a hotfix that restores model launches on current llama.cpp builds.

Fixed

• Launch failures on current llama.cpp. Upstream replaced --no-mmap, --mlock and --mmap with --load-mode, and current llama-server builds reject the old flags with invalid argument. Arc Llama now probes the binary and launches with --load-mode when it is available: no_mmap becomes --load-mode none, mlock becomes --load-mode mmap+mlock, and both together become --load-mode mlock. Binaries older than llama.cpp b10105 keep the legacy flags. (#86, #87)

Who should upgrade

Anyone running a recent llama.cpp build with a recipe that sets no_mmap or mlock. Both options are off by default, so recipes that leave them unset were not affected.

Validation

• Linux and Windows CI on Python 3.10, 3.12 and 3.14, bare-wheel installs, and the browser UI regression suite all pass.
• A real chat request through Arc Llama succeeded on llama.cpp b10380 and b11476 (Intel Arc Pro B60, Linux, SYCL); the launch used --load-mode none on both.
• Every flag Arc Llama can emit was accepted by b9174, b10380 and b11476.
• This hotfix was not re-tested on Windows hardware or other Arc models; Windows is covered by CI only.

Known limitation

Flags you add yourself under extra_flags are passed through unchanged, so a hand-written --no-mmap or --mlock there will still be rejected by current llama.cpp builds. Use no_mmap / mlock in the recipe instead.

Install

python -m pip install --upgrade arc-llama

Arc Llama 0.9.0

Arc Llama 0.9.0 is the stable release of a more complete local inference workflow for Intel Arc GPUs.

Highlights

• Guided setup and launch: arc-llama run accepts a registered model, local GGUF, or Hugging Face GGUF source, checks the launch plan, and starts the API. Use --setup-only to inspect the plan first.
• Better model-fit decisions: GGUF metadata informs context and VRAM estimates, and known oversized launch plans are rejected before startup.
• Local performance tuning: benchmark and tune model recipes for the installed GPU and llama.cpp build; speculative-decoding candidates can be measured and rolled back when they do not meet the configured improvement threshold.
• More ways to connect: OpenAI-compatible and Ollama-compatible routes, upstream model support, a browser UI, and an optional terminal UI.
• More reliable model management: safer model ownership checks and switching, request draining, clearer startup diagnostics, chat-history validation, and atomic history writes.
• Safer Linux model loading: avoid starting a model when host RAM is below the configured safety reserve.

Validation

The release branch passes Linux and Windows CI on Python 3.10, 3.12, and 3.14, including bare-wheel installs and the browser UI regression suite. Physical GPU validation for this release used an Arc Pro B60 on Linux with Vulkan and SYCL. This does not certify every GPU model, driver, or operating-system/backend combination.

Install

python -m pip install --upgrade arc-llama

Arc Llama 0.9.0rc1

Arc Llama 0.9.0rc1

Release candidate for Linux and Windows.

Included

• Core Arc Llama runtime, router, plugins, resources, streaming metrics, and web UI updates.
• Windows portability fixes for UTF-8 UI assets and POSIX /proc map parsing.
• One-command setup and release validation updates.

Validation

• Linux Python 3.10, 3.12, and 3.14 CI: passed.
• Windows Python 3.10, 3.12, and 3.14 CI: passed.
• Linux browser regression and Linux/Windows wheel installs: passed.
• Local Windows Arc Pro B60 SYCL smoke test: streaming and non-streaming chat passed.

The arc-llama-vision companion remains a separate repository and is not included in these artifacts.

arc-llama 0.8.0

Cross-platform release of arc-llama 0.8.0.

Highlights:

• Safe speculative decoding with capability validation and target-only fallback.
• Community tune-recipe registry with explicit sharing.
• Installable plugin extension point.
• Open WebUI and LM Studio integration support.
• Docker Compose support with Linux GPU group handling.
• Windows runtime installation fallback for assetless GitHub latest releases.
• Windows console encoding support for legacy code pages.

Validation:

• Linux Intel Arc B60 end-to-end inference passed.
• Windows Intel Arc B60 Vulkan end-to-end inference passed.
• Windows Intel Arc B60 SYCL GPU discovery, model startup, and streaming inference passed.
• 694 tests passed and 31 skipped in the Windows environment.
• Ruff passed.
• Wheel and source archive built successfully.

The same 0.8.0 package supports Linux and Windows. Windows llama.cpp runtime binaries are downloaded by install-runtime.

arc-llama 0.7.0

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog,
and this project adheres to Semantic Versioning.

[0.7.0] - 2026-08-19

Added

• Memory-aware context/batch defaults now respect ARC_LLAMA_MAX_CTX as a global ceiling on auto-suggested context length. (#55)
• arc-llama add gained --batch-size to complement --ubatch-size and prints the chosen ctx/ub/b values after registration. (#55)
• SYCL launches now auto-detect and source Intel oneAPI setvars.sh when the runtime libraries are not visible, supporting non-standard install prefixes such as /mnt/storage/opt/intel/oneapi. Set paths.oneapi_setvars in the config to override the auto-detected path. (#56)
• Added an opt-in inference smoke test (tests/test_smoke.py). Set ARC_LLAMA_SMOKE_MODEL to run it locally on a machine with a model and Intel Arc GPU.
• Windows GPU detection via WMIC with PowerShell fallback.
• Windows ggml backend sibling discovery (ggml*.dll).
• Windows oneAPI .bat setvars sourcing and Level Zero DLL discovery.
• GitHub Actions CI matrix for Ubuntu and Windows.

Changed

• Dockerfile now pins llama.cpp to b10280 (was b9946). (#54)
• CI now runs mypy src/arc_llama and builds the Dockerfile on every PR.
• README updated with Windows-specific setup instructions.

Fixed

• .dockerignore no longer excludes docker-entrypoint.sh, so the Dockerfile can build again.
• Cleaned up mypy errors in detect.py, runtime.py, server.py, and cli.py.
• Hardened Config.save() for concurrent Windows renames.
• Strengthened agent tool path traversal check for Windows drives.
• Tests now pass on Windows (library prefixes, .dll/.bat detection, symlinks, pwd, curl).

[0.6.2] - 2026-08-11

Identical code to 0.6.1; packaging fix only. The 0.6.1 sdist accidentally included a local virtualenv (.venv-test/), whose absolute symlinks broke the wheel build, so 0.6.1 shipped no wheel and a 62 MB sdist. The sdist now explicitly excludes local virtualenvs and scratch files. If you pinned 0.6.1, move to 0.6.2.

[0.6.1] - 2026-08-11

Bugfix release clearing the remaining concurrency and leak findings from the full-source audit. No new features, no config or API changes.

Fixed

• A streamed request whose BackgroundTask never ran (mid-stream upstream death, shutdown with an open stream) leaked the global in-flight counter permanently — and since the auto-tuner gates every sweep on that counter, one leaked stream silently disabled background tuning until restart. The decrement now runs from the body generator's finally, is idempotent, and floors at zero. (#27)
• Model eviction no longer kills an incumbent's generation mid-stream: _evict_for waits (bounded) for the incumbent's per-model in-flight count to drain, and rebuild_model does the same before applying a recipe edit to a running model. (#28, #31)
• The lock-free fast path in ensure_active could hand out a server that an eviction was about to stop, or be missed by the drain because the request hadn't been counted yet. Acquisition is now atomic with the readiness check, and a _stopping mark closes the teardown window. (#29)
• The auto-tuner could start a benchmark load that evicted a just-arrived real request; measure() now re-checks the abort hook after the edit POST, the only window where that request can register. (#30)
• The end-of-sweep recipe restore no longer leaves the last candidate persisted when the edit endpoint fails: it retries, then falls back to a direct config write. (#37)
• Upstream proxying leaked the httpx.AsyncClient connection pool on every call, and the response too on mid-stream failure; both are now closed exactly once. (#38)
• Autotuner.start() is serialized with stop(), so a racing start/stop can no longer kill the loop or orphan a task. (#34)
• The autotuner reads running models via Router.running_models() instead of poking router._servers directly. (#33)
• tune --dry-run no longer marks the model as tuned (#22); draft-MTP is no longer auto-enabled on hybrid SSM models (#23); health/admin endpoints report a model as loaded only once its health check passed (#40); the deferred restore no longer consumes shared abort signals (#32); config writes are atomic (#36).

[0.6.0] - 2026-07-30

Added

• Background auto-tuner. Once a model has been used and the router is idle, arc-llama serve runs the same staged KV/ubatch/flash-attention sweep that arc-llama tune uses, over loopback HTTP, aborting instantly when a real request arrives. The tuned recipe is written via the existing /admin/models/{name}/edit endpoint.
• [tune] config section (auto, idle_seconds, target, prompt_tokens, gen_tokens, min_uses, retune_on_fingerprint_change) and per-model tune state ([tune_state, tuned_at, tune_fingerprint, tune_error]) persisted in the existing config.
• Fingerprint-based invalidation. A fingerprint covers the model file, llama-server binary, GPU pci_slot/arch/backend, arc-llama version, and a TUNE_SCHEMA_VERSION constant. Stale tuned recipes are treated as untuned again without a time schedule.
• Admin endpoints: GET /admin/tune/status, POST /admin/tune/{name} (queue now), DELETE /admin/tune (abort running sweep). tune_state is exposed in GET /admin/status model entries for the web UI/TUI.
• arc-llama serve --auto-tune / --no-auto-tune CLI overrides and an auto-tune banner at startup.
• arc-llama tune --status prints the per-model tune state table without measuring.
• Manual arc-llama tune (single model and --all) now records the same fingerprint and tuned state via a shared helper.

Changed

• tune_model now accepts should_abort and on_stage callbacks. The recipe restore is wrapped in a try/finally, and TuneReport.aborted reports whether the sweep was abandoned mid-stage. Aborted sweeps restore only winners from fully completed stages.
• Router exposes last_activity and inflight counters; request completion bumps last_activity so long generations count as activity.
• MoE expert offload (n_cpu_moe) is now a measured tune axis: for MoE models that need offload, the sweep computes the minimum feasible layer count from per-layer expert-tensor GGUF accounting (against the winning KV type and workload context), measures it, and probes one step below/above. The /admin/models/{name}/edit endpoint now accepts n_cpu_moe, and ubatch candidates that would not fit under the chosen offload are skipped rather than measured into an OOM. TUNE_SCHEMA_VERSION bumped to 2, so previously tuned recipes are retuned.
• Registration-time MoE offload suggestion now uses the same VRAM estimator as the load-time guard (minimum feasible --n-cpu-moe layer count from tensor metadata), replacing the file-size/expert-count guess — --n-cpu-moe N is a layer count, not an expert count.

Fixed

• Exceptions or CancelledError during a tune sweep no longer leave the last losing candidate's recipe persisted — the original/winning recipe is restored in finally.
• The router's VRAM fit guard now subtracts the expert tensor bytes that --n-cpu-moe keeps on the host, instead of counting full weights and refusing exactly the models expert offload was enabled to rescue. When offload bytes cannot be determined, the guard warns and permits the load rather than silently disabling the feature.

[0.5.0] - 2026-07-23

Highlights: arc-llama install-runtime downloads a prebuilt portable Vulkan llama-server, removing the need to install oneAPI or build from source.

Added

• arc-llama install-runtime command to download a prebuilt llama-server from official ggml-org/llama.cpp releases. Vulkan is the default backend, portable, and requires no oneAPI installation. SYCL remains optional. Verified end to end on Battlemage B60.
• arc-llama tune staged autotuner for KV cache type, ubatch, and flash attention. It measures performance on your card and writes the winning recipe. Added arc-llama tune --all to sweep every registered model in one run.
• arc-llama benchmark harness for prompt-eval and generation tokens per second.
• Vulkan backend support via backend = "vulkan" in the config, alongside SYCL.
• Windows support for the launcher, CLI, and config paths. CI is green on windows-latest and ubuntu-latest for Python 3.10 to 3.12.
• Auto-scan for new GGUFs on serve startup, and auto-detection of sidecar speculative-draft (MTP) models.
• Experimental agent loop, terminal agent UI, MCP client, checkpoints, chat persistence, and chat export/import. Gated behind ARC_LLAMA_EXPERIMENTAL_AGENT.

Changed

• arc-llama init now writes a config even when no llama-server is present, and points users to install-runtime.
• init and install-runtime set each GPU's backend to match the actual binary. Previously, it always defaulted to SYCL, which mismatched Vulkan builds.
• doctor now surfaces AOT build guidance, device-ID VRAM fallback, metadata-based KV class, crash-log surfacing, config migration, and Prometheus-style metrics.

Fixed

• Backend detection now scans sibling libggml-*.so shared libraries for modern modular builds, and no longer false-matches bare backend NAME strings in libggml.so. Previously, a downloaded Vulkan build reported its backend as "unknown", then as "sycl".
• Benchmark generation and prompt-eval measurement accuracy.
• MTP ubatch_size regression.
• MoE detection.

[0.4.0]

Initial public releases.

v0.6.2 — audit-fix patch release

Bugfix release clearing the remaining concurrency and leak findings from the full-source audit. No new features, no config or API changes. (0.6.1, with identical code, shipped a broken sdist and no wheel — use 0.6.2.)

Fixes

• A dropped streamed request permanently disabled auto-tune. If a stream's BackgroundTask never ran (upstream dies mid-generation, shutdown with an open stream), the global in-flight counter leaked, and the auto-tuner gates every sweep on it. The decrement now runs from the body generator's finally, idempotently, with a zero floor. (#27)
• Model swaps and recipe edits no longer kill generations mid-stream. Eviction and rebuild_model both drain the incumbent's per-model in-flight count (bounded) before stopping it. (#28, #31)
• The lock-free fast path can no longer hand out a dying server. Acquisition is atomic with the readiness check, and a _stopping mark closes the teardown window. (#29)
• Auto-tune no longer evicts a just-arrived request. measure() re-checks the abort hook after the edit POST — the only window where that request can register. (#30)
• A failed end-of-sweep restore no longer persists the last candidate. It retries, then falls back to a direct config write. (#37)
• Upstream proxying no longer leaks the httpx client pool on every call, nor the response on mid-stream failure. (#38)
• Autotuner start/stop are serialized; a racing pair can't kill the loop or orphan a task. (#34)
• Plus: dry-run no longer marks models tuned (#22), no auto draft-MTP on hybrid SSM (#23), loaded-reporting waits for health (#40), deferred restore keeps abort signals (#32), atomic config writes (#36), audit finding #41 verified already fixed.

Verification

599 tests passing (19 new), ruff clean, and an end-to-end run on an Arc Pro B60: cold start, SSE streaming, model swap with eviction drain, a mid-stream swap race (stream completed cleanly while the swap waited), upstream proxy in both modes, admin edit → rebuild → reload, and clean shutdown with zero orphaned processes.

pip install --upgrade arc-llama

v0.6.0 — the package runs again

If you installed arc-llama from PyPI any time in the last three weeks, it did not work. This release fixes that. 0.4.0 and 0.5.0 could not run a single command, including arc-llama --version, on a default pip install. 0.3.0 was the last release that ran.

Fixes

The package works again. The CLI imported the optional [tui] extra at module scope, so every command died with arcllama requires textual unless you happened to install extras. (#49)

Vulkan no longer runs on the wrong GPU. arc-llama passed the SYCL device index as a Vulkan device index. SYCL enumerates Intel devices only; Vulkan enumerates every vendor. On a machine with a discrete NVIDIA or AMD card next to an Arc, models silently ran on the other vendor's GPU and looked like they were working. The index is now resolved from llama-server --list-devices, and when it cannot be resolved arc-llama warns rather than guessing. (#52, fixes #51)

The server shuts down. On Python 3.10 and 3.11, asyncio.wait_for discards a CancelledError that arrives after its inner future resolved, which is exactly the window the auto-tune loop's stop path hit every time. Shutdown hung forever and left llama-server running. (#48)

No more orphaned subprocesses or leaked log handles on shutdown. (#45)

The admin token is no longer printed at startup, and list sends its auth header. (#44)

Expert-count detection, read-only config survival, and --config actually being honoured. (#46)

Auto-tune on idle measures the recipe instead of guessing it. (#47)

Guardrails

CI now installs the built wheel with no extras and runs every command, so a packaging break cannot ship green again. (#50)

Verified on hardware before tagging

Bare wheel install on Python 3.10, init GPU detection, install-runtime download, generation on both the SYCL build and the downloaded portable Vulkan build, model swapping, SSE streaming, and clean shutdown on 3.11 and 3.12 with zero orphaned processes.

pip install --upgrade arc-llama