Skip to content

AI Explanations Setup

vibecoder880 edited this page Nov 19, 2026 · 1 revision

AI Explanations Setup

DiffLens can explain any single change in natural language (English or Vietnamese) using a small LLM running entirely on your machine. This is optional — every other feature works without it. This page covers what gets downloaded, where it goes, what it costs in disk/RAM, the privacy properties, and troubleshooting.

The sidecar model

DiffLens does not link llama.cpp into the app. Instead it downloads a pinned llama-server build (llama.cpp release b12503, MIT) and spawns it as a sidecar process only when you request an explanation:

  • Serves an OpenAI-compatible /v1/chat/completions endpoint with SSE streaming.
  • Binds loopback only (127.0.0.1, ephemeral port), started with a per-session API key.
  • Lifecycle is fully managed: spawned on demand, health-polled, killed on exit; a stale server from a crashed session is detected and replaced.

What gets downloaded, and where

Everything lands inside your OS app-data directory (app_data_dir):

<app_data_dir>/
├── bin/
│   └── llama-server-b12503/          # extracted bundle (server + shared libs)
├── models/
│   ├── Qwen3-4B-Instruct-2507-Q4_K_M.gguf
│   └── qwen2.5-3b-instruct-q4_k_m.gguf
└── difflens.sqlite3                  # session history (unrelated to AI)

Typical locations: %APPDATA%\app.difflens.desktop (Windows), ~/Library/Application Support/app.difflens.desktop (macOS), ~/.local/share/app.difflens.desktop (Linux). Deleting the bin/ and models/ folders fully resets the AI components.

Model table

Model Size Role Min RAM License
Qwen3-4B-Instruct-2507 (GGUF Q4_K_M, via unsloth) ~2.4 GB (2,497,281,120 bytes) Primary — best quality, strong Vietnamese 8 GB Apache-2.0
Qwen2.5-3B-Instruct (GGUF Q4_K_M, official Qwen) ~2.0 GB (2,104,932,768 bytes) Fallback for 6–8 GB machines 6 GB Apache-2.0
Qwen3-0.6B (Q8_0) ~0.6 GB CI smoke-test model — Apache-2.0
llama-server (llama.cpp b12503) ~13–25 MB per OS Sidecar binary — MIT

Explain runs use a 2048-token context and cap output at 512 tokens; the model is picked automatically (first manifest entry present on disk: primary → fallback → test).

Download integrity

  • URLs come only from resources/manifests/llm-manifest.json, embedded at compile time. The manifest requires https URLs and verified (verified: true) SHA-256 entries — a tampered manifest cannot disable checksums.
  • Each download streams into a .part file with a size cap checked against the manifest's expected byte count, then the SHA-256 is verified before the atomic rename into models/. A mismatch deletes the part file — a bad file never becomes "installed".
  • Interrupted downloads resume from the .part file; Cancel aborts the transfer cleanly (keep the .part for resume).
  • The e5-small embedding model (~118 MB ONNX, MIT) goes through the same manager with its own pinned commit + SHA-256.

Privacy properties

  1. Loopback-only: the sidecar binds 127.0.0.1 (or [::1]); since v1.0.1 only those literals are accepted — DNS names like localhost are rejected so a hostile resolver can never redirect the bearer token. The guard is re-validated on every call.
  2. Key hand-off off the command line: the per-session key (32 bytes of OS randomness) is passed via the LLAMA_API_KEY environment variable, not --api-key, so it never appears in the process table (v1.0.1 hardening).
  3. Single egress point: core/src/download/ is the only module in the entire codebase allowed to perform HTTP. This is not a convention — scripts/audit-egress.sh runs as a CI job on every push and fails the build if any HTTP client code appears anywhere else (the LLM client is additionally restricted to loopback).
  4. No telemetry: no crash reporting, no analytics, no update phone-home. The network stack has no other code path.

RAM/disk planning

  • Disk: ~2.5 GB (primary model) + ~65 MB (server) + ~118 MB (embeddings). Add headroom for the .part file during download.
  • RAM: the 4B model at Q4_K_M wants ≥ 8 GB total system RAM; on 6–8 GB machines pick the 3B fallback (the Downloads screen shows this guidance before you commit to a download).
  • First explanation after launch includes model load time (seconds on an SSD) — subsequent explanations reuse the running sidecar.

Troubleshooting

Symptom Cause / fix
"port conflict"-style spawn failure Another process grabbed the loopback port during spawn — DiffLens uses an ephemeral port, so this is rare; retry. A stale llama-server from a crashed session is killed and replaced automatically.
Explain button says no model Download a model on the Downloads screen; the panel's empty state links there.
"llama-server not installed" Install the sidecar on the Downloads screen (or a llama-server on your PATH is used as a fallback).
Model load failure / OOM Your machine may not have enough free RAM for the 4B model — use the Qwen2.5-3B fallback. Corrupted downloads are impossible to miss: the SHA-256 gate refuses to install a bad file.
Explanation stops mid-way You (or a 256 KiB output cap) cancelled the stream — safe by design; just request again.
Download stuck Cancel (really aborts since v1.0.1) and resume — the .part file is kept.

All events driving this UI: llm://download (server + model installs), llm://token (streamed deltas), llm://done (terminal: summary/impact or error).

Clone this wiki locally