Repository navigation
AI Explanations Setup
DiffLens can explain any single change in natural language (English or Vietnamese) using a small LLM running entirely on your machine. This is optional — every other feature works without it. This page covers what gets downloaded, where it goes, what it costs in disk/RAM, the privacy properties, and troubleshooting.
DiffLens does not link llama.cpp into the app. Instead it downloads a pinned llama-server build (llama.cpp release b12503, MIT) and spawns it as a sidecar process only when you request an explanation:
- Serves an OpenAI-compatible
/v1/chat/completionsendpoint with SSE streaming. - Binds loopback only (
127.0.0.1, ephemeral port), started with a per-session API key. - Lifecycle is fully managed: spawned on demand, health-polled, killed on exit; a stale server from a crashed session is detected and replaced.
Everything lands inside your OS app-data directory (app_data_dir):
<app_data_dir>/
├── bin/
│ └── llama-server-b12503/ # extracted bundle (server + shared libs)
├── models/
│ ├── Qwen3-4B-Instruct-2507-Q4_K_M.gguf
│ └── qwen2.5-3b-instruct-q4_k_m.gguf
└── difflens.sqlite3 # session history (unrelated to AI)
Typical locations: %APPDATA%\app.difflens.desktop (Windows), ~/Library/Application Support/app.difflens.desktop (macOS), ~/.local/share/app.difflens.desktop (Linux). Deleting the bin/ and models/ folders fully resets the AI components.
| Model | Size | Role | Min RAM | License |
|---|---|---|---|---|
| Qwen3-4B-Instruct-2507 (GGUF Q4_K_M, via unsloth) | ~2.4 GB (2,497,281,120 bytes) | Primary — best quality, strong Vietnamese | 8 GB | Apache-2.0 |
| Qwen2.5-3B-Instruct (GGUF Q4_K_M, official Qwen) | ~2.0 GB (2,104,932,768 bytes) | Fallback for 6–8 GB machines | 6 GB | Apache-2.0 |
| Qwen3-0.6B (Q8_0) | ~0.6 GB | CI smoke-test model | — | Apache-2.0 |
llama-server (llama.cpp b12503) |
~13–25 MB per OS | Sidecar binary | — | MIT |
Explain runs use a 2048-token context and cap output at 512 tokens; the model is picked automatically (first manifest entry present on disk: primary → fallback → test).
- URLs come only from
resources/manifests/llm-manifest.json, embedded at compile time. The manifest requires https URLs and verified (verified: true) SHA-256 entries — a tampered manifest cannot disable checksums. - Each download streams into a
.partfile with a size cap checked against the manifest's expected byte count, then the SHA-256 is verified before the atomic rename intomodels/. A mismatch deletes the part file — a bad file never becomes "installed". - Interrupted downloads resume from the
.partfile; Cancel aborts the transfer cleanly (keep the.partfor resume). - The e5-small embedding model (~118 MB ONNX, MIT) goes through the same manager with its own pinned commit + SHA-256.
-
Loopback-only: the sidecar binds
127.0.0.1(or[::1]); since v1.0.1 only those literals are accepted — DNS names likelocalhostare rejected so a hostile resolver can never redirect the bearer token. The guard is re-validated on every call. -
Key hand-off off the command line: the per-session key (32 bytes of OS randomness) is passed via the
LLAMA_API_KEYenvironment variable, not--api-key, so it never appears in the process table (v1.0.1 hardening). -
Single egress point:
core/src/download/is the only module in the entire codebase allowed to perform HTTP. This is not a convention —scripts/audit-egress.shruns as a CI job on every push and fails the build if any HTTP client code appears anywhere else (the LLM client is additionally restricted to loopback). - No telemetry: no crash reporting, no analytics, no update phone-home. The network stack has no other code path.
- Disk: ~2.5 GB (primary model) + ~65 MB (server) + ~118 MB (embeddings). Add headroom for the
.partfile during download. - RAM: the 4B model at Q4_K_M wants ≥ 8 GB total system RAM; on 6–8 GB machines pick the 3B fallback (the Downloads screen shows this guidance before you commit to a download).
- First explanation after launch includes model load time (seconds on an SSD) — subsequent explanations reuse the running sidecar.
| Symptom | Cause / fix |
|---|---|
| "port conflict"-style spawn failure | Another process grabbed the loopback port during spawn — DiffLens uses an ephemeral port, so this is rare; retry. A stale llama-server from a crashed session is killed and replaced automatically. |
| Explain button says no model | Download a model on the Downloads screen; the panel's empty state links there. |
| "llama-server not installed" | Install the sidecar on the Downloads screen (or a llama-server on your PATH is used as a fallback). |
| Model load failure / OOM | Your machine may not have enough free RAM for the 4B model — use the Qwen2.5-3B fallback. Corrupted downloads are impossible to miss: the SHA-256 gate refuses to install a bad file. |
| Explanation stops mid-way | You (or a 256 KiB output cap) cancelled the stream — safe by design; just request again. |
| Download stuck | Cancel (really aborts since v1.0.1) and resume — the .part file is kept. |
All events driving this UI: llm://download (server + model installs), llm://token (streamed deltas), llm://done (terminal: summary/impact or error).