Skip to content

Releases

inside-llm 1.0.0

Choose a tag to compare

@manuelgscom manuelgscom released this 15 Nov 18:50

inside-llm 1.0.0

See inside any GGUF: where two quantizations part ways, token by token, and what each sampler cuts. Live, in your browser.

First release. inside-llm loads one or two GGUF models with llama.cpp and shows in the browser what the model does while it generates: the probability of each token, the alternatives it discarded, what each sampler removes and, with two models (two quantizations, or a base model and its fine-tune), the divergence at every position and the exact token where their texts split. One native binary that serves its own web page; everything runs on your machine.

Try it first in your browser: the live demo replays recorded sessions, with nothing to install.

Highlights

  • Two models side by side: lockstep (Same text) or free (Each its own), with Jensen-Shannon divergence, KL both ways and the first token where the texts split.
  • The sampler chain, stage by stage: penalties, top-k, top-p, min-p, temperature and a GBNF grammar, with how many candidates survive each one.
  • Branching: click any candidate and generation continues from there; every branch stays in the tree, and returning to a node is exact.
  • Relief 3D and logit lens: the candidates of every position as a surface and, with --lens, what each layer would predict.
  • Models from the browser: pick A and B from your models folder or download GGUF files from Hugging Face, without restarting.
  • Recorded sessions: save a session and replay it later in a static page, with no engine and no GPU.

Downloads

Asset For SHA-256
inside-llm-v1.0.0-linux-cuda.tar.gz Linux + NVIDIA da38b830d21192af8a857fcb8bf28b4865f934e7476dd522c7feef2e83140704
inside-llm-v1.0.0-linux-vulkan.tar.gz Linux + any dedicated GPU (AMD, Intel Arc, NVIDIA) 79204920d848f1592dc6adb680a072051d874ebf06d9f1d922267208e96a8886
inside-llm-v1.0.0-windows-cuda.zip Windows + NVIDIA f13251b1bed8c6548b31eb47b4735fb271e2cd7b0227da914a4e6cba317040b8
inside-llm-v1.0.0-windows-vulkan.zip Windows + any dedicated GPU eb040f5945c0db1c3f6031d9000758b4b933188efc0b8e88f2b159a268a2ca29
SHA256SUMS checksums of the four packages
THIRD_PARTY_NOTICES licenses of the bundled components

Install

Pick the block for your system and paste it in a terminal. It downloads the package, checks its SHA-256 and, only if it matches, unpacks it and starts inside-llm. Which one? CUDA for an NVIDIA GPU; Vulkan for any other dedicated GPU (AMD, Intel Arc), or for NVIDIA if the CUDA package does not start.

Linux + NVIDIA (CUDA)

curl -LO https://github.com/manuelgscom/inside-llm/releases/download/v1.0.0/inside-llm-v1.0.0-linux-cuda.tar.gz
curl -LO https://github.com/manuelgscom/inside-llm/releases/download/v1.0.0/SHA256SUMS
sha256sum -c --ignore-missing SHA256SUMS && tar -xzf inside-llm-v1.0.0-linux-cuda.tar.gz && ./inside-llm-v1.0.0-linux-cuda/inside-llm

Linux + AMD, Intel Arc or NVIDIA (Vulkan)

curl -LO https://github.com/manuelgscom/inside-llm/releases/download/v1.0.0/inside-llm-v1.0.0-linux-vulkan.tar.gz
curl -LO https://github.com/manuelgscom/inside-llm/releases/download/v1.0.0/SHA256SUMS
sha256sum -c --ignore-missing SHA256SUMS && tar -xzf inside-llm-v1.0.0-linux-vulkan.tar.gz && ./inside-llm-v1.0.0-linux-vulkan/inside-llm

Windows + NVIDIA (CUDA)

In PowerShell:

curl.exe -LO https://github.com/manuelgscom/inside-llm/releases/download/v1.0.0/inside-llm-v1.0.0-windows-cuda.zip
curl.exe -LO https://github.com/manuelgscom/inside-llm/releases/download/v1.0.0/SHA256SUMS
$ok = (Get-FileHash inside-llm-v1.0.0-windows-cuda.zip -Algorithm SHA256).Hash -eq (Select-String SHA256SUMS -Pattern inside-llm-v1.0.0-windows-cuda.zip -SimpleMatch).Line.Split(" ")[0]
if ($ok) { Expand-Archive inside-llm-v1.0.0-windows-cuda.zip -DestinationPath . -Force; .\inside-llm-v1.0.0-windows-cuda\inside-llm.exe } else { "Checksum mismatch: delete inside-llm-v1.0.0-windows-cuda.zip and download it again" }

Windows + AMD, Intel Arc or NVIDIA (Vulkan)

In PowerShell:

curl.exe -LO https://github.com/manuelgscom/inside-llm/releases/download/v1.0.0/inside-llm-v1.0.0-windows-vulkan.zip
curl.exe -LO https://github.com/manuelgscom/inside-llm/releases/download/v1.0.0/SHA256SUMS
$ok = (Get-FileHash inside-llm-v1.0.0-windows-vulkan.zip -Algorithm SHA256).Hash -eq (Select-String SHA256SUMS -Pattern inside-llm-v1.0.0-windows-vulkan.zip -SimpleMatch).Line.Split(" ")[0]
if ($ok) { Expand-Archive inside-llm-v1.0.0-windows-vulkan.zip -DestinationPath . -Force; .\inside-llm-v1.0.0-windows-vulkan\inside-llm.exe } else { "Checksum mismatch: delete inside-llm-v1.0.0-windows-vulkan.zip and download it again" }

On Windows you can also unzip the package from the Explorer and double-click inside-llm.exe.

The browser opens with the model picker (or go to http://localhost:8642). Next time, just run ./inside-llm-v1.0.0-linux-…/inside-llm (Linux) or .\inside-llm-v1.0.0-windows-…\inside-llm.exe (Windows). To load two models straight away, add -a A.gguf -b B.gguf. More in the README.

Requirements

  • A dedicated GPU with enough memory for the model (or the pair). Integrated GPUs are never used; without a dedicated GPU it runs on the CPU, much slower.
  • Linux: glibc 2.35 or newer (Ubuntu 22.04, Debian 12, Fedora 36, Linux Mint 21 or later). CUDA: the NVIDIA driver. Vulkan: the GPU's Vulkan driver and libvulkan1.
  • Windows: the GPU driver. The CUDA packages carry the CUDA runtime and cuBLAS, so no CUDA toolkit is needed.
  • Model A and model B must share the tokenizer.

License

The code is under the MIT license. The licenses of the bundled components are in THIRD_PARTY_NOTICES.