picchio v0.5.0

Picchio v0.5.0 - First public release

Picchio is a pure-C streaming Mixture-of-Experts inference engine for the
GPT-OSS models. Its headline goal: run a 117-billion-parameter model on an
ordinary laptop by streaming expert weights from disk instead of holding the
whole model in RAM. This is the first public, open-source release.

Running GPT-OSS-120B on consumer hardware

The 120B (~117B parameters, 128 experts per layer) converts to about 66 GB and
runs on a normal machine that could never hold it in memory. This release
documents real, measured throughput on a deliberate worst case: a consumer
laptop with limited RAM, streaming experts from an external SSD, with the
shards even split across two drives. It is the hardest setup on purpose, not a
representative one; internal NVMe and more pinned RAM are faster.

Test config: --no-reasoning --ctx 4096 --pin-gb 6 --threads 6, 4 parallel I/O
threads, model on external SSD.

Three-turn chat, 16 tokens per turn:

Turn	KV reused	Prefill	Time-to-first-token	Decode rate	Overall rate
1 (cold)	0 / 86	86 tok	218 s	0.22 tok/s	0.038 tok/s
2 (warm)	95 / 110	15 tok	37 s	0.24 tok/s	0.16 tok/s
3 (warm)	126 / 147	21 tok	54 s	0.29 tok/s	0.15 tok/s

What this shows:

• Steady-state decode is a stable ~0.25 tok/s - the real hardware ceiling,
  set by streaming 4 of 128 experts per layer from the SSD on every token. It
  barely changes turn to turn.
• Perceived speed is dominated by the prefill. The first turn processes the
  whole prompt from scratch, so its overall rate collapses to ~0.04 tok/s. From
  the second turn on, Picchio reuses the KV-cache prefix (95/110, 126/147
  positions reused), so only the small delta is re-processed and the overall rate
  jumps about 4x, to ~0.15 tok/s. Short, continuous turns stay close to the
  decode ceiling.

In short: on this hardware the 120B is usable for patient, deliberate
exchanges - proof the model runs at all on consumer gear. For interactive use,
the 20B (roughly 0.6 s/token on an internal SSD) is the better choice.

Full walkthrough and the streaming, resumable conversion pipeline are in the
README.

Also in this release

• OpenAI-compatible HTTP server (server.py): point any OpenAI client at
  http://127.0.0.1:8000/v1 with per-request temperature, top_p, top_k,
  max_tokens, reasoning_effort, and no_reasoning.
• Token-exact chat client (chat.py) using the official Harmony encoding,
  with a redesigned terminal UI (banner, live spinner, per-turn stats) and a
  --no-reasoning fast mode.
• Streaming 120B conversion (convert_streaming.py): downloads and converts
  one shard at a time, resumable, with env-configurable paths and repo
  (PICCHIO_OUTPUT, PICCHIO_RAW, PICCHIO_REPO). Multi-disk loading via
  --model-aux.
• Performance: AVX2/FMA SIMD kernels and OpenMP; batched prefill with
  batch-union and double-buffered prefetch; LRU expert cache with post-MoE
  prediction; parallel expert reads; adaptive PIN_GB sized from physical RAM;
  INT8 embeddings and lm_head.
• Single static binary - build once, no DLLs to ship.
• Fixes & housekeeping: prefill keeps the fast path under a repetition
  penalty; real RSS reporting on Windows; full English translation of code and
  docs; repository cleanup; removed machine-specific logic. License: MIT.

Downloads

• picchio-v0.5.0-win64-avx2.exe - prebuilt static binary, no dependencies.
  Requires Windows x64 with an AVX2/FMA CPU (2013+). The binary is unsigned,
  so Windows SmartScreen may warn on first run.
  SHA256: b75fc938fd7071c4988c02a7a49939412510c5159dff04abf3c65cca0379884c
• Source code (zip/tar.gz) - build it yourself with .\build.bat (Windows)
  or make (Linux/macOS). Requires GCC (MinGW-w64 on Windows).

Verification

Builds cleanly and the self-test passes (picchio.exe --self-test): forward pass
plus unit checks for RMSNorm, softmax, F32 matmul, SiLU, RoPE, and INT4
quant/matmul.