The woodpecker drums a hundred times a second on a huge trunk; we drum 128 experts on a huge disk.
A streaming Mixture-of-Experts (MoE) inference engine for the GPT-OSS models (20B and 120B), written in pure C, that runs models larger than your RAM on ordinary consumer hardware.
Most of a MoE model's weight is in its experts, and only a handful of experts are used for each token. Picchio keeps only the small "dense" part of the model permanently in memory and streams the experts from disk on demand, caching the ones it has recently used. This is what lets a 14 GB model (20B) run comfortably on a 16 GB laptop, and makes the 66 GB model (120B) runnable at all without a datacenter GPU.
Inspired by Colibri (GLM), adapted for the GPT-OSS architecture.
New to this? Read the sections in order. Every command below is complete: nothing is assumed. Windows commands are shown for PowerShell; Linux/macOS equivalents are given where they differ.
- What you need (hardware & software)
- Install the toolchain
- Build the engine
- Download and convert a model
- Run it: the chat bridge (recommended)
- Run it: an OpenAI-compatible API server
- Running the big model (120B)
- Tuning & environment variables
- Troubleshooting
- Verifying correctness (optional)
- How it works & project layout
- License
| Resource | Minimum | Recommended (20B) | Notes |
|---|---|---|---|
| CPU | x86-64 with AVX2 | 6+ cores with AVX2/FMA | Almost every desktop/laptop CPU since ~2013 has AVX2. Without it the build fails or runs very slowly. |
| RAM | 8 GB | 16 GB | The 20B needs ~3 GB always resident + expert cache. More RAM = more cache = less disk reading = faster. |
| Disk | ~30 GB free | SSD/NVMe, ~30 GB free | The model is read from disk constantly, so an internal SSD matters a lot. A USB drive roughly doubles the I/O time. |
The 120B model additionally needs ~70 GB of free disk and benefits from as much RAM as you can give it (see section 7).
- A C compiler (GCC or Clang). On Windows this means MSYS2/MinGW.
- Python 3.9+ (for converting the model and for the chat/server bridges).
- An internet connection to download the model once from Hugging Face.
a) Install MSYS2 (provides the GCC compiler).
- Download and run the installer from https://www.msys2.org.
- Accept the default install location
C:\msys64. - Open the "MSYS2 MinGW 64-bit" terminal from the Start menu and install GCC:
pacman -S mingw-w64-x86_64-gcc make
build.batexpects the compiler atC:\msys64\mingw64\bin\gcc.exe(the default). If you installed elsewhere, edit theGCC=line inbuild.bat.
b) Install Python from https://www.python.org/downloads/ (tick "Add Python to PATH" during setup).
sudo apt install build-essential python3 python3-pip # Debian/Ubuntuxcode-select --install # gives you clang + make
brew install python # if you don't already have Python 3From the project folder (C:\picchio or wherever you cloned it):
If you would rather not build from source, download the prebuilt Windows binary from the Releases page:
- Save it as
picchio.exe. If the asset has a versioned name (e.g.picchio-v0.5.0-win64-avx2.exe), rename it topicchio.exe, or pass--exe <name>tochat.py/server.py. Every command below assumes the file is calledpicchio.exe. - It is a static build: no MinGW DLLs, runs from anywhere.
- Requires Windows x64 with an AVX2/FMA CPU (2013 or newer). The binary is unsigned, so Windows SmartScreen may warn on first run ("More info" then "Run anyway").
- Verify the download against the SHA256 published on the release.
Then skip to section 4 to get a model. To compile it yourself instead (any OS), continue below.
.\build.batThis produces a self-contained picchio.exe (statically linked, it does not
need any MinGW DLLs and runs from anywhere).
Or compile by hand from the MSYS2 MinGW terminal:
gcc -O2 -Wall -fopenmp -mavx2 -mfma -Wno-misleading-indentation \
-Wno-unused-function -static -Wl,--stack,8388608 \
-o picchio.exe picchio.c -lm -lpsapimake-fopenmp: enables multi-core. Without it, all matmuls run on one core and everything is several times slower.-mavx2 -mfma: enables the SIMD kernels. Without them the math falls back to slow scalar code. Your CPU must support AVX2.-static(Windows): bakes the OpenMP/pthread runtime into the exe so you don't needlibgomp-1.dll/libwinpthread-1.dllnext to it.
.\picchio.exe --self-test # Windows
./picchio --self-test # Linux/macOSThis runs the full forward pass on a tiny synthetic model, no model download
needed. You should see ── self-test PASSED ──. If you
do, the engine works.
GPT-OSS ships in a format Picchio can't read directly (MXFP4). You convert it once into Picchio's INT4 format. We'll use the 20B model, which is the recommended choice for 16 GB machines.
pip install torch safetensors numpy huggingface_hubpython convert.py --model openai/gpt-oss-20b --output C:\models\gptoss20b_i4 --download--model: the Hugging Face repo id (openai/gpt-oss-20b).--output: a folder you choose where the converted model will be written. Put it on your fastest internal disk. Use any path you like (e.g.C:\models\gptoss20b_i4or~/gptoss20b_i4).--download: fetch the model from Hugging Face automatically. Omit this if you already downloaded the raw model yourself and pointed--modelat a local folder.
This downloads several GB and writes a converted model of about 14 GB to the output folder. It only needs to be done once.
Reclaim space after converting. The raw Hugging Face download is left in a sibling folder named
<output>_raw(e.g.C:\models\gptoss20b_i4_raw). Only the--outputfolder is needed to run Picchio, so once the conversion finishes you can delete<output>_rawto free that extra space.
Hugging Face access: the GPT-OSS models are openly licensed and normally download without an account. If you ever get a
401/gated error, runpip install huggingface_hubandhuggingface-cli loginonce with a free token from https://huggingface.co/settings/tokens.
Picchio needs a small binary tokenizer file next to the model:
python export_vocab.py C:\models\gptoss20b_i4\tokenizer.json C:\models\gptoss20b_i4\picchio_vocab.bin(The two arguments are: the tokenizer.json that came with the model, and the
output path for the binary vocab. export_vocab.py has no dependencies.)
Your model folder is now ready to use.
chat.py is the recommended way to talk to the model. It uses OpenAI's
official "Harmony" library to format the conversation exactly the way GPT-OSS
expects, so the output is correct token-for-token.
pip install -r requirements-chat.txt(That installs openai-harmony, the only extra package needed to chat.)
python chat.py "Write a short greeting in English." --model C:\models\gptoss20b_i4 --pin-gb 4 --ctx 1024--model: the folder you converted in step 4. You must pass this (the built-in default points at a 120B path and won't match your setup).--pin-gb: how many GB of RAM to spend on the expert cache. More = faster (fewer disk reads).4is a good start on a 16 GB machine.--ctx: context window in tokens (how much conversation history fits).1024is fine to start.
Omit the prompt to get a chat loop that keeps the model and its cache in memory between turns:
python chat.py --model C:\models\gptoss20b_i4 --pin-gb 4 --ctx 1024 --max-tokens 200 --temperature 0.7Type your message after the you ❯ prompt. Type /exit or /quit to leave.
| Option | What it does |
|---|---|
--temperature 0.7 |
Randomness. Use ~0.7 for normal conversation. The default 0 (greedy) is deterministic but can make the model loop in its "thinking" channel without answering. |
--max-tokens 200 |
Maximum length of the reply. |
--no-reasoning |
Skip the internal "analysis" (chain-of-thought) and answer directly. Faster. |
--show-analysis |
Show the model's private reasoning instead of only the final answer. |
--top-p, --top-k, --seed |
Standard sampling controls. |
--reasoning low|medium|high |
How much the model thinks before answering. |
--json |
Print the structured reply as JSON. |
--dry-run |
Show the exact tokens that would be sent, without loading the model (handy for debugging). |
You can run the engine directly without Python. This uses a built-in
approximate tokenizer (not token-exact; prefer chat.py for real use):
$env:MODEL = "C:\models\gptoss20b_i4"
$env:INPUT = "The capital of Italy is"
$env:MAX = "40"
.\picchio.exeOn Linux/macOS:
MODEL=~/gptoss20b_i4 INPUT="The capital of Italy is" MAX=40 ./picchioserver.py exposes the model over HTTP with the same API shape as OpenAI, so any
OpenAI-compatible client or tool can talk to it. It uses only the Python standard
library plus openai-harmony (already installed in step 5a).
python server.py --model C:\models\gptoss20b_i4 --port 8000 --pin-gb 4 --ctx 1024It prints [server in ascolto su http://127.0.0.1:8000 ...] when ready.
POST /v1/chat/completions: streaming (SSE) and non-streaming.GET /v1/modelsGET /health
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")
resp = client.chat.completions.create(
model="gptoss20b",
messages=[{"role": "user", "content": "Say hello in one sentence."}],
max_tokens=64,
temperature=0.7,
)
print(resp.choices[0].message.content)curl http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"gptoss20b","messages":[{"role":"user","content":"Hello"}],"max_tokens":64}'Per-request options (in the JSON body): temperature, top_p, top_k,
max_tokens, reasoning_effort ("low"/"medium"/"high"), and
no_reasoning: true.
Note: the model is a single process with one KV-cache, so requests are handled one at a time (serialized). This is meant for personal/local use, not for serving many users concurrently.
The 120B converts to about 66 GB and runs on the same machine as the 20B, only much more slowly, because far more must be streamed from disk. It is a "works, with patience" model, not a daily driver: expect well under 1 token/s (see Measured performance below for real numbers on consumer hardware). For everyday use the 20B is the better choice.
- Disk space: you need about 70 GB free on the output drive.
On Windows, "used space" can be inflated by hidden shadow copies (System
Restore) under
System Volume Information: if a drive looks full but your files don't add up, reclaim it with Disk Cleanup, or from an Administrator prompt:vssadmin delete shadows /for=D: /all. - Dependencies (same as section 4, plus the fast downloader):
pip install torch safetensors numpy huggingface_hub hf_transfer
convert_streaming.py downloads and converts one shard at a time, never
keeping more than one raw shard (~4.6 GB) on disk. hf_transfer makes the
download several times faster (multi-connection: ~5 MB/s vs ~0.7 MB/s in testing):
$env:PYTHONUTF8 = "1" # progress symbols print correctly
$env:HF_HUB_ENABLE_HF_TRANSFER = "1" # multi-connection downloads (much faster)
$env:PICCHIO_OUTPUT = "D:\gptoss120b_i4" # where the converted shards go (~66 GB)
$env:PICCHIO_RAW = "D:\gptoss_tmp" # scratch for the single raw shard
python convert_streaming.pyPICCHIO_OUTPUT,PICCHIO_RAW, andPICCHIO_REPOare read from the environment; point them at a disk with room (defaults are set in the script).- Resumable: already-converted shards are skipped, so if the download drops or you stop it, just run the same command again and it continues.
- Do not use an HF mirror here.
HF_ENDPOINT=hf-mirror.comserves the small config files but fails on the large LFS shards. Download from Hugging Face directly (the default).
When it finishes, the output folder holds model-00000.safetensors through
model-00014.safetensors, plus config.json, tokenizer.json, and
picchio_vocab.bin (the vocab is generated for you). The expert biases are
baked into the shards (F32), so no separate sidecar is needed.
Keep the context and cache modest on 16 GB (the dense part alone is ~5 GB):
$env:PYTHONUTF8 = "1"
python chat.py --model D:\gptoss120b_i4 --no-reasoning --ctx 1024 --pin-gb 6 --max-tokens 200 --temperature 0.7On startup Picchio reads the architecture from config.json, opens all 15
shards, and loads only the ~5 GB dense part into RAM; the experts stay on disk
and are streamed on demand:
The first turn is slow (it streams every expert from disk); later turns reuse the
KV prefix and the learned hot-store, so they speed up. You can see this in a real
three-turn session: the reused counter on each stats line climbs from 0/82 to
169/187 to 291/307 as the KV-cache prefix is carried over between turns.
The numbers below are a deliberate stress test: the whole point of Picchio is
to prove a 117B-parameter MoE model can run at all on a consumer laptop with
limited RAM, streaming the experts from an external SSD. This is the hardest
case on purpose, not a representative one. On an internal NVMe drive, or with more
RAM devoted to the expert cache (--pin-gb), the rates are higher.
Test configuration:
| Model | GPT-OSS-120B, INT4 (gs64) experts, F32 attention (~66 GB) |
| Storage | external SSD (shards split across two drives via --model-aux) |
| Launch | --no-reasoning --ctx 4096 --pin-gb 6 --threads 6 --temperature 0 |
| Expert cache | 6 GB pinned (--pin-gb 6), 4 parallel I/O threads |
Three-turn chat, generating 16 tokens per turn:
| Turn | KV reused | Prefill | Time-to-first-token | Decode rate | Overall rate |
|---|---|---|---|---|---|
| 1 (cold) | 0 / 86 | 86 tok | 218 s | 0.22 tok/s | 0.038 tok/s |
| 2 (warm) | 95 / 110 | 15 tok | 37 s | 0.24 tok/s | 0.16 tok/s |
| 3 (warm) | 126 / 147 | 21 tok | 54 s | 0.29 tok/s | 0.15 tok/s |
Two things to read from this:
- Steady-state decode is stable at ~0.25 tok/s and is the real hardware ceiling: every token routes to 4 of 128 experts per layer, streamed from the SSD. This barely changes turn to turn.
- Perceived (overall) speed depends almost entirely on the prefill. The first turn must process the entire prompt from scratch (86 tokens, 218 s before the first token), so its overall rate collapses to ~0.04 tok/s. From the second turn on, Picchio reuses the KV-cache prefix (95/110, 126/147 positions reused), so only the small delta is re-processed and the overall rate jumps about 4x, to ~0.15 tok/s. Short, continuous turns stay close to the decode ceiling; long new prompts pay the prefill cost up front.
In short: on this hardware the 120B is usable for careful, patient exchanges, not interactive chat. If you want responsiveness, run the 20B.
You can spread the shards across two disks and pass the ones on the second disk
with --model-aux (semicolon-separated). For example, if the last shard lives on C::
python chat.py --model D:\gptoss120b_i4 --model-aux "C:\gptoss120b_extra\model-00014.safetensors" --no-reasoning --ctx 1024 --pin-gb 6--model-aux also carries any other loose files a model may need.
Legacy note (bias sidecar). Containers converted with older code quantized the expert biases by mistake and needed a separate F32 sidecar (
python download_expert_biases.pywritesexpert_biases.safetensors, passed via--model-aux). A fresh conversion with the currentconvert.pyincludes the biases in the shards, so you can ignore this.
Picchio is configured through environment variables (the chat.py/server.py
flags map onto these). The most useful:
| Variable | Default | Meaning |
|---|---|---|
MODEL |
(none) | Path to the converted model folder (or pass it as the first argument). |
PIN_GB |
auto | GB of RAM for the expert cache. The single biggest performance knob. By default it's sized automatically from your physical RAM (all RAM minus a ~6 GB reserve). A bigger cache means fewer disk reads. Setting a value overrides the auto-sizing. |
CTX |
512 | KV-cache size in tokens (max prompt+generation length). |
OMP_NUM_THREADS |
all cores | Number of CPU threads for the matmuls. |
MAX |
128 | Max tokens to generate (bare-metal run only). |
TEMPERATURE |
1.0 | Sampling temperature (0 = greedy). |
TOPP / TOPK |
0.95 / 50 | Nucleus / top-k sampling. |
SEED |
fixed | RNG seed for reproducible sampling. |
IO_THREADS |
4 | Threads used for reading experts from disk in parallel. |
MODEL_AUX |
(none) | Extra model files on other disks (semicolon-separated). |
Performance notes:
- On the 20B (6 cores, model on NVMe) expect roughly ~0.6 s per token.
- Keep the model on an internal SSD. From USB the I/O time roughly doubles.
- More RAM devoted to
PIN_GBis almost always the best speedup: going from a small cache to full residency on the 20B cut disk reads by ~53% in testing.
For the design rationale and measurements, see DESIGN.md.
picchio.exe exits immediately / "libgomp-1.dll not found".
You built without -static. Either rebuild with .\build.bat (which uses
-static), or run from the MSYS2 MinGW terminal / add C:\msys64\mingw64\bin to
your PATH.
"Illegal instruction" crash on startup. Your CPU lacks AVX2, or you built for a different CPU. Rebuild on the machine you run on. AVX2 is required.
The model keeps "thinking" and never gives an answer.
You're in greedy mode. Add --temperature 0.7 (chat) or set TEMPERATURE=0.7.
Output is gibberish / degenerates in long replies.
Make sure you converted with the current convert.py (it keeps the embedding and
output head at INT8 as required). Models converted with older code must be
reconverted.
Out of memory / very slow.
Lower PIN_GB (e.g. --pin-gb 2) and/or lower --ctx. Streaming still works with
a small cache; it just reads from disk more often.
Conversion download is extremely slow (120B).
See the mirror tip in section 7
(HF_ENDPOINT=https://hf-mirror.com).
Garbled accented characters in terminal output (Windows).
Set PYTHONUTF8=1 before running Python scripts.
If you want to confirm the math matches a reference implementation, there's a
lightweight numeric oracle (needs only numpy and safetensors):
pip install safetensors numpy
python make_test_model.py # writes a tiny synthetic model to ./test_model
python test_forward.py test_model # validates the forward pass against the oracleThe built-in picchio --self-test (section 3) is the quickest sanity check and
needs nothing at all.
The idea in one paragraph: the dense weights (attention, router, embedding, output head) stay resident in RAM. For each token the router picks the top-4 of 128 experts per layer; Picchio loads just those experts, computing them while an LRU cache keeps recently-used experts around and a learned hot-store keeps the most frequently used ones pinned. Because only a few experts are touched per token, total disk traffic is a fraction of the model size.
| Property | 20B | 120B |
|---|---|---|
| Total parameters | 21 B | 117 B |
| Active per token | ~3.6 B | ~5.1 B |
| Hidden size | 2880 | 2880 |
| Layers (all MoE) | 24 | 36 |
| Experts / layer | 32 | 128 |
| Active experts / token | 4 (top-4) | 4 (top-4) |
| Attention | GQA (64 Q / 8 KV heads), sliding-window + full, attention sinks, YaRN | same |
| Converted size | ~14 GB | ~66 GB |
Quantization: experts are INT4 (group-scaled, 64), the embedding and output head are INT8, attention is F32.
picchio.c The engine (single translation unit)
quant.h Quantized matmul kernels (F32 / INT8 / INT4) with AVX2/NEON
st.h safetensors reader (multi-shard, multi-disk)
json.h config.json parser
tok.h Built-in approximate tokenizer (fallback for bare-metal runs)
Makefile / build.bat Build for Linux/macOS and Windows
convert.py Convert a GPT-OSS model (MXFP4/BF16 -> INT4) for Picchio
convert_streaming.py Shard-by-shard download+convert for the 120B
export_vocab.py Build the binary tokenizer file
download_expert_biases.py Regenerate the 120B expert-bias sidecar
chat.py Token-exact chat bridge (Harmony), single-turn and multi-turn
server.py OpenAI-compatible HTTP API server
requirements-chat.txt Dependency for chat.py / server.py (openai-harmony)
make_test_model.py Generate a tiny synthetic model for validation
test_forward.py Numeric oracle to validate the forward pass
DESIGN.md Design notes, rationale, and measurements
For a much deeper dive into the numerics, the streaming/caching design, the
service protocol, and the measured results, read DESIGN.md.
MIT. See LICENSE.