Picchio v0.8.0 - MiniMax-M2 support

Picchio v0.8.0 - MiniMax-M2, a third model family

Picchio v0.8.0 adds MiniMax-M2 (230 B total, ~10 B active, 62 layers ×
256 experts) alongside GPT-OSS and Qwen3-MoE. At ~122 GB converted it is by far
the largest model the engine has run: it does not fit in RAM on any consumer
machine and streams from disk end to end, which is precisely the case Picchio was
built for.

Support is config-gated on model_type="minimax", like Qwen3 before it, so the
existing families are untouched - verified by the built-in self-test plus
end-to-end generation on all three.

Highlights

MiniMax-M2 support

Three architectural switches, each on only for MiniMax checkpoints:

• Partial RoPE - only the first rotary_dim (64) channels of each 128-wide
  head rotate; the rest pass through untouched.
• Whole-vector QK-Norm - a single RMSNorm across the entire concatenated
  multi-head Q (or K) vector, rather than per head as in Qwen3.
• Sigmoid routing - e_score_correction_bias selects the top-k experts, but
  the mixing weight is the unbiased sigmoid score, renormalised over the
  selection. The bias never reaches the weights. Applied in both top-k sites
  (single-token decode and the batched prefill loop); the GPU router is bypassed
  for this path, since its kernel assumes a pre-sigmoid linear bias.

Converter and chat bridge

• convert_minimax.py: GPTQ-INT4 → Picchio INT4 gs64, with --dense-bits 8 for
  INT8 attention, atomic per-shard writes, and an optional --delete-source that
  frees each source shard as it is consumed (destructive; the original checkpoint
  is gone afterwards).
• chat_minimax.py: MiniMax reasons unconditionally - its template's generation
  prompt ends in a literal <think>, so replies begin inside a reasoning block
  and there is no enable_thinking switch. The bridge splits the reply on the
  </think> token, hides reasoning behind a progress spinner by default
  (--show-thinking streams it), and records it through the template's own
  reasoning_content field.

Engine fixes

• ST_MAX_TENSORS raised from 32 768 to 131 072. That budget covers the whole
  tensor database rather than a single shard; MiniMax's expert tensors alone are
  ~63 k, and the old ceiling stopped registering tensors mid-load without
  reporting an error.
• cfg_rotary_dim() falls back to head_dim when rotary_dim is unset, so
  hand-built configurations (the synthetic self-test model, whose Cfg is
  zero-initialised) keep full rotation instead of silently skipping RoPE
  entirely - a defect the self-test could not have caught on its own, since both
  forward paths would have skipped it identically.
• chat_qwen.py repaired for transformers 5.x, where
  apply_chat_template(tokenize=True) returns a BatchEncoding instead of a
  list. The bridge was shipping the literal text input_ids attention_mask as
  token IDs, and the engine exited on it.

Measured performance

MiniMax-M2 on a 12-core AVX2 laptop, 32 GB RAM, model on an entry-level NVMe.
This is a different, larger machine than the 16 GB laptop used for the figures in
the README's main table, so these numbers are not comparable with those:

Configuration	tok/s	Expert-cache hit
--pin-gb 12	0.34	42.9%
+ --async-moe --direct	0.40	42.9%
+ --pin-gb 20	0.48	54.6%

The expert cache is the dominant lever: the experts total ~119 GB, so even 20 GB
holds only ~17% of them while every token touches 496 experts across 62 layers.
Raising --io-threads past the default made no difference - the NVMe is not
queue-depth limited. IDOT=1 gained ~5% but visibly changed the output, which is
expected of the approximate integer kernel and not a good trade. Unlike the other
families, repeat runs do not speed up: the learned hot-store converges
immediately and the resident expert set stops changing.

For scale, gpt-oss-120b reaches ~2 tok/s on this same machine, because it
streams roughly 4× less expert data per token.

Upgrade notes

• No action required for existing GPT-OSS or Qwen3 models: converted checkpoints,
  flat stores, and command lines are unchanged.
• The startup banner now reads GPT-OSS/Qwen3-MoE/MiniMax-M2.
• Anyone running the chat bridges on transformers 5.x should take this release:
  chat_qwen.py was broken there before it.
• Model weights are not included. Users must obtain and convert checkpoints in
  accordance with their original licenses.

Validation performed for this release

• Clean Windows x64 AVX2/FMA build.
• Built-in forward and math self-tests, including the RoPE, INT3/INT4 matmul, and
  pipeline byte-identity checks.
• End-to-end generation regression on all three families after the shared-code
  changes: GPT-OSS-120B (INT3 + INT8 attention), Qwen3-30B-A3B (INT4 + INT8
  attention) and MiniMax-M2 each answered a factual prompt correctly.
• minimax_forward_check.c matches the real upstream MiniMaxM2ForCausalLM to
  max|Δlogit| = 1e-6 on the committed fixture (verify_minimax.py).
• The MiniMax architecture was additionally cross-checked line by line against
  llama.cpp's own minimax-m2.cpp, which agrees on all three switches.
• The converted 230 B checkpoint loads all 64 349 tensors and generates coherent
  text.
• Python syntax validation and import/CLI smoke tests across the project's entry
  points.

Known limitations

• MiniMax-M2 is I/O-bound on consumer hardware; expect well under 1 tok/s unless
  a large fraction of the experts fits in the cache.
• convert_minimax.py emits INT4 experts only. The engine already supports INT3
  (picchio_expert_bits: 3), which would cut expert bytes ~22%, but the
  converter does not offer it yet.
• No full-model numeric oracle comparison against transformers for MiniMax-M2.
  This is blocked on the library itself: its in-tree MiniMax-M2 support drops the
  checkpoints' rotary_dim and applies full-head RoPE
  (huggingface/transformers#48241),
  so it is not a trustworthy reference for this architecture today. For the same
  reason the validation fixture is committed rather than regenerated, and
  make_minimax_test_model.py refuses to run on transformers >= 5.0 instead of
  emitting a silently wrong oracle.
• Carried over from 0.7.0: speculative decoding remains scaffolding and is off by
  default; GPU paths remain experimental and do not beat the CPU path on small
  (4 GB) cards; the server uses one model process and one KV cache, so requests
  are serialized.

Third-party code

reference/minimax_m2/ vendors the upstream MiniMax-M2 modeling code
(Apache-2.0, © 2024-2025 ModelCloud.ai) byte-identical and unmodified, used
only to build the validation fixture. It is never compiled into the engine. See
reference/README.md for attribution.

Downloads

• picchio.exe: self-contained Windows x64 CPU build. Requires AVX2/FMA. The
  binary identifies itself as v0.8.0 at startup and is unsigned, so Windows
  SmartScreen may display a warning.
• SHA256SUMS.txt: checksum for the downloadable binary.
• GitHub-generated source archives: build with build.bat on Windows or make
  on supported Unix-like systems.

Binary SHA-256:

72767a6d140d111ef97bad42c36a5891e3a89552ca0129614183129c465d4a28  picchio.exe

Full change history

Compare: v0.7.0...v0.8.0

Picchio v0.7.0 - INT8 attention

Picchio v0.7.0 - INT8 attention and format/engine co-design

Picchio v0.7.0 focuses on byte reduction, the dominant cost on a CPU that
streams a MoE model larger than its RAM. The headline is INT8 attention: the
attention weights, previously stored in F32, were the single largest chunk of
per-token byte movement. Storing them near-losslessly in INT8 roughly halves the
attention time and frees several gigabytes of resident RAM for the expert cache.
This release also adds dense-model conversion, speculative-decoding scaffolding,
and a VRAM-resident expert read-skip, on top of the multi-model streaming core
from 0.6.0.

Highlights

INT8 attention (--dense-bits 8)

• Added end-to-end INT8 storage for the attention projections (Q/K/V/O): per-row
  scales, run through the existing INT8 matmul kernel. Near-lossless in quality,
  it roughly halves t_attn and shrinks the resident dense part.
• Wired the previously inert --dense-bits 8 path in convert.py and taught
  convert_streaming_qwen.py to honor DENSE_BITS during shard-by-shard
  conversion.
• Added transcode_attn_to_int8.py: requantizes the attention weights of an
  existing converted model (INT4/INT3 experts + F32 attention) to INT8 in place,
  without re-downloading or re-converting the original checkpoint. Resumable.
• Applied and quality-checked across gpt-oss-20b, Qwen3-30B-A3B, and
  gpt-oss-120b: correct facts and arithmetic, coherent multi-turn prose.

Dense-model conversion

• A dense (non-MoE) checkpoint now converts as a 1-expert MoE: the single MLP
  becomes expert 0 with a zeroed router and a rewritten config
  (num_local_experts=1, num_experts_per_tok=1), so the streaming engine runs
  it unchanged.
• Added tie_word_embeddings handling (e.g. small Qwen3 dense checkpoints).
• Primary use is providing a small, RAM-resident draft model for speculative
  decoding.

Speculative decoding (experimental)

• Added a batched verification path (forward_verify, per-position argmax over a
  prefill range) and a draft-model loop (DRAFT_MODEL, SPEC_K) that proposes K
  tokens, verifies them in one pass, and accepts the longest correct prefix plus
  one bonus token.

• The draft model is loaded in isolation so two open model databases do not
  collide on the per-thread file-handle cache; output is byte-identical to
  greedy decode.

• Added SPEC_PROBE and SELF_DRAFT_PROBE diagnostics.

  This path is a net win only when the target model is memory-resident and the
  draft has a high acceptance rate. On a disk-bound target where ASYNC_MOE
  already hides the expert I/O, the extra draft and verification work is a net
  loss, so it is shipped as scaffolding, disabled by default, for larger-RAM
  and GPU configurations.

Streaming and cache refinements

• Skip the disk read for experts already resident in the VRAM expert cache
  (pgpu_moe_resident), with a safe CPU fallback if a resident-only expert is
  evicted.
• Fixed the safetensors per-thread file-handle cache so two open model databases
  (target + draft) no longer share handles keyed only by file index.
• Added ECAP to pin the entire expert tier resident when the model fits in RAM.

Documentation

• Reworked README.md: at-a-glance badges, a headline Measured performance
  table right after the intro, an ASCII engine schema (per-token flow and
  memory hierarchy), documentation for INT8 attention and the new environment
  variables, and an honest GPU note for small (4 GB) cards.
• Accuracy fix: routing is described as top-k with per-model expert counts
  (previously "top-4 of 128", which is GPT-OSS-specific; Qwen3-30B-A3B is top-8).

Measured results

Warm, greedy decode on one six-core AVX2 Windows laptop, 16 GB RAM, internal
NVMe, GTX 1650 4 GB (left idle). These describe one machine and are not hardware
guarantees.

Model	experts	F32 attention	INT8 attention
gpt-oss-20b	INT4	1.4 tok/s	3.3 tok/s	+136%
Qwen3-30B-A3B	INT4	2.2 tok/s	2.9 tok/s	+32%; dense resident 4.3 → 1.6 GB
gpt-oss-120b	INT3	0.5 tok/s	1.24 tok/s	66 GB streamed on 16 GB RAM

The gain is largest where attention dominates (the 20B). The 120B remains
storage-bound: its decode also scales with drive speed, and moving it to a faster
internal NVMe roughly doubled throughput, consistent with the streaming design.
On the 4 GB GTX 1650 the GPU paths did not beat the optimized CPU path; the
effective levers on this hardware are RAM residency and byte reduction (INT8
attention, INT3 experts).

Compatibility and upgrade notes

• Existing INT4/INT3 model directories from 0.6.0 remain usable unchanged; their
  attention stays F32.
• INT8 attention is opt-in. To use it, either convert with --dense-bits 8
  (or DENSE_BITS=8 for the streaming Qwen converter) or retrofit an existing
  model with transcode_attn_to_int8.py, then run against the new directory.
• DRAFT_MODEL, SPEC_K, SPEC_PROBE, SELF_DRAFT_PROBE, and ECAP are opt-in;
  the standard single-model CPU path remains the reference default.
• The GPT-OSS F32-attention path is unchanged; nothing in the default runtime
  behaves differently unless the new flags/env vars are set.
• Model weights are not included. Users must obtain and convert checkpoints in
  accordance with their original licenses.

Validation performed for this release

• Clean Windows x64 AVX2/FMA build.
• Built-in forward and math self-tests, including the INT8 matmul path and the
  speculative-verify byte-identity self-test.
• Quality spot-checks with INT8 attention on gpt-oss-20b, Qwen3-30B-A3B, and
  gpt-oss-120b (factual, arithmetic, and multi-turn coherence).
• Python syntax validation for convert.py, convert_streaming_qwen.py, and
  transcode_attn_to_int8.py.

Known limitations

• Speculative decoding is scaffolding: on a single disk-bound machine it does not
  improve throughput and is off by default.
• INT8 attention quality was validated by spot-checks, not a full per-checkpoint
  logit comparison against the original full-precision model.
• GPU paths remain experimental and, on small (4 GB) cards, do not beat the CPU
  path.
• The server uses one model process and one KV cache; requests are serialized.

Downloads

• picchio.exe: self-contained Windows x64 CPU build. Requires AVX2/FMA. The
  binary identifies itself as v0.7.0 at startup and is unsigned, so Windows
  SmartScreen may display a warning.
• SHA256SUMS.txt: checksum for the downloadable binary.
• GitHub-generated source archives: build with build.bat on Windows or make
  on supported Unix-like systems.

Binary SHA-256:

d21c90d855f438eeb1c001b4a162763167f560e15943d7441769af2996bce3d1  picchio.exe

Full change history

Compare: v0.6.0...v0.7.0

Picchio v0.6.0 - Multi-model MoE streaming

Picchio v0.6.0 - multi-model MoE streaming and storage-bypass I/O

Picchio v0.6.0 expands the project from a GPT-OSS streaming prototype into a
multi-model MoE runtime. This release adds Qwen3-MoE and INT3 support, introduces
an aligned expert store with direct and completion-driven asynchronous I/O, and
adds a two-node pipeline mode. It also includes substantial multi-turn,
conversion, correctness, and observability improvements.

Highlights

Qwen3-MoE support

• Added runtime and converter support for Qwen3-MoE checkpoints, including
  QK-Norm, plain SwiGLU, softmax-normalized top-k routing, and the Qwen rotary
  embedding layout.
• Added chat_qwen.py, which renders Qwen's native ChatML template through the
  Transformers tokenizer and uses Picchio's persistent SERVICE protocol.
• Added resumable shard-by-shard conversion with
  convert_streaming_qwen.py.
• Added an end-to-end synthetic Qwen3-MoE regression test covering conversion,
  safetensors inference, PicchioFlat, direct I/O, asynchronous MoE, and service
  framing.
• Validated the runtime on a converted Qwen3-30B-A3B checkpoint: all 25,013
  tensors loaded and synchronous/asynchronous greedy runs produced identical
  token IDs.

INT3 expert storage and kernels

• Added group-scaled INT3 expert storage end to end: quantization, conversion,
  loading, self-tests, and AVX2/FMA decode kernels.
• Added transcode_i4_to_i3.py to convert existing Picchio INT4 experts to INT3
  without downloading the original checkpoint again.
• Dense tensors retain the precision required by the existing runtime; INT3 is
  applied to the large streamed expert weights.

PicchioFlat and asynchronous storage pipeline

• Added the version 2 .picchioflat format: one contiguous, 4 KiB-aligned
  payload per expert, a resident SHA-256-protected index, and optional per-expert
  payload verification.
• Added flat_pack.py, flat_bench.py, and flat_bench_qd.py for packing,
  byte-verification, and queue-depth benchmarking.
• Added cross-platform direct expert reads with O_DIRECT on Linux and
  FILE_FLAG_NO_BUFFERING on Windows, with a safe buffered fallback.
• Added completion-driven ASYNC_MOE: ready experts can be computed while the
  remaining routed experts are still being read. Contributions are reduced in
  canonical router order, preserving deterministic output.
• Added configurable I/O concurrency through IO_THREADS and frontend flags
  --io-threads, --async-moe, and --direct.

Two-node distributed inference

• Added a persistent TCP pipeline that splits the transformer at a layer
  boundary across two machines.
• Added batched distributed prefill, avoiding one network round trip per prompt
  token.
• Added coordinator-driven sampling and persistent service/chat integration.
• Added loopback and byte-identity tests proving that the distributed layer split
  produces the same tokens as the monolithic forward path.

The current split is primarily a capacity feature. Both nodes need access to the
converted model files, and a network round trip is added to each decode step, so
single-stream speed is not guaranteed to improve.

Optional GPU and CPU experimental paths

• Added a runtime-loaded CUDA backend that leaves the default CPU build
  dependency-free.
• Added optional GPU lm_head and MoE expert-cache prototypes, with automatic
  CPU fallback.
• Added an optional INT8-activation × INT4-weight CPU kernel with AVX-VNNI
  runtime dispatch and an AVX2 fallback (IDOT=1).

These paths are experimental and remain disabled by default. Measurements on a
4 GB GTX 1650 Max-Q did not beat the optimized CPU path; GPUs able to keep a much
larger expert set resident are the intended target.

Chat, server, and runtime reliability

• Fixed long multi-turn GPT-OSS handling and improved KV-prefix reuse.
• Added live reasoning display and improved terminal status reporting.
• Added per-request repetition penalty support and robust expert reads.
• Kept the OpenAI-compatible HTTP server on the same persistent service path.
• Expanded runtime timing, cache, I/O, prefetch, and async-wait statistics.
• Corrected Qwen stop-token handling and service-mode tokenizer behavior.

Measured results

These measurements describe one six-core Windows test machine and are not
hardware guarantees.

• GPT-OSS-20B INT4, internal NVMe, constrained expert cache: complete
  PicchioFlat plus direct I/O and asynchronous MoE averaged 1.206 tok/s,
  versus 1.104 tok/s for asynchronous safetensors (+9.2%). Compared with
  one synchronous safetensors reference at 0.842 tok/s, the complete pipeline
  was approximately 43% faster. All generated token IDs matched.
• Qwen3-30B-A3B, short four-token greedy regression: asynchronous safetensors
  completed in 10.77 seconds versus 12.75 seconds synchronously, approximately
  +18.4% tok/s, with identical token IDs.
• GPT-OSS-120B remains strongly storage-bound on an external SSD; previously
  measured steady-state decode was approximately 0.25 tok/s. Faster storage
  and a larger RAM expert cache remain the most effective improvements.

See README.md, DESIGN.md, and DESIGN_STREAMING_IO.md for the complete test
conditions and interpretation.

Compatibility and upgrade notes

• Existing supported Picchio INT4 model directories remain usable.
• .picchioflat version 1 prototype files are not compatible with this release.
  Regenerate them with the current flat_pack.py.
• Models converted by older revisions that stored embed_tokens or lm_head in
  the legacy packed-U8 layout must be reconverted with the current converter.
• ASYNC_MOE, DIRECT, IDOT, and the CUDA backend are opt-in. The synchronous
  CPU path remains the reference default.
• PicchioFlat files are model-specific generated artifacts and are intentionally
  not included in the release.

Validation performed for this release

• Clean Windows x64 AVX2/FMA build.
• Built-in forward and math self-test, including INT4, INT3, canonical async-MoE
  reduction, and monolithic-versus-split byte identity.
• Two-stage pipeline self-test over a real loopback TCP socket.
• Python syntax validation for the converters, chat frontends, server, flat-store
  tools, distributed launcher, and smoke tests.
• Synthetic Qwen3-MoE conversion/runtime/service smoke test.
• Complete PicchioFlat expert payload verification against the source
  safetensors containers during development testing.

Known limitations

• Qwen3-MoE has synthetic oracle coverage and real-checkpoint runtime coverage;
  a full per-checkpoint numerical/logit comparison against the original
  full-precision Transformers model remains future validation work.
• Long Qwen multi-turn prefix-reuse sessions need broader testing.
• Distributed inference reduces resident layer memory per node but introduces
  network latency.
• The server uses one model process and one KV cache; requests are serialized.
• CUDA support requires a separately built picchio_cuda library and is not
  included in the standard CPU binary.
• Model weights are not included. Users must obtain and convert checkpoints in
  accordance with their original licenses.

Downloads

• picchio.exe: self-contained Windows x64 CPU build. Requires AVX2/FMA. The
  binary identifies itself as v0.6.0 at startup and is unsigned, so Windows
  SmartScreen may display a warning.
• SHA256SUMS.txt: checksum for the downloadable binary.
• GitHub-generated source archives: build with build.bat on Windows or make
  on supported Unix-like systems.

Binary SHA-256:

ca13b50d4fb1bddc4cd25ddef2247c774214313840eee5415f94051db7cdb51f  picchio.exe

Full change history

Compare: v0.5.0...v0.6.0

picchio v0.5.0

Picchio v0.5.0 - First public release

Picchio is a pure-C streaming Mixture-of-Experts inference engine for the
GPT-OSS models. Its headline goal: run a 117-billion-parameter model on an
ordinary laptop by streaming expert weights from disk instead of holding the
whole model in RAM. This is the first public, open-source release.

Running GPT-OSS-120B on consumer hardware

The 120B (~117B parameters, 128 experts per layer) converts to about 66 GB and
runs on a normal machine that could never hold it in memory. This release
documents real, measured throughput on a deliberate worst case: a consumer
laptop with limited RAM, streaming experts from an external SSD, with the
shards even split across two drives. It is the hardest setup on purpose, not a
representative one; internal NVMe and more pinned RAM are faster.

Test config: --no-reasoning --ctx 4096 --pin-gb 6 --threads 6, 4 parallel I/O
threads, model on external SSD.

Three-turn chat, 16 tokens per turn:

Turn	KV reused	Prefill	Time-to-first-token	Decode rate	Overall rate
1 (cold)	0 / 86	86 tok	218 s	0.22 tok/s	0.038 tok/s
2 (warm)	95 / 110	15 tok	37 s	0.24 tok/s	0.16 tok/s
3 (warm)	126 / 147	21 tok	54 s	0.29 tok/s	0.15 tok/s

What this shows:

• Steady-state decode is a stable ~0.25 tok/s - the real hardware ceiling,
  set by streaming 4 of 128 experts per layer from the SSD on every token. It
  barely changes turn to turn.
• Perceived speed is dominated by the prefill. The first turn processes the
  whole prompt from scratch, so its overall rate collapses to ~0.04 tok/s. From
  the second turn on, Picchio reuses the KV-cache prefix (95/110, 126/147
  positions reused), so only the small delta is re-processed and the overall rate
  jumps about 4x, to ~0.15 tok/s. Short, continuous turns stay close to the
  decode ceiling.

In short: on this hardware the 120B is usable for patient, deliberate
exchanges - proof the model runs at all on consumer gear. For interactive use,
the 20B (roughly 0.6 s/token on an internal SSD) is the better choice.

Full walkthrough and the streaming, resumable conversion pipeline are in the
README.

Also in this release

• OpenAI-compatible HTTP server (server.py): point any OpenAI client at
  http://127.0.0.1:8000/v1 with per-request temperature, top_p, top_k,
  max_tokens, reasoning_effort, and no_reasoning.
• Token-exact chat client (chat.py) using the official Harmony encoding,
  with a redesigned terminal UI (banner, live spinner, per-turn stats) and a
  --no-reasoning fast mode.
• Streaming 120B conversion (convert_streaming.py): downloads and converts
  one shard at a time, resumable, with env-configurable paths and repo
  (PICCHIO_OUTPUT, PICCHIO_RAW, PICCHIO_REPO). Multi-disk loading via
  --model-aux.
• Performance: AVX2/FMA SIMD kernels and OpenMP; batched prefill with
  batch-union and double-buffered prefetch; LRU expert cache with post-MoE
  prediction; parallel expert reads; adaptive PIN_GB sized from physical RAM;
  INT8 embeddings and lm_head.
• Single static binary - build once, no DLLs to ship.
• Fixes & housekeeping: prefill keeps the fast path under a repetition
  penalty; real RSS reporting on Windows; full English translation of code and
  docs; repository cleanup; removed machine-specific logic. License: MIT.

Downloads

• picchio-v0.5.0-win64-avx2.exe - prebuilt static binary, no dependencies.
  Requires Windows x64 with an AVX2/FMA CPU (2013+). The binary is unsigned,
  so Windows SmartScreen may warn on first run.
  SHA256: b75fc938fd7071c4988c02a7a49939412510c5159dff04abf3c65cca0379884c
• Source code (zip/tar.gz) - build it yourself with .\build.bat (Windows)
  or make (Linux/macOS). Requires GCC (MinGW-w64 on Windows).

Verification

Builds cleanly and the self-test passes (picchio.exe --self-test): forward pass
plus unit checks for RMSNorm, softmax, F32 matmul, SiLU, RoPE, and INT4
quant/matmul.