Skip to content

midge v0.1.0 — tiny engine, immense models

Choose a tag to compare

View all tags
@drmihirbrahme drmihirbrahme released this 15 Jul 00:07
· 17 commits to main since this release
v0.1.0
322c163

First release.

midge runs 100B+ mixture-of-experts LLMs on ordinary machines by keeping the dense trunk resident and streaming routed experts from disk. First supported models: openai/gpt-oss-120b and gpt-oss-20b (a 75 GB disk and 8 GB of RAM are enough for the 120B model).

Highlights

  • C engine (Linux): mmap'd experts with the OS page cache as the expert cache — bounded memory by construction, graceful under pressure. YaRN RoPE, attention sinks, sliding-window ring KV, MXFP4/int4/int8 kernels, usage-aware --preload-gb warmup.
  • MLX engine (Apple Silicon): the same converted model directory, experts streamed through a byte-budgeted LRU in unified memory, container blobs consumed directly by MLX's quantized kernels (affine int4 and native mode="mxfp4").
  • Converter: torch-free, shard-by-shard streaming (peak disk ≈ container + one shard), resumable after interruption, lossless MXFP4 transcode.
  • Validation: both engines checked per-position against an independent NumPy reference across all dtypes in CI; MXFP4 decoding verified bit-exactly against transformers' official dequantizer and cross-checked against OpenAI's reference implementation.

Quickstart

git clone https://github.com/drmihirbrahme/midge && cd midge
make && pip install numpy tokenizers huggingface_hub
./midge convert openai/gpt-oss-20b models/gpt-oss-20b
./midge chat models/gpt-oss-20b

See the README for the Apple Silicon (./midge-mlx) usage, capacity planning (./midge plan), and troubleshooting.

Assets 2
Loading