Repository navigation
Releases
LookAtMyQuant 2: Qwen3.5-4B in 1.23 GB, packed for the Apple Silicon runtime
Qwen3.5-4B compressed to 1.23 GB by the LookAtMyQuant 2 recipe (2.3 bits per weight), in the packed format that runtime_mlx/fast_q35.py reads. The page has the recipe and the results: 330 of 540 exact-answer math problems, and 0.45 nats of KL from the original on held-out text.
weights.binandartifact.json: the packed model (formatopgraph-packed-1)config.json,tokenizer.json,tokenizer_config.json,chat_template.jinja: Qwen3.5-4B's ownrecovery.json: the packer's record for each matrix (runtime_mlx/pack_dense.py)
Put the files in one folder. On a Mac with mlx and mlx_lm, from the repository root:
gh release download lm2-qwen35-4b-1.2gb --repo ShinMegamiBoson/lookatmyquant -D q35-1.2gb
python -c "
import sys; sys.path.insert(0, 'runtime_mlx')
import fast_q35
from mlx_lm import generate
model, tok = fast_q35.load('q35-1.2gb')
prompt = tok.apply_chat_template([{'role': 'user', 'content': 'What is 17 times 23? Show the steps.'}], add_generation_prompt=True, enable_thinking=False)
print(generate(model, tok, prompt, max_tokens=200))
"With the drafter from the lm2-drafter-4bit release it generates 1.5 to 2 times faster, with the same text:
gh release download lm2-drafter-4bit --repo ShinMegamiBoson/lookatmyquant -D drafter-4bit
python runtime_mlx/bench_drafted.py q35-1.2gb drafter-4bitDSpine drafter for the 1.2 GB Qwen3.5-4B file, at 4 bits a weight
The drafter that runtime_mlx/drafter_q35.py generates with, stored at 4 bits a weight by pack_drafter.py: 464 MB against 1.83 GB at 16 bits.
weights.binandartifact.json: the packed drafter (formatopgraph-packed-1)config.json: the drafter's configuration, as SpecForge exported itpack_report.json: each matrix's coding error
Put the files in one folder and pass it as the drafter:
gh release download lm2-drafter-4bit --repo ShinMegamiBoson/lookatmyquant -D drafter-4bit
python runtime_mlx/bench_drafted.py path/to/packed drafter-4bitOn 60 held-out prompts it kept 4.35 tokens per check, against 4.38 for the 16-bit drafter, with the same text.
It was trained with SpecForge's DSpine objective, starting from z-lab/Qwen3.5-4B-DFlash, on the 1.2 GB file's own greedy answers. It runs beside the packed 1.2 GB file, the lm2-qwen35-4b-1.2gb release.