ARustyCoder93: how it was built ("grow thy experts")

ARustyCoder93 is a 93-of-256 routed-expert cut of DeepSeek-V4-Flash, in GGUF, built for local agentic coding on a single 96 GB GPU. It is not a DeepSeek release; it is a derived model, and this repository is the complete recipe: the llama.cpp patch, the scripts, the run configs, the census files, and every checksum needed to rebuild it byte-for-byte.

Model weights: https://huggingface.co/ayourtch/ARustyCoder93

The idea in one paragraph

DeepSeek-V4-Flash routes each token through 6 of 256 experts per layer. Only a fraction of the experts matter for a given domain. Instead of measuring which ones with the full 291B model resident (a "truth census", which needs ~150 GB), we start from a random 8 experts per layer, keep the full 256-wide router as extra tensors, and let a patched llama-imatrix record which experts the router wants (its top-6 over all 256, including the absent ones) while running a small text corpus through the tiny model. Rebuild at 16 experts from those wants, census again, then 32, then 64. Peak resident memory is the keep-64 model (~45 GB), never the full one. Two grows were run — one on a code/systems corpus, one on a general-domain corpus — and ARustyCoder93 is the union of the two resulting keep-64 expert sets: 78..93 experts per layer (mean 87.3), padded to 93 in the smaller layers by merged rank, because a GGUF needs one expert count for all layers.

Why the union: each keep-64 alone predicts held-out text well (perplexity) but cannot finish generating. Both loop to the token cap on ordinary coding prompts. The union does generate cleanly. Perplexity never asks whether a model can produce text.

What is in this repository

path	what
llama.cpp-patch/wants-census.patch	54 added lines in 6 files, diff against llama.cpp 22b8e310b921d568e013e4533002be5a8fe53f17 (branch name wants-census): KV deepseek4.expert_router_count, optional tensors blk.N.ffn_gate_inp_full / blk.N.exp_probs_b_full, deepseek4 graph emits ffn_moe_wants-<layer>, llama-imatrix counts them into <out>.wants.tsv
scripts/apply_wants_patch.py	the same change as a source-edit script (how it was originally applied)
scripts/prune_experts.py	the pruner: rewrites a deepseek4 GGUF keeping K experts per layer (raw bytes, any quant), remaps router rows / bias / hash tables. Selection via env: PRUNE_RANK=low|high|random, PRUNE_COUNTS=<imatrix>, PRUNE_WANTS=<tsv>, PRUNE_FULL_ROUTER=1, PRUNE_SEED
scripts/grow.sh	general-domain grow driver (random keep-8 → 16 → 32 → 64), as run
scripts/grow-coder.sh	code-domain grow driver, as run (writes its own config.txt provenance)
scripts/grow2.sh	resume-after-crash variant of grow.sh used for the general run's 16 → 64 stages
scripts/merge_wants.py	union/merge of two censuses (see "Merge rule")
scripts/compare_sets.py	per-layer Jaccard of a wants census vs a truth imatrix census
scripts/ppl-paired.py	paired per-chunk perplexity test over two llama-perplexity outputs (stdlib only)
scripts/grow-ppl-paired.sh, grow-ppl-ci.sh	drivers that ran the perplexity comparisons
scripts/code-smoke.py	the six-prompt generation smoke test
results/	the raw perplexity outputs, the smoke-test responses, and RESULTS.md recomputing every number quoted below
llama.cpp-patch/BASE_COMMIT.txt	the llama.cpp commit + branch name the patch applies to
scripts/build_corpus.py, build_coder_corpus.py	corpus builders (sources listed below)
configs/grow-general/, configs/grow-coder/	per-stage census + prune logs and the driver stdout (grow.log, grow-coder.log) of the two grows; grow-coder/config.txt is the full run record
configs/grow-union/	config.txt + prune.log of the ARustyCoder93 build itself
configs/grow-merge/	the earlier keep-128 union+filler experiment (deleted model; kept for the record)
wants/coder-census-64.wants.tsv	final code-grow census (input A)
wants/general-census-64.wants.tsv	final general-grow census (input B)
wants/merged-wants.tsv	the exact ranking file the model was pruned with
PROVENANCE.md	every sha256, version and commit
MODEL_CARD.md	the HuggingFace model card
LICENSE, LICENSE-DeepSeek-V4-Flash	MIT for this repo; the upstream MIT notice that travels with the weights

Reproduce

Hardware used: one NVIDIA RTX PRO 6000 Blackwell (96 GB), driver 610.57.04, CUDA 13.3, Ubuntu; llama.cpp built with -DGGML_CUDA=ON Release. Python 3.12.3 with gguf==0.19.0, numpy==2.5.2.

1. Source model: antirez/deepseek-v4-gguf file DeepSeek-V4-Flash-MXFP4Experts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-mxfp4-0731.gguf (155,976,458,848 bytes; sha256 in PROVENANCE.md). Kept experts are copied byte-for-byte from it, so the quantization is inherited (MXFP4 experts, Q8 attention/shared/out, F16 compressor/indexer).
2. git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp && git checkout 22b8e310b921d568e013e4533002be5a8fe53f17 && git apply /path/to/wants-census.patch, build with CUDA. (The patch is only needed for the census stages; the final model runs on stock llama.cpp — the full-router tensors are not written into it.)
3. Build the corpora (or skip and use ours by checksum; PROVENANCE lists the source repos and commits). The builders expect a working directory that holds checkouts book/, ripgrep/, CheatSheetSeries/ plus man.txt, gsm8k.jsonl, agentic.txt, and ~/llama.cpp for the C/C++/Python sources: python3 build_coder_corpus.py and python3 build_corpus.py. The general corpus mixes in GSM8K and a set of our own reasoning traces; it is not redistributed here (checksums are). All scripts hard-code the paths they ran with (~/hf-venv/bin/python, ~/llama.cpp/build-mtp/bin, ~/models, ~/corpus); edit the top of each before running.
4. Grow, twice: bash grow-coder.sh (code corpus) and bash grow.sh (general corpus). Each stage: prune → llama-imatrix -c 2048 -b 2048 -ub 2048 --chunks 100 → next size from <census>.wants.tsv. Seed: PRUNE_SEED=1 for the random keep-8.
5. Merge: python3 merge_wants.py coder-census-64.wants.tsv general-census-64.wants.tsv 64 merged-wants.tsv — prints per-layer union sizes; the max (93) is the keep count.
6. Prune the final model: PRUNE_WANTS=merged-wants.tsv python3 prune_experts.py SRC.gguf arustycoder93.gguf 93 6 → 62,213,283,424 bytes, sha256 90a4c8a8a32fc664d822f1df093d5e228111c94eebf9398ae5c5737391ecb058.
7. Serve: llama-server -m arustycoder93.gguf -ngl 99 -c 262144 -fa on (~67 GB resident with 256K context; ~46 tok/s decode, ~470 tok/s prompt on the PRO 6000).

Merge rule (and the trap in it)

Per layer: normalise each census to sum 1 (so neither dominates on raw count volume), add them, then give every expert in the union of the two top-64 sets a +1e7 bonus so it cannot be displaced. Without the bonus, plain normalised-sum ranking silently dropped 1-2 union members in 6 of 40 layers: an expert strong in one census and near-zero in the other loses to experts moderate in both. Verified 0/40 misses with the bonus.

merged-wants.tsv is the original file the model was pruned with; that merge was run as an inline one-off, and merge_wants.py is its re-implementation. Verified: for all 40 router layers the top-93 set from merge_wants.py is identical to the top-93 set of merged-wants.tsv (the two files differ in scale, and in tie order below rank 93 because the original rounded to integers). Rebuilding from either file gives the same model.

Numbers behind the choices (all on the machine above, 2026-08-15/16; recompute from results/)

• Random keep-8 census already lands mean Jaccard 0.415 vs the full-model truth top-64 (chance ~0.14); it climbs 0.549 → 0.659 → 0.768 over the 16/32/64 stages.
• Grown keep-64 beats a truth-census keep-64 on held-out perplexity, paired per chunk: 3.742 vs 3.838, t = -3.69, df 67, p = 0.00045 (general heldout); replicates on the code heldout (p = 3e-13). Growing on code specialises: coder-grown 2.964 vs general-grown 3.210 on code heldout (p = 0.004).
• Overlap between the two final keep-64 sets: 40.7/64 per layer (Jaccard 0.469; chance 16.0). Union 78..93, mean 87.3.
• Both keep-64 models loop to the token cap on ordinary coding prompts. The union generates: 6/6 coding asks (Rust, Python fix, C explanation, JS refactor, tool call, multi-file plan) finish cleanly, 86-1993 completion tokens (results/code-smoke-keep93.json).
• Known limit: one harder single-shot prompt (a generic LRU cache) fell into a verbatim repetition loop in its reasoning to the 12k cap (results/code-smoke-keep93-lru-loop.json). A DRY sampler breaks the loop but corrupts identifiers in code. No standard benchmark has been run on this cut; treat it as an experiment that happens to be useful.

Not done / open

• keep-88 (pure union without filler), keep-80/96 to locate the coherence cliff, and a keep-128-with-random-filler control (if that also talks, expert identity matters less than count past a threshold).
• A generation smoke test at every grow stage — it would have caught the keep-64 looping at keep-8.
• A coding-shaped benchmark. Perplexity and an internal 92-question reasoning set are the wrong instruments for agentic coding.

License

Scripts and patch in this repository: MIT (LICENSE). The llama.cpp patch is a diff against MIT-licensed llama.cpp. The model weights inherit the upstream terms: DeepSeek-V4-Flash is released under MIT by DeepSeek (notice in LICENSE-DeepSeek-V4-Flash, shipped with the weights too); the source GGUF repo antirez/deepseek-v4-gguf is tagged MIT on HuggingFace. ARustyCoder93 is a derivative made by pruning. DeepSeek did not produce or endorse it.