mashbox 0.1.13

Changes since 0.1.12.

• The kernel package is mashbox_engine_kernels-0.1.13-py3-none-any.whl, an asset of this release (sha256 57aafadae34cc3da52ca2ab875c2e84d45cdffe88bbd487df7d0062bce7b1608), loaded with no flag on TPU v6e. A decode call at one input row now runs on its own kernel class. On one TPU v6e chip, per call: a 9728 x 2560 Q4_K weight takes 22.6 us instead of 31.9 us, a 9728 x 2560 Q2_K weight 25.5 us instead of 57.6 us, and a 151936 x 2560 Q6_K output head 474 us instead of 544 us. In a check of 17 weight types over 10 shapes, at 1 to 8 and at 64 input rows, every output is bit for bit the same as with 0.1.12. On the files measured, peak memory and time to the first token are no higher than with 0.1.12.
• Every GGUF file's short check against llama.cpp ran again on 0.1.13's code and kernel package: 33 files pass, 20 of them on a TPU v6e-1 and 13 on one chip of a TPU v6e-4. Every GGUF row's steady decode speed at 1 user is measured on this code and kernel package, on the same machine as its check.
• CONTRIBUTING.md, AGENTS.md and the pull request template give one path for contributors: python3 -m pytest tests/contracts is the check every pull request gets, then the tests/cpu areas python3 tools/ci_select.py --base origin/main selects. CONTRIBUTING.md now says how to add a GGUF architecture or ask for a GGUF file, CI.md says where chip results are kept, and the README links the release notes.
• The Qwen3.8-27B compile caches for TPU v6e-1 and v6e-4 made for 0.1.11 are used by 0.1.13 as they are (Hugging Face revision 98139d38): nothing that shapes its programs changed.
• 37 models and files run in mashbox: 4 MASHBOX builds and 33 GGUF files of 20 models.

Install on a TPU VM: curl -fsSL https://mashbox.gg/install.sh | sh (or python3 -m pip install uv && python3 -m uv tool install --python 3.12 "mashbox[tpu]")

mashbox 0.1.12

Changes since 0.1.11.

• The kernel package is mashbox_engine_kernels-0.1.12-py3-none-any.whl, an asset of this release (sha256 cd46db5e3e46d8f6ee2c546477f07541a045d0bfab926b5c6c8be11dca210d62), loaded with no flag on TPU v6e. A decode call on a quantized weight now runs on a wider N tile when the step cost model says that at least halves the cost. On one TPU v6e chip, one input row, per call: a 9728 x 2560 Q4_K weight takes 31.4 us instead of 86.8 us, and a 151936 x 2560 Q6_K head takes 541 us instead of 1471 us. Every output value is unchanged, and at 1 to 8 input rows the outputs equal 0.1.11's bit for bit. Peak memory is up by under 3.1 MB on the files measured, and their time to the first token is within 0.02% of 0.1.11's.
• Qwen3.5 and Qwen3.8 GGUF files read and write their linear-attention state in place at decode and speculation verify, instead of gathering their rows from the whole slot pool and scattering them back at every layer group. A TPU v6e compile at 8 slots x 2048 shows 2 pool scatters per decode program before and 0 after, and XLA's bytes accessed per 8-row decode call fall from 1.42 to 1.18 GiB. Generated ids, slot state and logits are byte-identical before and after.
• Every GGUF file's short check against llama.cpp ran again on 0.1.12's code and kernel package, on one chip of a TPU v6e-4: 33 files pass. bartowski's granite-3.3-2b-instruct Q4_K_M still differs from llama.cpp and leaves the list, and so does bartowski's Mistral-7B-Instruct-v0.3 Q3_K_M, which was listed on a CPU check alone. A row now shows its CPU-check line only when that check ran on this code.
• Every GGUF row now shows a steady decode speed at 1 user on its chip. The Qwen3.5 and Qwen3.8 rows are measured on this code; the others were measured with this kernel package on code whose executed path is the same.
• mashbox.gguf.resolve is now mashbox.gguf.pick, with the same code. Code that imported mashbox.gguf.resolve imports mashbox.gguf.pick; the function is mashbox.gguf.pick.pick.
• LFM3-1.7B for the Apple Neural Engine (Core ML) is on the supported-models page, with its own docs page. mashbox run does not start it; the page shows how to run it from Swift.
• The Qwen3.8-27B compile caches for TPU v6e-1 and v6e-4 made for 0.1.11 are used by 0.1.12 as they are (Hugging Face revision 98139d38): nothing that shapes its programs changed.
• 37 models and files run in mashbox: 4 MASHBOX builds and 33 GGUF files of 20 models.

Install on a TPU VM: curl -fsSL https://mashbox.gg/install.sh | sh (or python3 -m pip install uv && python3 -m uv tool install --python 3.12 "mashbox[tpu]")

mashbox 0.1.11

Changes since 0.1.10.

• The kernel package is mashbox_engine_kernels-0.1.11-py3-none-any.whl, an asset of this release (sha256 70de57d262e2e04e2a41d258eaddc30c511b852b4070c4413048d1baab2f53ec), loaded with no flag on TPU v6e. It adds six quantized types to the kernel path: Q2_K, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS and NVFP4, which ran the exact path before. IQ1_S and IQ1_M stay on the exact path. On one v6e chip at batch 1, one small file per new type (Qwen3-0.6B; Qwen3.5-4B for NVFP4) decodes at 69 to 388 tokens per second with the package and at 1.2 to 9.0 on the exact path. It is a compiled binary under the MASHBOX Kernel Licence 1.0 (LICENSE-KERNELS). The rest of mashbox is Apache-2.0.
• A layer scan reads a quantized weight's layer from the whole stack inside the kernel call, so XLA no longer copies each layer's planes before every call. On one v6e chip, Qwen3.8-27B UD-Q4_K_M with speculation off decodes at 36.4 ms per step at batch 1 (53.8 ms with the 0.1.9 package) and 51.9 ms at batch 8 (69.4 ms). A stacked weight now takes its kernel class even where 0.1.9 named an exact override for its shape: on 3 of that file's 35 (type, shape) pairs the output differs from the override's by at most 3.04e-08 (relative L2), and it is equal on the other 32.
• The K/V cache is updated in place for every model, in a layout that does not pad. XLA on TPU no longer copies the whole cache at each step: 6 whole-cache copies per program become 0 on every model that had them (decode, sampled decode, prefill, speculation verify). Greedy ids equal 0.1.10's on Qwen3-4B, Llama-3.2-1B, Qwen3.5-4B and gemma-3-1b on one v6e chip.
• Gemma's sliding-window layers keep a ring of min(window, context) positions per slot. The decode of unsloth/gemma-4-12b-it Q4_K_M at 8 slots x 2048 now fits one v6e chip, with a peak of 16.04 GiB (12.39 GiB with the ring); 0.1.10 asked 35.33 GiB of temporaries against 31.24 GiB.
• Every compiled program changed. A compile cache, host-cache entry or AOT program table made by 0.1.10 is not used by 0.1.11; it is made again. The Qwen3.8-27B compile caches for TPU v6e-1 and v6e-4 are built from 0.1.11's programs (Hugging Face revision 98139d38).
• Every GGUF file's short check against llama.cpp ran again on 0.1.11's code and kernel package, one file per TPU v6e chip. A row now says only whether the check passed, and no row shows a first-request speed. Six rows show a steady decode speed with its condition (one warm request, then the median of 5 requests of 128 new tokens, 1 user). Two rows are new: byteshape Qwen3.8-27B IQ2_XXS and cdiamond Qwen3.8-27B iMatrix NVFP4 MTP.
• The README, llms.txt and the supported-models page list the five weights-only repos (Real-ESRGAN, Audio2Face-3D v3.0, granite-speech 4.1 2B, Z-Image-Turbo and maya1), which mashbox run does not start, each with a link to how it runs.
• 39 models and files run in mashbox: 4 MASHBOX builds and 35 GGUF files of 21 models.

Install on a TPU VM: curl -fsSL https://mashbox.gg/install.sh | sh (or python3 -m pip install uv && python3 -m uv tool install --python 3.12 "mashbox[tpu]")

mashbox 0.1.10

Changes since 0.1.9.

• TRELLIS.2 (image to 3D): mashbox run trellis-2 --image object.png --out object.glb turns one picture of an object, with the background cut out, into a textured 3D mesh (a GLB file). It runs on one TPU v6e chip, and the GLB export runs on the VM's CPU in the same command. Install it with the trellis-2 extra; the model page has the commands and the weights it downloads. The card is mashbox-labs/TRELLIS.2-TPU.
• mashbox run, mashbox download and mashbox inspect take a Hugging Face safetensors repo (OWNER/NAME) or a folder that holds one. This path reads dense text models of the llama, Mistral, Qwen2, Qwen3 and Qwen3.5 architectures, on one chip. Other architectures, routed experts, quantized weights and a model that fits no chip stop with a one-line reason.
• mashbox run, download and inspect resolve every target the same way. A catalog model is found by its id, its name or its Hugging Face repo (mashbox-labs/..., with or without https://huggingface.co/), and a GGUF entry of the catalog by its id. A target that is none of a catalog entry, a GGUF file or repo, a BSHJ file or a safetensors model stops with one line, and an unknown name exits with code 2 (before: 1).
• A GGUF entry's id runs the exact file the entry pins, at its revision: the Hub's sha256 of each shard must be the entry's, and the downloaded bytes are checked again. A file changed or gone upstream stops the run with one line that names the unpinned command. mashbox run repo:tag still runs the repo's current file.
• On TPU, a start with no kernel package for its chip family and JAX says in one line that every quantized type runs on the exact path.
• On an install without JAX (the client that install.sh puts on a laptop), run, download and inspect stop with one line before any download: for a TPU model it gives mashbox tpu up <model> and the Cloud TPU VM install line. mashbox env prints jax as null.
• mashbox run hy-nomas without its extra stops with one line naming the install line, before the chip check and before any download.
• A download that fails (offline, an HTTP error, a connection or read failure, a full disk) stops with one line naming the file, its repo and revision, and why. Files already downloaded are kept.
• A GGUF file with no tokenizer (no tokenizer.ggml.model key) stops run with one line before the weights load, and inspect says it does not run here. llama.cpp does not load such a file either.
• Qwen3.8-27B has a compile cache for TPU v6e-4 at tensor parallel 4 (the default on that host), next to the v6e-1 one. With the weights on disk, a v6e-4 start is ready in 63.55 s with it and 229.75 s without it.
• --speculation dflash --drafter-dir DIR: a trained block drafter (DFlash) guesses a block of tokens in one pass from the model's own layer outputs. Qwen3.8-27B only, on one chip; not on GGUF files. The model checks every token the drafter guesses, but on a TPU v6e-1 the output can differ from decoding without speculation in places. auto never picks it.
• With the kernel package in use, F16 weights (GGUF F16 tensors and Hugging Face float16 weights, except stacks of experts) are held on the chip as their float16 values instead of being rebuilt from their bytes at every call. On TPU v6e a few outputs of a call can differ from the exact path's, by at most 0.016. None of the 28 GGUF files in the catalog has an F16 weight.
• Qwen3.8-27B reads its int8 embedding row at decode from the whole tile, with no relayout of the table at each step.
• A GGUF gemma4 file that stores feed_forward_length per layer stops with a one-line reason instead of a traceback.
• The kernel package is unchanged: mashbox_engine_kernels-0.1.9-py3-none-any.whl from the 0.1.9 release (sha256 56b0d2a8fd470a0d68ce744a14a0653ddf132bc2936317d02e555a98afaee178), loaded with no flag on TPU v6e. It is a compiled binary under the MASHBOX Kernel Licence 1.0 (LICENSE-KERNELS). The rest of mashbox is Apache-2.0.
• 32 models and files run in mashbox: 4 MASHBOX builds and 28 GGUF files of 19 models.

Install on a TPU VM: curl -fsSL https://mashbox.gg/install.sh | sh (or python3 -m pip install uv && python3 -m uv tool install --python 3.12 "mashbox[tpu]")

mashbox 0.1.9

Changes since 0.1.8.

• The kernel package is mashbox_engine_kernels-0.1.9-py3-none-any.whl from this release (sha256 56b0d2a8fd470a0d68ce744a14a0653ddf132bc2936317d02e555a98afaee178). On TPU v6e, mashbox run loads it with no flag, as in 0.1.8. It is a compiled binary under the MASHBOX Kernel Licence 1.0 (LICENSE-KERNELS). The rest of mashbox is Apache-2.0.
• The kernel package covers weight shapes by class instead of a fixed list. One class covers every weight shape of its types inside the range it states, and a shape the 0.1.8 package listed runs the same program as before. A shape outside every class takes the exact path, as before.
• Q4_0, Q4_1 and Q5_0 weights now run on kernels. With 0.1.9, every 2-D quantized weight of the 28 GGUF files in the catalog has a kernel.
• mashbox loads only the kernel package its engine_kernels row names. Another package, such as 0.1.8, is refused with one line, and the exact path runs.
• A compile cache is unpacked only on the deployment it was built for: the chip type and count (v6e-1, v6e-4) and the tensor-parallel size. Any other start logs one line and compiles at first use, as it does with no cache.
• 31 models and files run in mashbox: 3 MASHBOX builds and 28 GGUF files of 19 models.

Install on a TPU VM: curl -fsSL https://mashbox.gg/install.sh | sh (or python3 -m pip install uv && python3 -m uv tool install --python 3.12 "mashbox[tpu]")

mashbox 0.1.8

Changes since 0.1.5, including the changes tagged v0.1.6 and the changes of 0.1.7, which did not go to PyPI.

• On TPU v6e, mashbox run loads the MASHBOX kernel package with no flag. The first TPU start downloads mashbox_engine_kernels-0.1.8-py3-none-any.whl from this release, checks its sha256 against the engine_kernels row in models.json, and keeps it in ~/.cache/mashbox/kernels/ (or under MASHBOX_CACHE), so later starts read it from the cache. The row names jax 0.11.2 and libtpu 0.0.48, the versions mashbox[tpu] installs. Off TPU, or on a chip or JAX the row does not name, nothing is downloaded and the exact path runs. MASHBOX_GGUF_KERNELS=off turns the kernels off.
• The kernel package is a compiled binary under the MASHBOX Kernel Licence 1.0 (LICENSE-KERNELS). The rest of mashbox is Apache-2.0.
• With the kernel package loaded, a quantized GGUF weight of type IQ3_S, IQ4_NL, IQ4_XS, Q3_K, Q4_K, Q5_K, Q6_K or Q8_0 runs on a kernel when the package lists its exact type and shape. Every other weight runs on the exact path.
• mashbox reads from a GGUF file's metadata whether it has routed experts (mixture of experts). Their experts run through one routed-expert op, on the kernel package where it covers them. A file with routed experts runs only when the catalog has a passing chip row for that exact file, matched by its sha256. Any other such file stops with one line before anything loads, including a file written under a dense architecture's name (as llama.cpp writes Mixtral). A Hugging Face model folder with routed experts is refused at its config.
• /health has a routed_experts field: the path the routed experts took (kernel, exact or mixed), the kernel's version, and why any call ran on the exact path. It is null for a dense model. mashbox inspect shows the path a file's experts will take.
• An F32 weight that the exact path multiplies whole, such as a mixture-of-experts router, is kept in memory as float32 values. Outputs are unchanged.
• --draft-model refuses a mixture-of-experts draft model.
• A GGUF file that holds only a draft head for speculative decoding, not a whole model, stops with one line that says so.
• The GGUF tokenizer reads the minicpm5 and granite-docling pre-tokenizer names and splits text as llama.cpp does for them.
• llms.txt lists every GGUF file in the catalog.
• Speculative decoding for every architecture with a verify pass. mashbox run takes --speculation auto|off|mtp|ngram|draft and --draft-k K; --mtp K stays. --draft-model FILE (a GGUF file or repo) drafts for a GGUF target with a smaller model. With auto, a drafter turns on only where it passed its checks for that model on that chip. No drafter has passed them on a chip in this release, so auto decodes without speculation. See docs/speculation.md.
• Speculation runs inside the batch. A speculation call that fails before it reaches the chip is replayed with plain decoding, and speculation stays off after it.
• Copy from the conversation (--speculation ngram) is tested on GGUF files of the llama, qwen2, qwen3 and qwen35 architectures. It also runs on exaone and granite files.
• GGUF files whose layers use different block types (Q4_K_M, dynamic quants) run their layers in one scan, so a compiled program no longer grows with the layer count.
• 26 models and files run in mashbox: 3 MASHBOX builds and 23 GGUF files of 18 models. The README and the supported models page list them, rendered from models.json, and mashbox models groups them by publisher. A GGUF row shows a decode speed only when it was measured on that exact file.
• A file that is not a llama.cpp model file stops with one line that says so.
• mashbox tpu up --chip: for a chip type the model runs on, it names the host to ask for ("On v6e: use --chip v6e-1"). The price line names the chip it creates. For a model that runs on a CPU, such as Track2Rig, it says no TPU is needed and prints the install and run lines.
• A result's condition is printed wherever its number is: Hy-Nomas on one TPU v6e chip reads 2.46 s of compute per second of video, main model quantized.
• The app and the module pages use the mashbox.gg fonts and colours, light and dark. The fonts ship with their OFL licence texts.

Install on a TPU VM: curl -fsSL https://mashbox.gg/install.sh | sh (or python3 -m pip install uv && python3 -m uv tool install --python 3.12 "mashbox[tpu]")

mashbox 0.1.5

• Hy-Nomas on one TPU v6e chip (v6e-1) is listed in models.json, llms.txt and the docs with its measured speed: 2.46 s of compute per second of video. This version of mashbox run starts on a v6e-4. Chips nobody has run yet read "Not tested yet".
• Hy-Nomas: the live player reads each 40-frame part in place, so the first frame after a part seam is no longer late on a busy host. The frames are pixel for pixel the same.
• mashbox run MODEL --port 0 binds a free port, and the "listening" line and the --ui address name that port.
• Ctrl-C in mashbox run no longer ends in a rare abort at exit: the server's supervisor thread finishes before it closes.

Install on a TPU VM: curl -fsSL https://mashbox.gg/install.sh | sh (or python3 -m pip install uv && python3 -m uv tool install --python 3.12 "mashbox[tpu]")