v0.4.12

A Metal decode runs 1.4 to 1.8 times faster because the matvec stopped launching a block four times wider than the row it was reducing. Trying to remove the reduction underneath it as well is worth nothing, which is how this release also establishes that the matvec is no longer where a small model's decode goes.

Changed

• The matvec launches 32 threads a row on Metal rather than 128. TILE had been 128 since the kernel was written and it was never a decision about the matvec, it is the width every other kernel launches at. On an M4 the output head of a small model, 49152 rows of 576, was running at a tenth of what the same bytes read with no arithmetic on them. End to end that is 1.4 times on SmolLM2 135M, 1.8 on Qwen 2.5 0.5B and 1.5 on Llama 3.1 8B, same binary either side of the one constant. The logit corpus passes all thirteen cases on both widths, with every figure moving in the fifth or sixth significant digit because the additions happen in a different order.
• The same sweep now has its answer on CUDA. A 4090 at load 0.06 finds the same pattern, the head 1.85 times faster at 32 and the wide control preferring 128, but end to end it is worth about one per cent, so CUDA keeps 128.
• A shuffle reduction under the narrower block was written, measured and reverted. It is a wash on both small models and it changes the order the partial sums are added in, which is not worth paying for nothing.

What the negative results establish

Narrowing the block was worth 1.4 to 1.8 times and removing the reduction under it is worth zero, so what 128 threads a row was costing was never the reduction. It was the threads that arrived at a 576 column row with nothing in them.

On Metal, which runs one path whatever the model, decode time a token against bytes a token fits a slope of 29.4 ms a GiB and an intercept of 13.6 ms a token that does not move with the model. A thirty layer token is around 450 launches at the 19.6 us measured, which is 8.8 ms, so most of that intercept is submission and Metal decode on a small model is launch bound.

CUDA is a different picture and the first version of these notes got it wrong. Models under a gibibyte of layer weights decode through the fused kernel there, so SmolLM2 and Qwen are fused and the 8B is not, and a line through all three has an intercept that means nothing. With the path pinned, a fused SmolLM2 token is 2.16 ms against 5.58 unfused, and the fused path is already one launch a layer, so its submission is about 0.15 ms and its weights about 0.22 ms. The remaining 1.9 ms is the fused kernel's own grid and barriers. On the 8B fused and unfused are within 5 per cent, 12.4 against 13.0, because the bus is the rest.

So kernel tuning is finished as a way to move a small model on either backend, and what is in the way is different on each: submission on Metal, and the fused kernel's grid on CUDA.

What's Changed

• Launch the Metal matvec at 32 threads a row, and release 0.4.12 by @tamnd in #229

Full Changelog: v0.4.11...v0.4.12

v0.4.11

A fused session holds a fifth of the host memory it held yesterday.

The 1.2 GiB it carried was one pinned allocation, made to read the three integers the grid barrier writes, and not the compile it was blamed on. The first pinned allocation in a process costs that whatever its size. Reading the same words with an ordinary device to host copy takes SmolLM2 135M on a 4090 from 1468 MiB to 279 and Qwen 2.5 0.5B from 1481 to 550, both under llama.cpp on the same file in the same sitting, with decode and card figures unmoved.

The gpc section of docs/validation/bench.md was retaken on this build, rivals included, five runs each at a 514 token prompt and 128 generated.

model	molla host	llama.cpp host	molla decode	llama.cpp decode
SmolLM2 135M q8_0	279	444	490.4	894.3
Qwen 2.5 0.5B q4_K_M	550	807	383.2	752.0
Llama 3.1 8B q4_K_M	1044	5027	95.5	163.3

Source only, for the licensing reason in the changelog. Build with pixi run build.

What's Changed

• Cover the empty fused plan by @tamnd in #224
• Read the barrier words back without a pinned host buffer by @tamnd in #225
• Release 0.4.11 by @tamnd in #226

Full Changelog: v0.4.10...v0.4.11

v0.4.10

A decode runs a layer in one launch on CUDA. A Llama shaped layer used to be twelve launches and a thirty layer token 363 of them, and it is 33 now. On a 4090 at a 512 token prompt and 128 generated, SmolLM2 135M goes from 264.5 to 492.3 tokens a second and Qwen 2.5 0.5B from 304.8 to 367.8, both against llama.cpp at 888.9 and 767.9 and Ollama at 682.0 and 490.8 in the same sitting.

It is off for a large model and off on Metal, and both of those are measurements rather than caution. Llama 3.1 8B decodes at 81.8 tokens a second fused against 93.7 unfused, because a few hundred blocks is narrower than the block a row the unfused matvec launches and a model that reads five gibibytes a token is bound by that rather than by submission.

See CHANGELOG.md for the full entry.

What's Changed

• Say why a repack failed instead of what the file it did not replace looks like by @tamnd in #221
• One kernel a layer by @tamnd in #200
• Retake the gpc tables on the fused build and cut 0.4.10 by @tamnd in #223

Full Changelog: v0.4.9...v0.4.10

v0.4.4

Three of the kernels a layer launches were doing one flop per element, and every one of them can be done by the kernel in front of it for free.

A matvec thread block owns one output row and has that row in a register when its reduction finishes. Anything that reads only that row, and nothing else the same launch is still writing, is work the block can finish itself. The projection bias, the residual add and the gate half of a gated MLP are all exactly that, so all three are epilogues now rather than launches. A layer goes from fifteen launches to twelve on a model without qkv bias and from eighteen to twelve on one with it.

Decode on an RTX 4090, eight interleaved repetitions: SmolLM2 135M q8_0 from 282.2 tokens per second to 310.8, Qwen 2.5 0.5B q4_K_M from 244.9 to 309.2, Llama 3.1 8B q4_K_M from 99.8 to 101.4. The 8B barely moves because at 5.6 GiB a token its launches were already hidden behind its weight reads.

It is exact, so the logit corpus is byte identical to 0.4.3 on both backends and no tolerance moved.

Full changelog: https://github.com/tamnd/molla/blob/v0.4.4/CHANGELOG.md

What's Changed

• Fold the bias, the residual add and the glu into the matvec epilogue by @tamnd in #194
• Release 0.4.4 by @tamnd in #195

Full Changelog: v0.4.3...v0.4.4

v0.4.3

The decode matvec spent more of its time turning integers into floats than multiplying them, and there is a way to do that with no conversion instruction in it.

Float32(n) for a small integer compiles to a convert, and on NVIDIA a convert issues on the same unit as transcendentals at a quarter of the rate of a multiply. A byte of two four bit weights costs four of them, which is worth sixteen multiplies before any multiplying has happened, and the kernel only does four. A float32 whose exponent is 23 has a mantissa step of exactly one, so the representable numbers between 2^23 and 2^24 are the integers and their bit patterns are 0x4B000000 plus the integer. Nothing carries out of the mantissa, so the addition is an or. Or the weight in, subtract 2^23 back off, and the conversion is an integer or and a float subtract, both full rate on both vendors.

Decode on Llama 3.1 8B q4_K_M goes from 72.9 tokens per second to 99.7 on an RTX 4090 and from 2.0 to 2.9 on an Apple M4, against llama.cpp's 163.7 on the same card and file. The byte wide types do not move and were not expected to, because a signed byte load already sign extends in the hardware.

It is exact rather than close, so the whole logit corpus is unchanged in every digit on both backends and no tolerance moved.

Full notes in CHANGELOG.md.

What's Changed

• Turn a quant into a float with a bit pattern instead of a convert by @tamnd in #192
• Release 0.4.3 by @tamnd in #193

Full Changelog: v0.4.2...v0.4.3

v0.4.2

What's Changed

• chore: build the test suite without warnings by @tamnd in #189
• perf: index the scale plane with a shift, which doubles Metal decode by @tamnd in #190
• chore: release 0.4.2 by @tamnd in #191

Full Changelog: v0.4.1...v0.4.2

What's Changed

• chore: build the test suite without warnings by @tamnd in #189
• perf: index the scale plane with a shift, which doubles Metal decode by @tamnd in #190
• chore: release 0.4.2 by @tamnd in #191

Full Changelog: v0.4.1...v0.4.2

v0.4.1

What's Changed

• The first map_to_host in a process costs 1.3 GiB by @tamnd in #175
• Give back each tensor's pages once the card has them by @tamnd in #176
• The research on using MAX instead of rolling our own by @tamnd in #177
• Bound how far the read stage gets ahead of the drain loop by @tamnd in #178
• docs: one byte per weight is the memory problem and half the throughput problem by @tamnd in #179
• fix: the macOS memory column hid whichever weights the engine held by @tamnd in #184
• perf: pack the planar quant plane at four bits by @tamnd in #185
• docs: there is no map_to_host arena on Metal by @tamnd in #188
• chore: release 0.4.1 by @tamnd in #187

Full Changelog: v0.4.0...v0.4.1

v0.4.0

What's Changed

• The first numbers against llama.cpp and Ollama by @tamnd in #164
• The architecture that reaches two times llama.cpp at half the memory by @tamnd in #173
• chore: release 0.4.0 by @tamnd in #174

Full Changelog: v0.3.3...v0.4.0

v0.3.3

What's Changed

• The logit corpus on every backend by @tamnd in #162
• chore: release 0.3.3 by @tamnd in #163

Full Changelog: v0.3.2...v0.3.3

v0.3.2

What's Changed

• feat(engine): the whole forward pass on the device by @tamnd in #159
• A backend every command agrees on by @tamnd in #160
• chore: release 0.3.2 by @tamnd in #161

Full Changelog: v0.3.1...v0.3.2