Skip to content

v0.0.5

Latest

Choose a tag to compare

@github-actions github-actions released this 19 Nov 20:38
· 36 commits to main since this release
5b92da4

Vx 0.0.5 runs programs on Intel GPUs. Two language models written in Vx decode as fast as the usual servers for them, or faster. A 30B Qwen3 on an M4 decodes 1–4% faster than llama.cpp, and GLM-5.3 on 8 H200 GPUs decodes 7% faster than SGLang. The new types Distinct and DisjointRanges let parallel blocks write through a table of indices with no unsafe. The release also adds low-precision matmuls on the tensor cores of NVIDIA GPUs, and new channels and a semaphore in std::sync.

Highlights

Language models written in Vx

Qwen3-Coder-30B-A3B on a 32 GB M4 (vx-lang/qwen-vx), against llama.cpp 0.7.0 (build 11669, Metal). Generated tokens a second:

Prompt tokens llama.cpp Vx
60 45.0 46.8
2,014 40.6 41.3
8,005 31.6 31.8
16,026 22.4 22.5
  • The setup. 256 generated tokens, decode time only, the best of 2 runs (3 at 16,026). Both programs use Q4_0 weights with a 6-bit output matrix, an f16 KV cache and f32 arithmetic, with batch 1 and greedy decoding. No drafts.
  • The weight files. llama.cpp ran unsloth's Q4_0 GGUF. It stores the router in F32, so llama.cpp reads 1.87 GB for each token, and Vx 1.84 GB. The 4-bit weights of the two files have about the same perplexity (3.692 and 3.688).
  • The prompt. llama.cpp reads a 60-token prompt twice as fast as Vx. From 2,000 tokens the two are about the same.
  • What Vx brought. Each kernel of the model is Vx code, and the kernels match llama.cpp's hand-written Metal kernels. The GPU picks each token, so the CPU reads only every eighth token. That saves 0.25–0.32 ms for each token.
  • Speculative decoding: 1.14 times as fast as llama.cpp. Vx drafts up to 7 tokens from the context: the tokens after the longest earlier match. The main model then checks all of them in one pass, and keeps each token that it would have chosen itself. So the output is the same as without drafts. Over 12 code prompts, this is 1.14 times as fast as llama.cpp's llama-lookup, which drafts in the same way and checks the same 8 positions at most. On edits of a file, Vx with drafts is 1.84 times as fast as without them. The design is in docs/discussions/predictive_pipelining.md (#3676).
  • Memory. On the Mac, the GPU reads the mapped weight file of the model in place, and GPU memory stays resident (#3606). With --machine fleet/m4-uma.vx, a borrowed tensor reaches a kernel with no copy (#3502, #3510, #3507). These are why the 30B model fits on a 32 GB M4.
How we measured the Qwen numbers: the setup, the limits and all 96 runs

Measured on 2026-11-19 on one Mac. All 96 runs of table 1 are in the CSV at the end.

Results in short

Comparison (geometric mean over 12 prompts) Speedup Prompts where qwen-vx is faster
qwen-vx against llama.cpp, both without drafts 1.06 12 of 12 (1.03 to 1.12)
qwen-vx --draft 7 against qwen-vx without drafts (same output) 1.19 7 of 12 (0.99 to 1.92)
qwen-vx --draft 7 against llama-lookup 1.14 9 of 12 (0.93 to 1.40)
qwen-vx --draft 7 against the faster llama.cpp program for each prompt 1.05 9 of 12 (0.93 to 1.19)

From a context of 8,000 tokens, the two programs decode at the same speed (table 3).

Held constant

Both programs
Machine Apple M4 (Mac16,12), 10-core GPU, 32 GB, macOS 27.0.1. A Metal kernel reads at most 104 GB/s
Model Qwen3-Coder-30B-A3B-Instruct: 48 layers, 8 of 128 experts for each token
Weights Q4_0 (4-bit codes, an f16 scale for each 32 weights) for attention and experts
KV cache f16
Arithmetic f32 activations and f32 sums
Batch, sampling 1 sequence, greedy (temperature 0)
Prompts the same 12 files. Each one holds the chat markup, and neither program adds a template
Output 512 tokens. Neither program stops at end-of-text
Clock decode only: ms for each token after the first. Prompt, load and compile time are not in it
Runs 2 rounds of all 48 runs, the second in reverse order. Before each run, no process above 20% CPU, then 60 s idle

Different

llama.cpp qwen-vx
Program llama.cpp 0.7.0 build 11669 (Homebrew), Metal, all layers on the GPU, 4 CPU threads qwen-vx a50a604, compiled by a local debug vxc of Vx 9d7cf3a49 (main with #3667, #3673 and #3674), not the 0.0.5 release binary
Weight file unsloth Qwen3-Coder-30B-A3B-Instruct-Q4_0.gguf convert.py --q4 --importance --q6-lm-head from the bf16 model
Router F32 bf16
Expert down, layers 0–5 Q4_1 Q4_0
lm_head Q6_K (6.56 bits a weight) 6-bit codes, an f16 scale for each 32 weights (6.5 bits)
Norms F32 bf16
Bytes read for each token about 1.87 GB about 1.84 GB
Token choice logits go to the CPU, then the greedy sampler argmax on the GPU
Flash attention auto. What it chose was not recorded a port of llama.cpp's one-token kernel
Context size 8,192 prompt + 512

The model stores its router in bf16. So the F32 router of the GGUF holds the same values at twice the
size. The 32 MB extra for each token costs llama.cpp about 0.31 ms at 104 GB/s. That is about a fifth
of the gap in table 1.

The two files round the weights differently, so the programs generate different text. Perplexity
with one program (reference.py), lower is better:

Weights MetalImage.cpp calls.rs
bf16 3.6813 5.9254
qwen-vx Q4_0 3.6881 5.9664
unsloth Q4_0 codes in a qwen-vx file 3.6918 5.9535

Table 1: 12 code prompts, 512 tokens

ms for each token: the geometric mean of 6 runs (3 prompts, 2 rounds), and the fastest and slowest
run. Prompt lengths in tokens: edit 1,477–2,101, complete 1,439–1,654, write 46–59, explain
1,280–1,747.

Prompts llama.cpp llama-lookup qwen-vx qwen-vx --draft 7
Edit a file 25.12 (24.43–27.08) 12.52 (12.07–12.97) 24.06 (23.49–24.64) 13.13 (12.28–14.01)
Complete code 25.33 (24.61–26.40) 26.27 (24.52–29.95) 23.64 (23.45–23.95) 23.62 (23.27–23.97)
Write new code 23.38 (22.46–24.37) 29.30 (26.93–30.81) 21.78 (21.72–21.86) 21.82 (21.77–21.86)
Explain code 25.09 (24.26–26.10) 26.15 (23.72–27.48) 23.69 (23.25–24.09) 21.71 (19.91–22.69)
All 12 24.72 22.41 23.27 19.58

Between the two rounds of one prompt, plain llama.cpp changed by up to 8.6% (median 2.6%), and qwen-vx by
up to 1.2% (median 0.3%).

Table 2: drafts

Both programs check at most 8 positions in one pass (--spec-draft-n-max 8, --draft 7). The counts
are for the 3 prompts of each kind, and were the same in both rounds.

Prompts llama-lookup accepted / drafted qwen-vx accepted / drafted qwen-vx accepted for each check
Edit a file 1,316 / 1,505 (87%) 1,301 / 1,330 (98%) 6.7
Complete code 604 / 1,840 (33%) 68 / 124 (55%) 1.8
Write new code 333 / 2,036 (16%) 21 / 54 (39%) 1.2
Explain code 538 / 1,754 (31%) 257 / 303 (85%) 4.8

qwen-vx drafts only after a match of 6 tokens or more. After two checks in a row that accept
nothing, it waits for a longer match. llama-lookup drafts at almost every step. That is why it is slower than plain llama.cpp on
new code. With and without drafts, qwen-vx gave the same text in all 24 pairs.

Table 3: plain decoding by context length

256 tokens after a prompt of plain text, no chat template. ms for each token, each round:

Prompt tokens llama.cpp qwen-vx
60 22.24, 24.54 21.36, 21.46
2,014 24.62, 24.70 24.19, 24.38
8,005 31.68, 31.69 31.41, 31.82
16,026 44.74, 45.14, 48.14 44.38, 50.51, 47.02

This table used an earlier qwen-vx (vx-lang/qwen-vx#60 and vx-lang/qwen-vx#61, before the drafts), with a debug vxc whose
commit was not recorded. The table at the top of the notes gives the faster round of each. Prompt processing is slower in qwen-vx on short prompts: 8.84 against 4.46 ms a token at 60
tokens. It is about the same from 2,000 tokens (2.71 against 2.57).

Where the plain gain comes from

Change in qwen-vx ms for each token Evidence
The GPU picks each token, and the CPU reads 8 tokens at a time (vx-lang/qwen-vx#60) 0.25–0.32 measured with and without, in separate runs
2 SIMD groups of 4 rows in each Q4_0 product block, as llama.cpp does (vx-lang/qwen-vx#61) 0.3–0.6 measured with and without, in separate runs
The GGUF reads 32 MB more for each token about 0.31 estimated from the bytes

The gap in table 1 is 1.45 ms for each token.

Limits

  • One machine, one model, greedy decoding, one sequence.
  • We tuned the draft thresholds (6, 8 and 12 tokens of match) on these 12 prompts. There is no
    held-out set.
  • The two programs drafted on different text, because the weight files differ.
  • Not compared: time to the first token, and the start of a session.
  • Two rounds of each run.
  • A local debug vxc compiled qwen-vx, not the 0.0.5 release binary.
  • llama.cpp chose its flash-attention setting itself (auto), and nobody recorded the choice.

Reproduce

vxc src/main.vx --package qwen=src --host default --machine fleet/m4-uma.vx -- \
    qwen3-30b-q4-q6s.vxw --tokenizer qwen3.vxt --prompt-file PROMPT --steps 512 [--draft 7]
llama-completion -m Qwen3-Coder-30B-A3B-Instruct-Q4_0.gguf -f PROMPT -n 512 --temp 0 -ngl 99 \
    -no-cnv --no-escape -c 8192 --ignore-eos
llama-lookup -m Qwen3-Coder-30B-A3B-Instruct-Q4_0.gguf -f PROMPT -n 512 --temp 0 -ngl 99 \
    --no-escape -c 8192 -b 4096 --spec-draft-n-max 8 --ignore-eos

All runs of table 1

ms for each token. drafted and accepted are the draft counts of llama-lookup and qwen-vx.

prompt,kind,prompt_tokens_llama,round,program,ms_per_token,lookup_decoded,vx_rounds,drafted,accepted
edit1,edit,1703,1,vx_plain,23.998,,,,
edit1,edit,1703,1,vx_draft7,13.103,,64,431,425
edit1,edit,1703,1,llama_completion,25.200,,,,
edit1,edit,1703,1,llama_lookup,12.551,517,,494,435
edit2,edit,1477,1,vx_plain,23.712,,,,
edit2,edit,1477,1,vx_draft7,12.296,,64,440,440
edit2,edit,1477,1,llama_completion,24.430,,,,
edit2,edit,1477,1,llama_lookup,12.066,518,,517,448
edit3,edit,2101,1,vx_plain,24.645,,,,
edit3,edit,2101,1,vx_draft7,13.981,,66,459,436
edit3,edit,2101,1,llama_completion,24.930,,,,
edit3,edit,2101,1,llama_lookup,12.973,513,,494,433
complete1,complete,1654,1,vx_plain,23.950,,,,
complete1,complete,1654,1,vx_draft7,23.266,,12,48,30
complete1,complete,1654,1,llama_completion,26.400,,,,
complete1,complete,1654,1,llama_lookup,24.536,513,,606,235
complete2,complete,1551,1,vx_plain,23.688,,,,
complete2,complete,1551,1,vx_draft7,23.739,,8,17,8
complete2,complete,1551,1,llama_completion,24.990,,,,
complete2,complete,1551,1,llama_lookup,29.949,513,,479,106
complete3,complete,1439,1,vx_plain,23.476,,,,
complete3,complete,1439,1,vx_draft7,23.754,,17,59,30
complete3,complete,1439,1,llama_completion,25.220,,,,
complete3,complete,1439,1,llama_lookup,25.117,521,,755,263
write1,write,48,1,vx_plain,21.818,,,,
write1,write,48,1,vx_draft7,21.824,,5,11,5
write1,write,48,1,llama_completion,22.460,,,,
write1,write,48,1,llama_lookup,26.945,513,,547,106
write2,write,59,1,vx_plain,21.735,,,,
write2,write,59,1,vx_draft7,21.858,,8,32,13
write2,write,59,1,llama_completion,24.210,,,,
write2,write,59,1,llama_lookup,30.357,513,,729,124
write3,write,46,1,vx_plain,21.855,,,,
write3,write,46,1,vx_draft7,21.807,,5,11,3
write3,write,46,1,llama_completion,24.370,,,,
write3,write,46,1,llama_lookup,30.517,513,,760,103
explain1,explain,1280,1,vx_plain,23.250,,,,
explain1,explain,1280,1,vx_draft7,22.685,,10,55,37
explain1,explain,1280,1,llama_completion,24.960,,,,
explain1,explain,1280,1,llama_lookup,27.470,513,,593,150
explain2,explain,1453,1,vx_plain,23.679,,,,
explain2,explain,1453,1,vx_draft7,19.905,,29,164,150
explain2,explain,1453,1,llama_completion,25.600,,,,
explain2,explain,1453,1,llama_lookup,23.758,513,,569,223
explain3,explain,1747,1,vx_plain,24.035,,,,
explain3,explain,1747,1,vx_draft7,22.652,,15,84,70
explain3,explain,1747,1,llama_completion,26.100,,,,
explain3,explain,1747,1,llama_lookup,27.427,513,,592,165
edit1,edit,1703,2,llama_lookup,12.534,517,,494,435
edit1,edit,1703,2,llama_completion,24.700,,,,
edit1,edit,1703,2,vx_draft7,13.196,,64,431,425
edit1,edit,1703,2,vx_plain,23.969,,,,
edit2,edit,1477,2,llama_lookup,12.066,518,,517,448
edit2,edit,1477,2,llama_completion,24.480,,,,
edit2,edit,1477,2,vx_draft7,12.283,,64,440,440
edit2,edit,1477,2,vx_plain,23.488,,,,
edit3,edit,2101,2,llama_lookup,12.955,513,,494,433
edit3,edit,2101,2,llama_completion,27.080,,,,
edit3,edit,2101,2,vx_draft7,14.011,,66,459,436
edit3,edit,2101,2,vx_plain,24.575,,,,
complete1,complete,1654,2,llama_lookup,24.520,513,,606,235
complete1,complete,1654,2,llama_completion,24.660,,,,
complete1,complete,1654,2,vx_draft7,23.288,,12,48,30
complete1,complete,1654,2,vx_plain,23.662,,,,
complete2,complete,1551,2,llama_lookup,28.105,513,,479,106
complete2,complete,1551,2,llama_completion,24.610,,,,
complete2,complete,1551,2,vx_draft7,23.974,,8,17,8
complete2,complete,1551,2,vx_plain,23.609,,,,
complete3,complete,1439,2,llama_lookup,25.841,521,,755,263
complete3,complete,1439,2,llama_completion,26.170,,,,
complete3,complete,1439,2,vx_draft7,23.733,,17,59,30
complete3,complete,1439,2,vx_plain,23.451,,,,
write1,write,48,2,llama_lookup,26.934,513,,547,106
write1,write,48,2,llama_completion,22.520,,,,
write1,write,48,2,vx_draft7,21.766,,5,11,5
write1,write,48,2,vx_plain,21.716,,,,
write2,write,59,2,llama_lookup,30.813,513,,729,124
write2,write,59,2,llama_completion,24.320,,,,
write2,write,59,2,vx_draft7,21.820,,8,32,13
write2,write,59,2,vx_plain,21.752,,,,
write3,write,46,2,llama_lookup,30.552,513,,760,103
write3,write,46,2,llama_completion,22.500,,,,
write3,write,46,2,vx_draft7,21.862,,5,11,3
write3,write,46,2,vx_plain,21.784,,,,
explain1,explain,1280,2,llama_lookup,27.483,513,,593,150
explain1,explain,1280,2,llama_completion,24.260,,,,
explain1,explain,1280,2,vx_draft7,22.694,,10,55,37
explain1,explain,1280,2,vx_plain,23.378,,,,
explain2,explain,1453,2,llama_lookup,23.717,513,,569,223
explain2,explain,1453,2,llama_completion,25.040,,,,
explain2,explain,1453,2,vx_draft7,20.013,,29,164,150
explain2,explain,1453,2,vx_plain,23.705,,,,
explain3,explain,1747,2,llama_lookup,27.405,513,,592,165
explain3,explain,1747,2,llama_completion,24.650,,,,
explain3,explain,1747,2,vx_draft7,22.555,,15,84,70
explain3,explain,1747,2,vx_plain,24.088,,,,

GLM-5.3 on 8 H200 GPUs (vx-lang/glm-vx): the server decodes at 8.55 ms per output token at batch 1 with tensor parallelism. That is 1.07 times as fast as SGLang 0.5.22 (9.18 ms). It is the median over 10 requests of 1000 tokens in and 1000 out. Every kernel of its decode step is a Vx spawn kernel. Its GSM8K score matches SGLang's (0.920, SGLang 0.915-0.935).

Intel GPUs through SPIR-V

  • A spirv64 topology now runs its kernels on an Intel GPU. Vx 0.0.4 compiled each kernel to a SPIR-V image, but nothing loaded the image. A new runtime dispatch library loads it through Level Zero, and runs a matmul through oneMKL. It runs on the Intel Arc A770, whose machine model is fleet/arc-a770.vx (#2763).
  • The build makes this library only on a machine with oneAPI, so the downloaded toolchains do not include it. To use it, build Vx from source on a machine with oneAPI.
  • SPIR-V kernels can compute with bf16, i4, u4, fp8 and fp4. They call the libm functions of the device, and a failed assert in a kernel stops the program on the host (#3582, #3604, #3603, #3608, #3610).
  • More kernels get an image: an if/else whose branches read different tensors, and 1-bit multiply, divide and shift. A kernel that computes with an integer wider than 64 bits gets an error (#3581, #3580, #3602).
  • A kernel that fails to compile no longer costs the other kernels their image. The error names the kernel (#3368, #3603).

Parallel writes through a table of indices: Distinct and DisjointRanges

  • A block loop can write y[idx[b]] when idx is a Distinct tensor: a rank-1 integer tensor whose elements are all different. The race check then knows that two blocks write different rows, so the write needs no unsafe. A mixture-of-experts layer uses this write to put the result of each token in its row (#3565).
  • DisjointRanges is a table of [start, end) ranges that do not overlap. Block t can write the rows from tiles[t][0] up to tiles[t][1], as a grouped GEMM does (#3664). The race check also accepts the row and column layout of a tensor-core result, such as (l % 32) / 4. So a grouped GEMM stores its tiles with plain writes, not through raw addresses (#3670).
  • Distinct::check(t) and DisjointRanges::check(t) check the values on the CPU. Each gives Some with the new type, or None when the check fails. So a program needs no unsafe to make one. unsafe { Distinct(t) } trusts the program instead (#3567, #3669).
  • A program cannot write to either type, so the claim stays true. The book covers them, and a blog post says why they exist (#3566).

Apple GPU

  • The kernels of a batch run at the same time, unless one uses memory that another one writes (#3674). The expert gate and up products of the 30B model, split into 8 kernels a layer, take 7.09 ms in place of 8.89 ms. VX_METAL_CONCURRENT=0 gives the old order.
  • The running sums of a simdgroup matmul stay in its matrices, not in shared memory (#3477, #3523, #3529). Tiles over 48 KB stay static, and the compiler checks the 32 KB limit of threadgroup memory (#3525).
  • Short loops unroll (#3571). A vector sum across a SIMD-group is one simd_sum (#3615). A constant offset goes into the address of a shared-memory access (#3574).
  • An mlir! block can be the body of a threadgroup, with vectors and simdgroup matrices (#3558). It may write the elements that its writes_only clause names (#3556).

NVIDIA GPUs

  • A lane matmul of two fp8 tiles goes to the tensor cores. It is a native fp8 mma.sync on sm_89. On sm_90, ptxas converts both tiles to f16 for each mma.sync. A lane matmul of an f16 tile by an fp8 tile converts only the fp8 tile, which suits Hopper (#3646, #3648). Both ran on an H200.
  • lanes::matmul_scaled multiplies fp4 tiles with block scales, in NVFP4 or MXFP4 form. It runs on the CPU, in GPU loops and in block code (#3647, #3649, #3652). The new type f8e8m0 holds the scales of MXFP4 (#3649).
  • The compiler can emit lanes::matmul_scaled for the tensor cores of sm_120a, and for the tcgen05 units of the B200 with VX_TCGEN05=1. The tests check only the PTX of these two. Neither ran on such a GPU yet (#3650, #3652).
  • A tensor-core matmul keeps its sums in registers and loads its inputs with ldmatrix. Its tiles have no bank conflicts in shared memory (#3388, #3462, #3475, #3406, #3409).
  • bf16 results go out with wide stores and one cvt instruction for each conversion (#3492, #3493, #3503). Several cp.async copies end with one wait (#3445).
  • The tests of #3388, #3409, #3445, #3462 and #3503 check the PTX. These changes did not run on a GPU before they merged.
  • Short loops of constant bounds unroll (#3588). A lane loop as wide as its block has no loop in the PTX (#3354). On an A100, this made llm.c's GELU with fast_tanh 0.047 ms in place of 0.051 ms. It went from 9.5% slower than nvcc's llm.c kernel to 2% slower.
  • benchmarks/gpu_gemm has eight GEMM versions, timed against cuBLAS (#3448). At 4096 x 4096 x 4096 on an A100, the best one gives 47.3 TFLOP/s, and cuBLAS gives 260.1 TFLOP/s. #3400 tracks the work on this gap.

Parallel loops and lanes

  • lanes::copy and lanes::fill copy and set a tile in shared memory. lanes::scale_rows multiplies each row of a tile by a scale. A block can copy the rows that a list of ids chooses (#3459, #3460, #3491, #3490). lanes::copy_async copies bf16 and f16 tiles (#3383).
  • The proof that no two passes write the same element accepts more index forms (#3386, #3411, #3670):
    • blk / n and blk % n
    • an index that divides the variable of an inner loop
    • a part of a part of the lane index
  • A spawn loop with bounds known only at run time runs in parallel (#3584).
  • A pass of an across loop may borrow a tensor that it declares as &mut (#3656).

Tasks, channels and time

  • std::sync has oneshot, broadcast, watch and work-queue channels, and a semaphore (#3428, #3424). std::sync::jobs::run_jobs runs jobs with at most n tasks at a time (#3424).
  • A cancel ends a sleep. Interval runs periodic work, and sleep_until waits for a time. Other tasks run while a task sleeps (#3419, #3341).
  • On an NVIDIA GPU, a tensor in GPU memory leaves a channel with no CPU copy. This fixes the crash of 0.0.4 (#3311).
  • A closure that two tasks call must be Sync (#3439).

Transfers and memory

  • transfer_async to a GPU copies in the background. A large copy goes through pinned memory (#3483, #3500).
  • q[a..b] is a view of the rows a to b - 1 of q. For now, a and b must be number literals, and --legacy-codegen refuses a range. tensor_view_1d gives a rank-1 view of raw memory (#3473, #3501).
  • A topology can have several devices, with count:. Each device has its own memory in the capacity check (#3378, #3384).

Standard library

  • std::nn has kernels that run a decode step for n positions at once, with the same bits as for one position (#3673). It groups the experts of a mixture-of-experts layer on the GPU, and its attention splits the positions among blocks (#3569, #3575, #3216).
  • Four Q4_0 products of std::nn give a block 2 SIMD groups of 4 rows, as llama.cpp does (#3667).
  • min, max and clamp of the integers work in GPU kernels (#3524).
  • print! prints a u128 and an i128, and evaluates every argument before it prints (#3545, #3544).

Language

  • A match on a float compares each number arm with == (#3332).
  • A ? can follow an unsafe block (#3443).
  • A loop with no break can end a function (#3470).
  • A tuple pattern works in a GPU region (#3600).
  • An i128 or u128 constant, and a match arm above the largest i64, keep their real value (#3331, #2826).

Compiler and tools

  • On Linux, a crash report names the function of each frame. With --legacy-codegen it also gives the file and line. The toolchain includes llvm-symbolizer for this. The handler uses only calls that are safe in a signal handler (#3642, #3636).
  • An error points at its declaration, function or statement, and a number literal has a place in the source (#3303, #3302, #3298, #3295).
  • vx-format.toml configures vx-format. vx-format --check prints the changes as a diff, and writes no file (#2350).
  • --action emit-obj builds the object as run-jit does. So a shared library no longer exports a wrapper of a function that Vx hides (#3539).
  • vx-analyzer --version and --help print and exit (#1516).
  • The installer takes the build with no CUDA when the CUDA 12 runtime is missing. A relative VX_HOME works (#3325, #3438).
  • New book chapters: how to write parallel code, loop optimizations, and exact and fast math (#3323, #3397, #3355).

Changes that can break a program

Most of these are new errors for code that was wrong before. The error message says what to change.

  • Unary operators bind tighter than as, as in Rust. *r as T is now (*r) as T, and -m as i64 is (-m) as i64. x as f64.sqrt() and x as i64? are now parse errors. Write (x as f64).sqrt() (#3294).
  • These are new errors:
    • - on an unsigned integer or a bool, E3030 (#3404)
    • +, -, * and / on two bool values, E3030 (#3583)
    • a cast from a number to bool, E3011. Compare with zero instead (#3405)
    • a compare of a reference with a value (#3535)
    • a write through a *const pointer, or through a view of one, E4006 (#3507)
    • Memory::GPU_HBM as the memory of a topology whose arch: is not nvptx64, E6016. Declare a memory of your own (#3314)
    • a task that calls a closure parameter, E3055. A closure type such as |i32| -> i32 does not say what the closure captures, so the checker cannot tell that it is Sync (#3439)
  • sleep returns a bool, not an i32: true when the whole time passed, and false when a cancel or within ended it early (#3419).
  • Arithmetic on two rank-1 f16 or bf16 tensors gives a tensor of the same type, not an f32 tensor (#3550).
  • std::nn::nn_attention takes scratch space from nn_attention_scratch in place of scores (#3216).
  • print! evaluates every argument before it prints anything, so output from a call in an argument now comes first (#3544).
  • The dispatch id of NUMA node N is now 1100 + N, not 1000 + N, because 1000 is the SPIR-V device. Rebuild a dispatch library that you built against the 0.0.4 header (#2763).
  • == on two names for string literals compares their text, not their addresses (#3530).
  • The Qwen app moved from apps/qwen to vx-lang/qwen-vx (#3407).

Known issues

  • On macOS, a crash prints no report, only the signal that ended the program. The report there lost the function that crashed (#3663, #3678).
  • With --legacy-codegen, transfer(&view, ..) of a view of raw memory fails MLIR verification (#3660).
  • With --legacy-codegen on the Apple GPU, a tensor that gets a new tensor in a loop can be read before its kernel ends (#3661).

Install

curl -fsSL https://vxlang.org/install.sh | sh

The Linux toolchains include the LLVM they need. On macOS, Vx requires LLVM 22. For
--verify-seams, every platform needs the z3 binary. The installer checks for what
is missing and prints the command to install it. See
the install guide.

Platforms

Platform Artifact
macOS, Apple Silicon vx-<version>-aarch64-apple-darwin.tar.gz
Linux x86_64, glibc 2.35+ vx-<version>-x86_64-unknown-linux-gnu.tar.gz
Linux x86_64 with an NVIDIA GPU, glibc 2.35+ vx-<version>-x86_64-unknown-linux-gnu-cuda.tar.gz

Verify a download against its .sha256 before unpacking.

What's Changed

  • release: build macOS on macos-15, and let a manual run build a tag by @hiraditya in #3305
  • release: build with Rust 1.99.1, and package with either cargo layout by @hiraditya in #3312
  • README and homepage: the features and install steps of 0.0.4 by @hiraditya in #3310
  • diagnostics: a number literal has a place in the source by @hiraditya in #3295
  • diagnostics: an error with no place points at its statement by @hiraditya in #3298
  • diagnostics: a check of the whole program points at the function by @hiraditya in #3302
  • diagnostics: an error about a declaration points at it by @hiraditya in #3303
  • codegen: compare a match arm above the largest i64 by its value by @bajpainaman in #2826
  • format: add project configuration and non-writing CI checks by @minzyfox in #2350
  • prover: raw-pointer writes ask the solver a linear number of questions by @hiraditya in #3327
  • channels: a tensor in GPU memory leaves a channel without a CPU copy by @hiraditya in #3311
  • tests: the backend tests pass on a machine with one A100 by @hiraditya in #3313
  • topologies: a GPU that is not NVIDIA cannot use GPU_HBM as its memory by @hiraditya in #3314
  • codegen: an i128 or u128 constant keeps its value outside the i64 range by @hiraditya in #3331
  • match: a match on a float compares each number arm with == by @hiraditya in #3332
  • install: take the build without CUDA when the CUDA 12 runtime is missing by @hiraditya in #3325
  • book: a chapter on how to write parallel code, with a table from tokio to Vx by @hiraditya in #3323
  • fuzz: generate programs with tasks, channels, a Mutex and atomics by @hiraditya in #3320
  • tests: run each test of tasks again under other orders of its tasks by @hiraditya in #3300
  • formal: fix memory model README, add mapping of files where documentation is. by @hiraditya in #3245
  • tasks: report a deadlock in a function that C calls when a seed is set by @hiraditya in #3301
  • blog: what compiling against z3 taught us about trusting provers by @hiraditya in #3330
  • across: name the function and its fix when a call in a loop may write by @hiraditya in #3339
  • tasks: let other tasks run while a task sleeps by @hiraditya in #3341
  • book: across loops and lanes in the parallel-code chapter, and machine files from the install by @hiraditya in #3345
  • lower: a task can take a let mut tensor on the AST code generator by @hiraditya in #3344
  • gpu: a lane loop as wide as its block runs once, with no loop in the PTX by @hiraditya in #3354
  • blog: plain loops at the speed of CUDA by @hiraditya in #3356
  • blog: compare the GPU code of OpenMP, OpenACC and C++ parallel algorithms by @hiraditya in #3360
  • ci: update the GitHub Actions that still run on Node 20 by @hiraditya in #3358
  • book: a chapter on exact and fast math, and when to use fast_exp by @hiraditya in #3355
  • blog: correct four claims in "plain loops at the speed of CUDA" by @hiraditya in #3389
  • gpu: keep the accumulator of a tensor-core matmul in registers in a loop by @hiraditya in #3388
  • lower: print an if or a match on the AST code generator by @hiraditya in #3361
  • tests: a SPIR-V image adds the offset of a view that starts inside a buffer by @hiraditya in #3376
  • flatten: keep the half type of a + b on f16 and bf16 tensors of rank 2 and up by @hiraditya in #3362
  • lanes: copy tiles of bf16 and f16 with lanes::copy_async by @hiraditya in #3383
  • topologies: a declared topology can have more than one device, with count: by @hiraditya in #3378
  • topologies: each device has its own copy of its memory in the capacity check by @hiraditya in #3384
  • across: accept a tile picked with blk / n and blk % n by @hiraditya in #3386
  • book: explain functions that never return in the contracts chapter by @hiraditya in #3390
  • lower: pass a tensor from transfer by value, and free it once by @hiraditya in #3377
  • lower: a SPIR-V kernel that fails no longer costs the others their image by @hiraditya in #3368
  • book: a chapter on loop optimizations by @hiraditya in #3397
  • std::nn: attention splits the positions among blocks by @hiraditya in #3216
  • std::nn: a block of the Q4_0 down product of the experts is one SIMD group by @hiraditya in #3211
  • lower: let a struct with an enum field be an enum payload by @minzyfox in #3143
  • codegen: write and read a field of a struct behind an indexed pointer by @hiraditya in #3366
  • checker: refuse unary minus on an unsigned integer or a bool by @hiraditya in #3404
  • checker: refuse a cast from a number to bool by @hiraditya in #3405
  • apps: move the Qwen app to the vx-lang/qwen-vx repository by @hiraditya in #3407
  • fix(analyzer): print the version instead of starting the server by @SakarPaisen in #1516
  • across: let an index divide the variable of an inner loop by @hiraditya in #3411
  • install.sh: install in a folder given by a relative VX_HOME by @hiraditya in #3438
  • tests: check an enum inside an enum at a later payload position, and with -j 2 by @minzyfox in #3431
  • parser: unary operators bind tighter than as, as in Rust by @SycamoreQ in #3294
  • gpu: load the inputs of a tensor-core matmul with ldmatrix by @hiraditya in #3406
  • gpu: lay out matmul tiles in shared memory with no bank conflicts by @hiraditya in #3409
  • ci: run the jobs that use the LLVM cache on Ubuntu 26.04 by @hiraditya in #3453
  • docs: a design for one select over channels, sockets, timers and tokens by @hiraditya in #3420
  • flat codegen: return a struct or an enum from inside a tasks block by @hiraditya in #3446
  • benchmarks: eight GEMM versions on an NVIDIA GPU, timed against cuBLAS by @hiraditya in #3448
  • codegen: choose backend from arch and device index, not ID bands by @SycamoreQ in #3387
  • drops: a return in the last if of a task frees the task's values by @hiraditya in #3364
  • lanes: copy a tile to or from shared memory with lanes::copy by @hiraditya in #3459
  • lanes: set each element of a tile in shared memory with lanes::fill by @hiraditya in #3460
  • gpu: keep the running sums of a tensor-core matmul in registers, not in shared memory by @hiraditya in #3462
  • check: let a loop with no break end a function by @hiraditya in #3470
  • legacy codegen: give a borrowed struct parameter one address by @hiraditya in #3447
  • flat codegen: free each local once in a match whose arms all return by @hiraditya in #3440
  • Apple GPU: keep the running sums of a simdgroup matmul in its matrices, not in shared memory by @hiraditya in #3477
  • gpu: keep a lane's elements of a tile in registers through its lane loops by @hiraditya in #3475
  • Views: range slicing q[a..b] is a view of q's rows by @uakarsh in #3473
  • sync: add oneshot, broadcast, watch and work-queue channels by @hiraditya in #3428
  • models: check the work queue and the broadcast channel by @hiraditya in #3478
  • dialect: lower every lane op through one interface by @hiraditya in #3474
  • transfer: copy in the background for transfer_async to a GPU by @hiraditya in #3483
  • docs: explain with pictures how a lane keeps its elements of a tile in registers by @hiraditya in #3484
  • lanes: copy the rows that a list of ids chooses into a shared tile by @hiraditya in #3490
  • lanes: keep the keys after the count in a block sort on the GPU by @hiraditya in #3486
  • lanes: multiply each row of a tile by a scale with lanes::scale_rows by @hiraditya in #3491
  • gpu: convert f32 to bf16 with one cvt instruction on NVIDIA GPUs by @hiraditya in #3492
  • agents: give each worktree its own build directory by @hiraditya in #3495
  • gpu: store neighbouring bf16 elements of a swizzled tile with one wide store by @hiraditya in #3493
  • transfer: copy large transfer_async data through pinned memory by @hiraditya in #3500
  • tensors: add tensor_view_1d, a rank-1 view over raw memory by @hiraditya in #3501
  • tests: run the corpus tests in parallel, and link programs faster by @hiraditya in #3506
  • transfer: read a *const view in place over a shared edge, and wait for the device at return by @hiraditya in #3507
  • gpu: join the bf16 stores in LLVM alone, so the swizzle costs no registers by @hiraditya in #3503
  • fleet: the M4's CPU memory is shared with its GPU, so a borrowed tensor reaches a kernel with no copy by @hiraditya in #3502
  • transfer: read a tensor reached through a & parameter in place over a shared edge by @hiraditya in #3510
  • tests: weights in a struct reach the GPU of the M4 with no copy by @hiraditya in #3511
  • borrow: a row of a tensor holds no reference, so a copy into it is no escape by @hiraditya in #3516
  • prover: leave out a comparison of floats, which z3 rejected in an integer query by @hiraditya in #3517
  • core: min, max and clamp of the integers compare values, so a GPU kernel can call them by @hiraditya in #3524
  • metal: keep sums in simdgroup matrices when lanes::scale_rows rescales them by @hiraditya in #3523
  • metal: keep the tiles of an Apple GPU kernel static above 48 KB, and check its 32 KB limit by @hiraditya in #3525
  • check: leave sums that a GPU keeps in registers out of the shared memory size by @hiraditya in #3526
  • check: sums of an f32 tile times an f16 tile stay in simdgroup matrices too by @hiraditya in #3529
  • check: == on a name for a string literal compares the text by @hiraditya in #3530
  • gpu: check the whole tile of lanes::copy once, not each element by @hiraditya in #3531
  • match: an arm that ends in a match whose arms all return gives no value by @hiraditya in #3532
  • check: a compare of a reference with a value is a type error by @hiraditya in #3535
  • check: decide when a whole-tensor assignment writes into its destination by @hiraditya in #3536
  • vec: a reference to a tensor in a Vec compiles on both code generators by @hiraditya in #3537
  • check: a result keeps only a shared borrow of an argument behind a & by @hiraditya in #3538
  • emit-obj: build the object as run-jit does, without the execution engine by @hiraditya in #3539
  • inline: keep a callee that ends in llvm.unreachable a call by @hiraditya in #3540
  • codegen: a large tensor that a call returns goes on the heap by @hiraditya in #3542
  • parse: keep the tensor alive for a let of a row of a temporary by @hiraditya in #3534
  • mlir!: a string reaches MLIR with the escapes MLIR reads by @hiraditya in #3543
  • print: evaluate every argument before printing anything by @hiraditya in #3544
  • print: print a u128 and an i128 by @hiraditya in #3545
  • across: an mlir! block may write the elements that its writes_only clause names by @hiraditya in #3556
  • metal: an mlir! block can be the body of a threadgroup, with vectors and simdgroup matrices by @hiraditya in #3558
  • tensors: the sum of two rank-1 f16 tensors is an f16 tensor by @hiraditya in #3550
  • tests: drop a value that owns something two enums deep by @minzyfox in #3548
  • gpu: the CPU stub of GPU-only code works in a program with its own write by @hiraditya in #3563
  • codegen: rank-1 tensor arithmetic compares element types, not their MLIR spellings by @hiraditya in #3564
  • check: a block may write at a row that it reads from a Distinct tensor by @hiraditya in #3565
  • core: Distinct::check(t) makes a Distinct tensor with no unsafe by @hiraditya in #3567
  • benchmarks: llama.cpp's expert product and flash attention as mlir! blocks on the Apple GPU by @hiraditya in #3568
  • std::nn: group the pairs of a mixture-of-experts layer by expert on the GPU by @hiraditya in #3569
  • metal: unroll short loops, and a scheduling hint for simdgroup matrices by @hiraditya in #3571
  • docs: Distinct tensors in the book, and a blog post on why they exist by @hiraditya in #3566
  • parse: let a ? follow an unsafe block by @hiraditya in #3443
  • time: let a cancel end sleep, and add Interval for periodic work by @hiraditya in #3419
  • nvidia: unroll short loops of constant bounds by @hiraditya in #3588
  • across: every lane computes the lets before a shared-memory fill by @hiraditya in #3586
  • blog/vx meets postgres by @minzyfox in #3494
  • docs: what we learned from unrolling short loops on NVIDIA by @hiraditya in #3591
  • metal: put a constant offset into shared memory in the address of the access by @hiraditya in #3574
  • std::nn: overlap the loads and spread the tile writes in the offsets kernel of the expert grouping by @hiraditya in #3575
  • spirv: write 1-bit multiply, divide and shift as logic before the backend by @hiraditya in #3580
  • spirv: an if/else whose branches read different tensors gets an image by @hiraditya in #3581
  • check: refuse +, -, * and / on two bools by @hiraditya in #3583
  • spawn: a loop with run-time bounds runs in parallel by @hiraditya in #3584
  • core: let a tuple pattern work in a GPU region by @hiraditya in #3600
  • check: give the bool hint only when the operands are single values by @hiraditya in #3601
  • spirv: refuse a kernel that computes with an integer wider than 64 bits by @hiraditya in #3602
  • spirv: give fp8 and fp4 kernels an image, and blame only the kernel a stage fails on by @hiraditya in #3603
  • metal: keep GPU memory resident, and read a mapped weight file in place by @hiraditya in #3606
  • tests: use all cores of a CI runner for the corpus tests by @hiraditya in #3509
  • lanes: make the op of each lane routine from one table by @hiraditya in #3472
  • tasks: check that a closure two tasks call is Sync by @hiraditya in #3439
  • gpu: combine back-to-back waits for cp.async copies into one wait by @hiraditya in #3445
  • across: order the count of passes for the device, not the system by @hiraditya in #3609
  • check: read the op of a lane routine from its table entry by @hiraditya in #3611
  • spirv: compute libm methods with the device's own functions by @hiraditya in #3608
  • spirv: report a failed assert in a kernel through a flag by @hiraditya in #3610
  • metal: reduce a vector across a SIMD-group with one simd_sum by @hiraditya in #3615
  • spirv: compute bf16 in f32 and hold it as i16, so a bf16 kernel compiles by @hiraditya in #3582
  • spirv: compute 4-bit integers in 8 bits, so i4 and u4 kernels are right by @hiraditya in #3604
  • legacy codegen: a method call on a field behind a reference field by @hiraditya in #3627
  • runtime: the SIGSEGV handler uses only calls that are safe in a handler by @hiraditya in #3636
  • flat codegen: a GPU kernel reads a parameter, not its slot of the host by @hiraditya in #3637
  • runtime: name the functions and lines of a crash backtrace by @hiraditya in #3642
  • gpu: multiply fp8 tiles on the tensor cores of sm_89 and later by @hiraditya in #3646
  • lanes: a matmul of fp4 tiles with block scales, on the CPU and in GPU loops by @hiraditya in #3647
  • sync: add a semaphore by @hiraditya in #3424
  • gpu: multiply an f16 tile by an fp8 tile on the tensor cores by @hiraditya in #3648
  • types: f8e8m0 for MXFP4 scales, and MXFP4 tiles in lanes::matmul_scaled by @hiraditya in #3649
  • gpu: run lanes::matmul_scaled on the tensor cores of sm_120a by @hiraditya in #3650
  • gpu: lanes::matmul_scaled in block code, and on tcgen05 for the B200 by @hiraditya in #3652
  • ci: do not check open-std.org links in the docs link check by @hiraditya in #3654
  • across: a pass may borrow a tensor that it declares as &mut by @hiraditya in #3656
  • docs: design notes for Distinct tensors and for ranges that do not overlap by @hiraditya in #3658
  • flat codegen: &mut y passes the tensor when y gets a new tensor in a loop by @hiraditya in #3662
  • check: a block may write the rows of its range from a DisjointRanges table by @hiraditya in #3664
  • nn: run four Q4_0 products with 2 SIMD groups of 4 rows in a block by @hiraditya in #3667
  • core: DisjointRanges::check(t) makes a table of ranges with no unsafe by @hiraditya in #3669
  • xpu: a runtime dispatch library for the Intel Arc by @epsilon-phase in #2763
  • check: E6026 asks the device a value is on, not the owner of its memory by @hiraditya in #3444
  • lower: preserve byte types when indexing string literals by @minzyfox in #3655
  • codegen: a match takes its value type from its own arms, not from old locals by @minzyfox in #3665
  • check: lanes may write rows that use a part of a part of the lane index by @hiraditya in #3670
  • nn: kernels that run a decode step for n positions at once by @hiraditya in #3673
  • metal: let the kernels of a batch run at the same time by @hiraditya in #3674
  • docs: a design for several generated tokens for each read of the weights by @hiraditya in #3676
  • runtime: no crash handler on macOS for now by @hiraditya in #3678
  • release: version 0.0.5 by @hiraditya in #3679

New Contributors

Full Changelog: v0.0.4...v0.0.5