v2.3-kv260: cross-operator fusion, prefill up to 1.9x, decode 1.05-1.14x

The four HuggingFace LLMs of v2.2.1-kv260 with cross-operator fusion: decode's SwiGLU, decode's attention and a prefill chunk's attention each in one dispatch per layer, all on by default in compile_sa.sh and bit-exact on the board. The overlay, runtime and firmware are those of v2.2.1 (byte-identical); the arithmetic is unchanged, so v2.2's quality figures still hold. Tag v2.3-kv260, branch kv260-v2.3; every number below comes from a board log in kv260_v2.3_board_results.tar.gz, the full tables with provenance in docs/results_v2.3.md, the designs in docs/fusion_design.md.

On the board (overlay d16_100mhz_w128_p6_gemv_np4_pcpi, 100 MHz); v2.2 in parentheses

	SmolLM2-135M	Gemma-3-270M	Qwen3-0.6B	Llama-3.2-1B
Decode, tok/s (after a 32-token prompt)	27.0 (23.6)	17.6 (16.5)	7.94 (7.21)	4.64 (4.42)
Prefill, tok/s (chunks of 16, T = 64)	73.6 (57.8)	78.9 (68.5)	25.9 (19.0)	16.8 (15.4)
A prompt filling the context, prefill tok/s	32.3 (2032 tokens)	69.0 (496)	13.4 (7.0)	11.1 (8.9)
Decode ms per step at T = 2048	91.2 (99.3)	63.1 at 512 (67.2)	293.1 (317.8)	303.7 (321.5)
Executables (all on the accelerator)	1070 (1127)	699 (739)	1007 (1051)	608 (665)

Decode is 1.05–1.14× faster, prefill 1.09–1.37× at T = 64 and up to 1.9× per chunk at T = 2048 (Qwen3: each head's cache is now read and transposed once a chunk instead of once a row). In all four: 0 dispatches on the host, logits bit-exact with the simulator in 17/17 steps, prefill vs decode-only bit-exact over a prompt that fills the context, plan A / plan B on the RISC-V core bit-exact.

What changed since v2.2.1

• SwiGLU in decode (--iree-sa-fuse-ffn): the two FFN projections, the gating, the amax and the int8 output in one dispatch, each chunk's gating overlapping the next chunk's weight streaming.
• Attention in decode (--iree-sa-fuse-attn): the q path, scores, probabilities, p · V and the int8 output in one dispatch per layer: a region pass after IREE's region formation (IREE patch 0004, a plugin hook) and a kernel attn_layer.
• Attention in prefill (--iree-sa-fuse-attn-chunk): a chunk's 16 rows in one kernel attn_chunk per layer: two passes over the cache in blocks of 64 keys, every vector operation over the 16 rows, an arithmetic causal mask per row; the frontend writes the rows' K / V first. IREE patch 0005 works around an insert_slice fold that dropped all rows but the last (fixed upstream after the pinned revision by iree-org/iree#24266 / #24276).
• SA_FUSE_FFN=0, SA_FUSE_ATTN=0, SA_FUSE_ATTN_CHUNK=0 turn them off; an export from before the frontend change keeps its prefill rows unfused.

Assets

File	Contents
picorv32_kv260_d16_100mhz_w128_p6_gemv_np4_pcpi.bit / .hwh	the overlay (as v2.2.1)
rt_fw.bin, sa-llm-run-aarch64	the PicoRV32 firmware v7 and the aarch64 runtime (as v2.2.1)
kv260_v2.2_build_reports.tar.gz	the overlay's Vivado reports (unchanged overlay)
kv260_v2.3_board_results.tar.gz	the four bundles' board results and test logs
results_v2.3.md	the results document
SHA256SUMS	checksums

No model weights are included (export them with compiler/frontend/hf_generic.py from the public checkpoints, compiler/scripts/deploy_longctx.sh).

v2.2.1-kv260: four HuggingFace LLMs whole on the KV260, RISC-V backend

Four unmodified HuggingFace LLMs compiled whole by IREE for the KV260 accelerator, every dispatch on the accelerator, board logits bit-exact with the functional simulator. This release freezes tag v2.2-kv260 plus documentation (the LLM and compiler learning paths); branch kv260-v2.2. Every number below comes from a board log in kv260_v2.2_board_results.tar.gz; the full table with provenance is docs/results_v2.2.md.

Since v1.1-kv260-k1: D = 16, 128-bit DMA, a batched SFU, a streaming GEMV unit fed by four DMA ports (also driven over PCPI), the generic HuggingFace frontend with public W8A8 checkpoints (generality plan WP0–WP9, tag v2.0-kv260), long context (tag v2.1-kv260-longctx), the RISC-V backend (C7) and one executable per kernel instead of one per layer (IREE patch 0003, upstream as iree-org/iree#25059).

On the board (overlay d16_100mhz_w128_p6_gemv_np4_pcpi, 100 MHz)

	SmolLM2-135M	Gemma-3-270M	Qwen3-0.6B	Llama-3.2-1B
Executables (all on the accelerator)	1127	739	1051	665
Decode, tok/s (after a 32-token prompt)	23.6	16.5	7.21	4.42
Prefill, tok/s (chunks of 16, T = 64)	57.8	68.5	19.0	15.4
Decode ms per step at T = 2048	99.3	67.2 (T = 512)	317.8	321.5
Teacher forcing PPL, board (fp32), 2 windows of the context	13.79 (13.48)	43.59 (43.98)	18.58 (18.77)	13.56 (13.21)
top-1 vs fp32 / KL	90.7% / 0.025	76.9% / 0.213	88.4% / 0.055	91.9% / 0.037
Decode read rate, share of the four-port peak	66%	84%	80%	96%

Compiled for 2048 positions (Gemma: its 512 window); int8 weights and activations, int8 KV cache (K per channel, KV scales calibrated on 2048 WikiText-2 train tokens). In all four: 0 dispatches on the host, logits bit-exact with the simulator in 17/17 steps, prefill vs decode-only bit-exact over a prompt that fills the context, no per-step host copies.

RISC-V backend (decode ms per step, 40-token prompt; tokens identical, last logits bit-exact in every case):

	SmolLM2	Gemma	Qwen3	Llama
ARM loop, descriptor lists	42.6	60.7	138.8	226.6
Plan B: decode loop on the PicoRV32	37.8	61.2	136.7	225.7
Plan A inside plan B: every dispatch as rv32 code generated by iree-compile	67.0	76.4	166.7	253.5
Decode rv32 working set	24 KB	20 KB	34 KB	41 KB

Plan A's gap is instruction fetch from DDR (about 27 cycles per instruction); the decode step's code now fits a 32–64 KB instruction memory.

Overlay

Config	LUT	BRAM36	DSP	Setup WNS / hold WHS
d16_100mhz_w128_p6_gemv_np4_pcpi	101,995 (87.1%)	138 / 144	349	+0.286 / +0.010 ns

Built with Vivado 2024.1 at commit a6974fc (boards/kv260/scripts/build_bitstream.sh; the configuration is in build_info.txt of the build reports).

Assets

File	Contents
picorv32_kv260_d16_100mhz_w128_p6_gemv_np4_pcpi.bit / .hwh	the overlay; PYNQ needs the .bit and .hwh to share a basename: rename both to picorv32.* for a bundle
rt_fw.bin	the resident RISC-V command processor (firmware/rt, v7: plan B's loop, RT_CALL, R5, performance counters)
sa-llm-run-aarch64	the IREE runtime with the sa HAL driver (static aarch64 binary)
results_v2.2.md	the consolidated results with the bundle, commit and log behind each number
kv260_v2.2_board_results.tar.gz	the board logs of all tests and each bundle's results/ tree (run logs, event-counter profiles, board snapshot), VERSION, build_info.txt, teacher-forcing scores
kv260_v2.2_build_reports.tar.gz	the overlay's build info, address map, timing, utilization, clocks, check_timing
SHA256SUMS	checksums

The compiled models (sa.vmfb, sa_packed.irpa) are not attached: they contain the models' weights, which are under their own licenses. Rebuild them with compiler/scripts/build_board_bundles.sh (see compiler/README.md; IREE e4a3b0405d plus compiler/patches/iree).

Board setup

Ubuntu 22.04 for Kria with Kria-PYNQ. The accelerator's memory is a 1280 MiB shared-dma-pool at 0x1000_0000 with a u-dma-buf node on it, reserved at boot (boards/kv260/scripts/accel_mem_install.sh install); see boards/kv260/README.md and docs/memory_model.md. Board tests: compiler/scripts/run_board_tests.sh <bundle>… (one entry point, compiler/tests/board_run.py).

Learning material

• LLM learning path: models, numerics, quantization, the inference system, long context, measuring speed and quality.
• Compiler learning path: MLIR and IREE, the sa backend pass by pass, the HAL driver, verification, the RISC-V backend, upstreaming.
• Hardware learning path.

Next

Lifetime-based local-memory planning (3a / 3b in docs/kv260_upgrade_plan.md), first bounded with the performance model; W4A8 decode; an instruction memory for the control core.

v1.1-kv260-k1: Kria KV260, stage K1 (50 / 100 MHz)

First KV260 release: stage K1 of the upgrade plan. It runs the PYNQ-Z1's L2 accelerator unchanged (D = 8, one 64-bit DMA port) on the Kria KV260: K1a at 50 MHz, K1b at 100 MHz. Both pass the board regression (REGRESSION PASS). On every test, the generated tokens are identical to the PYNQ-Z1 freeze baselines (tests/baselines/pynq-z1/).

From this release on, main is the KV260 version. The PYNQ-Z1 version is frozen at v1.0-pynq-z1.

On the board (Ubuntu 22.04 for Kria + Kria-PYNQ)

Test	PYNQ-Z1 (50 MHz)	K1a (50 MHz)	K1b (100 MHz)
stories15M decode (C3)	17.62 tok/s	18.27	35.44
stories15M prefill + decode	17.41	18.06	34.84
SmolLM2-135M decode, hand-written qhf path	2.22	2.27	4.43
SmolLM2-135M prefill + decode, qhf	2.17	2.22	4.30
SmolLM2-135M, unmodified HuggingFace model (generic path)	1.87	1.88	3.73
DMA, contiguous LD / ST	397 MB/s	397	794 (99.2% of 8 B/cycle)

• Speed: at 100 MHz the wall time per step halves. The A53 host adds very little.
• Device cycles: within 0.03% of the Z1 at 50 MHz, and 0.2–0.4% more at 100 MHz. DDR latency is fixed in nanoseconds, so it costs twice the cycles at 100 MHz.

Overlays

Config	LUT	BRAM36	Setup WNS / hold WHS
d8_50mhz (K1a)	40,257 (34.4%)	130 / 144	+7.800 / +0.010 ns
d8_100mhz (K1b)	40,275 (34.4%)	130 / 144	+1.171 / +0.010 ns

The overlays were built with Vivado 2024.1 (boards/kv260/scripts/build_bitstream.sh [-sa_mhz 100]). Both contain the LDPARAM register stage in sa_cmdfetch.v, which K1b needed to close timing at 100 MHz.

Assets

File	Contents
picorv32_kv260_d8_100mhz.bit / .hwh	the K1b overlay. PYNQ needs the .bit and .hwh to share a basename; rename both to picorv32.* for the deploy bundles
picorv32_kv260_d8_50mhz.bit / .hwh	the K1a overlay
rt_fw.bin	the resident RISC-V command processor (firmware/rt)
bwtest_fw.bin	the DMA bandwidth self-test firmware
sa-llm-run-aarch64	the IREE runtime with the sa HAL driver (static aarch64 binary)
board_regress.py, run_board.sh	the board regression runner
kv260_k1_board_results.tar.gz	the full board results of K1a and K1b: logs, tokens, event-counter profiles, VERSION
kv260_k1_build_reports.tar.gz	per config: build info, address map, timing, utilization, clocks, check_timing
SHA256SUMS	checksums

Board setup

The kernel has CONFIG_STRICT_DEVMEM=y, so the runtime's DDR window comes from the u-dma-buf module. Load it after every boot:

sudo insmod u-dma-buf.ko udmabuf0=536870912

The driver runs xmutil unloadapp before it loads the overlay. See boards/kv260/README.md and the memory model in docs/memory_model.md.

Next

K1c: D = 16. KC: the VE sequencer and the out-of-order scheduler. K2: DMA and on-chip memory bandwidth.

v1.0-pynq-z1: PYNQ-Z1 final release

The final PYNQ-Z1 version. Development moves to the AMD Kria KV260
(plan);
branch pynq-z1 only receives critical fixes.

On the board (L2 bitstream, D = 8, 50 MHz): all bit-exact

Test	Result
DMA bandwidth (bwtest)	397 MB/s per direction (99% of 64 bit × 50 MHz), 788 MB/s LD + ST
stories15M decode (C3)	76/76 steps bit-exact with the hand-written L5 path, 17.6 tok/s
stories15M prefill + decode	prefill 75.6 tok/s, decode 17.4 tok/s
SmolLM2-135M, hand-written qhf path	decode 2.22 tok/s; prefill 7.13 tok/s
SmolLM2-135M, unmodified HuggingFace model (generic path)	decode 1.87 tok/s, entirely on the accelerator; prefill 3.18 tok/s

Decode time breakdown from the accelerator's event counters
(tests/baselines/pynq-z1/z1_profile.md): the linear layers run at the
weight-load floor; fixed overhead is 1–2%; the VE / SFU work not hidden behind
the loads is 22% (hand-written) / 33% (generic) of a SmolLM2 decode step.

Compiler: IREE plugin with the sahl / sahw dialects; the micro-kernels are
written out as sahl operations (docs/iree_compiler_plan.md §8.20); a golden
corpus of 4002 dispatches guards the descriptors.

Assets

File
picorv32_l2.bit, picorv32_l2.hwh	the L2 overlay (PicoRV32 + systolic-array accelerator); same as RISCV-on-PYNQ-Z1/bitstreams/l2
rt_fw.bin	the resident RISC-V command processor (firmware/rt)
bwtest_fw.bin	the DMA bandwidth self-test firmware
sa-llm-run-armv7	the IREE runtime with the sa HAL driver (static armv7 binary)
board_regress.py	the board regression runner
z1_board_results.tar.gz	the full logs, token sequences, profiles and event-counter CSVs of the freeze regression
SHA256SUMS

To reproduce: compiler/scripts/deploy_z1_freeze.sh stages every board test
(it builds the models and modules on the host), then board_regress.py on the
board; compiler/tests/make_z1_baselines.py compares against
tests/baselines/pynq-z1/.