2026.09.02

Release notes — 2026-09-02

A 24-hour measurement pass across both rigs this project runs day to day: the 4× Tesla P100 rig
(llama.cpp-based engine, hybrid MoE, PXQ_UNIVERSAL 4-bit) and the 2× Tesla V100 rig (vLLM-based
sm_70 serving line, 27B dense-hybrid, PXQ4). Everything below traces to bench/fair-battle.md,
docs/PXA-SM70-SERVING.md, or the raw arms this document itself describes, protocol per
bench/fair/protocol.md unless stated otherwise (llama-server/vLLM /completion, temp 0, n=7
median, 1 warmup discarded, unique prompt per repeat).

Two sections: what's already live in the launcher, and what's queued in the next engine build
(not yet in a tagged release — everything under "Next engine build" needs the candidate
binary described there, not today's release).

Launcher-level wins already live (2026-09-01)

• 4× P100 rig: -ub 2048 -wgt 8 (was -ub 1024). -wgt 8 zeroes a 248k-vocab logits
  reservation, which is what lets the larger micro-batch fit at all — -ub 2048 alone does not
  boot. Measured deep-fill (86,401-token prompt, 3 reps): prefill 223.4 → 229.6 t/s (+2.8%),
  decode flat. Boots clean at -c 150016.
• 2× V100 rig: --max-num-batched-tokens 4096 (was 2048). Six interleaved boots: prefill
  918–920 → 940–945 t/s @3k (+2.5%), 878–880 → 908–912 t/s @20k (+3.3%), decode and aggregate
  flat, KV pool unchanged. 8192 was tried too — no further prefill gain and a 15% smaller KV pool
  from the larger reserved scratch — so 4096 is the ceiling here, not a stop along the way.
• 2× V100 rig: --gpu-memory-utilization 0.92 (was 0.85). A capacity lever, not a speed one —
  roughly 44% more KV pool at the same accuracy, verified against the linear pool-vs-GMU
  relationship measured earlier. (The next-engine-build config below runs at 0.88, one notch back
  down — see the NCCL section.)

Next engine build (not yet in a tagged release)

Pipelined prefill

Prefill on the P100 rig runs mostly single-stream even though the scheduler supports a second
CUDA copy stream. Root cause was two bugs, not a missing feature: the graph allocator re-planned
its buffers on every prompt chunk instead of once per request (a stale reserve/real graph size
mismatch caused by state-reset nodes only emitted at position 0), and the batched MoE
row-mapping step forced a host sync inside the per-layer loop on every batched call. Fixed:
re-plans per 20k-token prefill went from 12 to 2, and the MoE row map moved fully on-device (next
item). With the fix, on top of the full ship set, prefill at -c 98304 (PXA_PIPELINE_PP=1 GGML_SCHED_MAX_COPIES=2):

arm	prefill @3,121	prefill @20,801
single-stream (ship set, PP off)	487.06	410.61
pipelined, two streams	515.36	501.79 (+22.2% vs single-stream)

Byte-identical across 15 comparisons including sequence switches. Known limit: it does not fit
at the P100 rig's production -c 150016 — compute buffer allocation fails on card 1 there, and
the no-PP fallback does not recover either. It ships as opt-in at reduced context (-c 98304)
today, not the default. Also fixed on this branch, unconditionally, regardless of
whether pipelining ships: a per-slot attention state window on the hybrid architecture that was
never reset between requests, so a second request could inherit the first request's window.

GQA-packed attention

PXA_FA_GQA_PACK=4 reads each attention key/value once per query group instead of once per
head. An earlier version of this same lever was measured as noise at low context fill and left
off. Re-measured at 86,401 tokens, where the re-read volume the lever removes is actually large:

arm	decode @8	decode @86,401
control	26.08	12.52
PXA_FA_GQA_PACK=4	26.22	17.54 (+40%)

Output-identical. Stacked with the host-overhead cuts below, the full shipped set measures
+49% decode at 86,401 tokens, +2.3% at low fill, over a bracketing control pair, output
identical in every arm.

Host-overhead cuts

Four more bit-identical fixes, instrumented with a per-token host/GPU timing split
(PXA_HOST_TIMING): a struct-of-arrays KV-sequence mask, bounded top-k sampling off raw logits
instead of a full sort, a trimmed KQ-mask host-to-device upload, and a router aliasing barrier
for multi-row MoE dispatch (a correctness guard, zero measured cost, kept on). Together with
GQA-packed attention: measured host time per token at deep fill, 6.0 ms → 1.6 ms; GPU submit
time per token, 73.6 ms → 48.7 ms. The full stack (these four plus GQA-pack plus five smaller
bit-identical prefill micro-fixes — narrowed get_rows, a fast-div copy kernel, a flattened concat
path, a register-cached norm, and lazy scheduler resets) measured +2.6–4.6% prefill, +2.3%
low-fill decode, +49% deep-fill decode, output-identical at temp 0 in every arm.

Device-side MoE row map

The table mapping tokens to experts used to get built on the host and copied down once per
batched MoE layer, forcing a device-to-host sync (prepare_row_mappigs) on the critical path.
It now builds entirely on-device (hist/scan/fill in one pass); self-check mode compares the two
paths and found them bit-identical across 64 cases. This is the second half of the pipelined
prefill fix above — without it, the sync alone would have capped the pipelining win regardless of
the scheduler fix.

PXQ on CPU

An AVX2 int8 dot product (16-entry pshufb lookup, maddubs/madd, per-32-block fp32
anchor/scale) makes CPU-only and partial-offload PXQ inference a real option instead of a
technically-correct fallback. It's panel-tiled to match PXQ's native 64-row panel layout rather
than going through a generic per-row vector-dot trait, which the panel layout can't address.
Measured on a 12-thread Xeon E5-2699 v3, PXQ4, 128-token prompt: 14.3 → 110.6 t/s prefill
(7.7×). Cross-checked against the CUDA decode path: 0 ULP difference across 196 tensors.
Default on (PXA_PXQ_CPU_DOT).

Export and requantize

llama-pxq-export in.gguf out.gguf [--type f16|f32] [--cpu] streams a PXQ GGUF tensor-by-tensor,
decoding PXQ tensors in whole-panel chunks (GPU or CPU) and copying everything else byte for
byte. llama-quantize --allow-requantize now accepts a PXQ source and routes it through the same
decode path before re-quantizing — one step from PXQ straight to, say, Q4_K_M (an
--i-know-this-is-double-lossy flag is required for a PXQ or Q8_0 source, since that's quantizing
twice). Verified: 196 tensors CPU-vs-CUDA decode at 0 ULP, 114/114 non-PXQ tensors byte-identical
through the export, chunked output identical to whole-tensor output, and greedy generation
identical after a round trip.

One quantizer footgun found and not yet fixed: --token-embedding-type pxq* produces a model
that generates nothing, because the token-embedding lookup is a plain row-gather that never goes
through a PXQ decode path. The row-gather guard that should block this keys on tensor name and
size, not on quant type, so it doesn't catch it. Consumer-based guard queued; until then, don't
quantize the token embedding tensor to a PXQ type.

Correctness fixes

• get_rows grid overflow — ported from ik_llama.cpp upstream (78ce50c1): a grid-size
  overflow in the get_rows kernel at large expert counts (relevant at 512-expert scale).
• MMQ fusion-chain guard — ported from upstream (c49f7db3): guards the MMQ fusion chain
  for non-MMQ quant types that shouldn't enter it.
• Quantized cpy launch fix — ported from upstream (7642ac3e): quantized cpy kernels were
  launched with 1-thread blocks; fixed.
• PLE window reset — found here, not upstream: the per-slot convolution window on the hybrid
  architecture's local-embedding path was never reset between requests, so a second request on a
  reused slot could inherit the first request's window. Fixed regardless of whether pipelined
  prefill ships, since it's a correctness bug on its own.
• Token-embedding PXQ guard — found, not yet fixed; see "Export and requantize" above.

Rejected levers (so nobody re-tries them)

• PXA_FA_KEYS_PER_SPLIT ladder (1024/512/256) — negative on the GQA-packed D=256 decode
  kernel at 86k-token fill: more splits means more combine work once K/V reads are already shared
  across four heads. Stays off.
• PXA_GEMV_RPB=-1 — −8% decode at low fill on this kernel family; the GPU-submit time got
  worse, not better. Off.
• CUDA graph capture on P100 — hard-gated below Ampere in the backend; forcing it measured
  −2.8%. Dead, not revisited.
• Fused TP2 all-reduce + RMSNorm fusion (V100) — engagement confirmed in the boot log, but
  measured prefill −1.5…−1.9%, under the noise floor at best. Not shipped.
• QPN v13 kernel — a from-scratch PXQ4 GEMV rewrite, measured slower than the v12b tensor-core
  path at batch 16 on this rig's actual tensor shapes, and found on inspection to inherit the
  same split-K partials-arena capture-order hazard v12 had before the v12b fix (see
  docs/PXA-SM70-SERVING.md). Not shipped.
• PXQ4 output head (4-bit lm_head) — fails the 70-prompt same-top-token gate at 97.1%
  (bar is ≥98%); see docs/PXA-SM70-SERVING.md. Kept the fp16 head.
• PXA_X_CACHE (reused f32→f16 activation conversion across QKV matmuls) — bit-identical but
  measured null on this model's tensor shapes. Left in the tree, default off, not chased further.
• CPU governor, ondemand → performance — kept (it's harmless and already persisted), but
  the deep-fill A/B measured +0.9%, under this rig's ~3% noise floor. Not claimed as a win.

Known limits

• Pipelined prefill does not fit at the P100 rig's production context — compute buffer
  allocation fails on card 1 at -c 150016, and the no-PP fallback doesn't recover either. It's
  real and byte-identical at reduced context (-c 98304); it isn't shipped at full context yet.
• Volta dense decode still loses to MXFP4 — a structural DP4A-scale-fix...

pxq_llama v2026.08.31

pxq_llama — v2026.08.31

Release from PXA Network. Supersedes v2026.08.28-rc3.

Low-bit MoE inference for Pascal and Volta — the cards current releases no longer
build kernels for.

What is new

Unified KV ring (--kv-unified, -kvu). By default a server with -np N
divides its context into N fixed slices, so a long request fails even when the
cache is nearly empty. With this lever every slot addresses the whole attention-KV
ring instead, backed by admission control that defers a request when the ring is
genuinely full rather than evicting a live sequence. Measured on a 150k-context
seat: a single slot processed 86,401 tokens where the static split would have
capped it at 75,008.

Elementwise chain fusion (PXA_EW_FUSE). Collapses straight-line elementwise
runs into one interpreter kernel, replaying the standalone kernels' formulas
verbatim. Bit-identical by construction. Worth about 1.6% on a five-card seat;
measured neutral-to-negative on a four-card shared-GPU seat, so it ships
default-off and wants an A/B on your own hardware.

Flash-next MTP work and the granite tier recipe, plus a spec tracer, an
async-commit path, and FA tile f32 accumulation.

Honest notes

Speculative decoding remains a net loss on P100 in our measurements — verify
decomposes to a fixed cost plus a per-row cost that the acceptance rate does not
repay. The lever exists; the numbers did not.

Quantizing the KV cache is a memory lever, not a speed one. q8_0 halves the
cache and q4_0 quarters it, but neither is faster than f16 at any prompt
length we measured; the cost is 2-16%, shrinking as context grows.

The pxa-vllm serving stack is distributed separately as prebuilt images — see
docs/PXA-SM70-SERVING.md and docker/vllm-pxq4/.

pxq_llama v2026.08.28-rc3

pxq_llama — v2026.08.28-rc3

Release candidate from PXA Network. Supersedes
v2026.08.25-rc2; 47 commits.

This release adds support for a hybrid-attention MoE architecture that upstream
llama.cpp cannot load at all, a mixed per-tensor quantization tier
(PXQ_UNIVERSAL), two correctness fixes — one of which affected every
architecture — and a launcher that now configures a heterogeneous card pool on its
own. The llama.cpp engine and the PXQ4 vLLM backend ship from one tree.

Nothing in this release changes numerics on an existing PXQ4 deployment. Every
new code path is either a new architecture, a new quantization tier, or a guard that
declines where it previously faulted.

---

How to read the numbers in this document

Every performance figure below names its instrument and its hardware class.
None of them are estimates, and where nothing was measured the text says so instead
of guessing.

The reference hardware is a single workstation of used datacentre cards — the same
one described in bench/README.md, which is where the
reproduction scripts live:

• Tesla P100-PCIE-16GB (sm_60), Tesla V100-PCIE-16GB (sm_70),
  GeForce GTX 1080 Ti 11GB (sm_61)
• NVIDIA driver 580.142, CUDA 12.8.1, built and run inside
  nvidia/cuda:12.8.1-devel-ubuntu24.04
• PCIe x4 per card (bifurcation risers). Decode is single-card-resident and
  unaffected; only multi-card -sm layer hand-off touches the bus.

Match the class, not the machine. If your numbers differ on comparable silicon,
that is worth an issue.

Build gate for this tag. The tree was booted on the new architecture before the
tag was cut: 49/49 layers offloaded, KV allocated at exactly 3840.00 MiB, the
PXQ2/PXQ3/PXQ6 fused kernels engaged, a nine-item factual probe answered correctly
with a clean EOS, and decode at 26.86 tok/s. Instrument: llama-server +
llama-cli, PXQU artifact, five cards (4× Tesla P100 + 1× GTX 1080 Ti), 160k
context.

---

Read this first

This release adds a model architecture that upstream cannot load.
Qwen3.8-Flash-Next declares general.architecture = qwen4exp. Current mainline
llama.cpp (ghcr.io/ggml-org/llama.cpp:server-cuda, ggml 0.19.0) answers:

llama_model_load: error loading model: unknown model architecture: 'qwen4exp'

Mainline registers qwen3next and has none of the hc_* hyper-connection or
ple_* tensors this architecture is built from. That is a statement about
coverage, not a benchmark: there is no engine-vs-engine number for this model,
because there is no other engine to compare against.

Two latent correctness bugs are fixed, and one of them affected every model.
inp_ple_rows / inp_ple_conv_hist were read on every architecture but assigned
only by the qwen4exp graph builder, with no initialiser — so on any other model
the guard dereferenced a wild pointer. Measured 1 abort in 3 runs before, 0 in 15
after. The older binary was not correct, only lucky.

The launcher will now pick a working config for a heterogeneous card pool.
--model plus --cards is enough: it derives the tensor split, the micro-batch and
the CPU offload itself, prints the evidence behind each choice, and refuses rather
than guessing.

---

Both engines, one tree

piece	what it is
tools/vllm-pxq4/	the PXQ4 quantization backend / vLLM plugin, with its own build and conversion tooling
tools/pxa-launch.py	the unified launcher over both engines (docs/LAUNCHER.md)
scripts/pxa-serve-sm70.sh	PXQ4 on vLLM, Tesla V100 class (docs/PXA-SM70-SERVING.md)
scripts/pxa-serve-sm60.sh	PXQ4 on vLLM, Tesla P100 class (docs/PXA-SM60-SERVING.md)
scripts/pxa-serve-flashnext.sh	Flash-Next PXQU on llama.cpp, multi-card, 160k context
pxa/pxq4/	six prebuilt kernel libraries (four sm_60, two sm_70), a README describing what each one is, and a MANIFEST carrying sizes and md5s so a deployment can be verified against the artifacts that produced the recorded numbers

Every serving script takes MODEL= and --help and runs the shipped binary
directly; a container is opt-in via IMAGE=. Both vLLM paths were booted from this
tree and verified end to end — including tool calling, which needs
--enable-auto-tool-choice and --tool-call-parser qwen3_coder. Without those two
flags every tools= request returns 400. See Operating notes.

---

New architecture: qwen4exp (Qwen3.8-Flash-Next)

48 layers, 512 experts with 10 active, hybrid attention, and two structures that do
not appear in any other architecture this engine supports.

piece	what it is	why it needed new code
Per-layer embeddings (PLE)	per_layer_token_embd, 51.2e9 elements, ~51 GiB	a pure GET_ROWS gather — one lookup per token per head, no GEMM. It belongs in host RAM; the n-gram hash that indexes it needs int64 multiply and xor, which ggml does not have, so it is computed host-side
Hyper-connections	hc_*, a wide 10240 residual with 4 streams	the mixer replaces all norms; the head mixer is the output norm
GDN linear attention	36 of 48 layers	only 12 layers keep a KV cache (full_attention_interval = 4)

Converter

convert_qwen4exp.py takes HF safetensors to GGUF, streaming, and carries a
213-case self-test (--self-test <reference.gguf>) so a conversion can be gated
before it is quantized.

Four layout rules were each established by byte-comparison against a reference
GGUF. Each one silently corrupts the model if missed — the file loads and generates
plausible-looking garbage:

• PERM48 = arange(48).reshape(16,3).T.ravel() applies to ssm_a, ssm_dt.bias,
  ssm_alpha, ssm_beta, every attn_gate, the v-portion of attn_qkv and of
  ssm_conv1d, and the input dim of ssm_out
• ssm_a = -exp(A_log), under that permutation
• every RMSNorm weight is stored as (w − 1) — linear_attn.norm is the sole
  exception
• ssm_conv1d permutes only its v-portion

Scope, stated plainly: the vision tower (333 tensors) and the MTP head (31
tensors) are not converted. Text generation only.

---

PXQ_UNIVERSAL: a mixed per-tensor tier map

A PXQU artifact carries different PXQ tiers on different tensors, driven by a
.tiers map (docs/PXQU-CONVERT.md). The Flash-Next build is
96.77 GiB across 1224 tensors:

type	tensors	bytes	share
q8_0	123	51974 MiB	52.5%
pxq3	106	34606 MiB	34.9%
pxq2	38	8574 MiB	8.7%
pxq4	276	1460 MiB	1.5%
f16	290	1220 MiB	1.2%
q6_K	2	995 MiB	1.0%
f32	389	250 MiB	0.3%

The q8_0 share is almost entirely the CPU-resident PLE table. Excluding it, the
GPU-resident bytes are 94.7% PXQ-family — which is why this build needs
--pxq-composition-override: the composition floor measures whole-file share, and
that is the wrong measure for an architecture whose largest tensor never reaches a
GPU.

Two guards were added to the quantizer, both because the failure was hit for real:

• panel codecs are refused on row-gather tensors. A panel-packed codec on a
  GET_ROWS target gathers nonsense.
• any tensor with ne0 % 32 != 0 is ruled to f32, with a preflight that
  refuses to start rather than dying 59 tensors into a multi-hour run.

Quality caveat, unresolved. The build emits 154 pxq3 reconstruction ceiling 0.8957 × row absmax warnings: row-peak weights are mildly clipped, so expect
elevated top-1% error. This artifact was quantized without an imatrix. Both
correctness batteries passed, but if output quality is ever questioned on a PXQU
build, this is the first thing to suspect.

---

Correctness fixes

fix	symptom	evidence
inp_ple_rows / inp_ple_conv_hist initialised	wild-pointer dereference on any non-qwen4exp model; a 0.6B dense model aborted on a PLE assert after an unrelated relink	1 abort / 3 runs before, 0 / 15 after, correct output in all 15
PXA_ROUTER_FUSE mode-3 base-pointer guard	the guard checked ne00 % 4 and row stride nb[1] % 16, but never that src0->data / src1->data were themselves 16-byte aligned, while the kernel casts both to float4* — a misaligned-address abort, not a wrong number	default-OFF path; it now declines the fused path instead of faulting

The first one carries the more useful lesson: the failure moved with an unrelated
relink, because the value tracked heap layout. When a failure moves with a
relink, suspect uninitialised memory before you suspect the change.

---

Performance

llama.cpp engine, qwen4exp

Instrument: llama-cli, --temp 0 --ignore-eos, 2 repetitions, 2931-token
prompt for the prefill figure. (Earlier prefill figures on this architecture came
off a 15-token prompt and measured fixed overhead, not throughput. They are
withdrawn.)

PXQ4, six cards — 4× Tesla P100-PCIE-16GB + 2× Tesla V100-PCIE-16GB, PLE
resident in host RAM:

-ub	prefill	decode
512 (adaptive default)	337–366 tok/s	30.6–30.9 tok/s
1024	430–439 tok/s	30.6–31.1 tok/s
2048	OOM — needs its own -ts	—

-ub 1024 is +20% prefill for nothing: the decode ranges overlap, so it is not
a trade. The launcher now emits it for this architecture only; it is not
generalised to other architectures, which have different activation shapes.

Decode reached 32.4 tok/s on a short prompt on the same six cards, via two
levers committed in this release:

• a byte-proportional six-card -ts rebalance that moves 4 layers off the Pascal
  cards onto the Volta cards;
• a dedicated small-K / large-R f16 GEMV for the hyper-connection up-projection.

PXQU, four Tesla P100-PCIE-16GB: 27.79 tok/s decode; prefill 411–414 tok/s at
-ub 1024, 424–430 at -ub 2048.

**PXQU, five cards...