Skip to content

0.0.0-alpha.12.2

Pre-release
Pre-release

Choose a tag to compare

View all tags
@giovannifil-64 giovannifil-64 released this 12 Jun 21:15
0.0.0-alpha.12.2
4eeb46f
  • Qwen3.5 hybrid linear-attention support (Qwen3_5ForConditionalGeneration, text-only) across three checkpoint formats: BF16 safetensors, compressed-tensors pack-quantized INT4 (full and mixed BF16/INT4), and GGUF (llama.cpp qwen35 arch). All four tested 4B variants pass a 13-test adversarial battery covering long-context recall, identical-token adversarial prompts, byte-identical batched-vs-single decode, mixed prefill/decode concurrency, structured output, and streaming parity.
  • Prefill now runs on Metal 4 TensorOps (mpp::tensor_ops), which engages the M5 neural accelerator: dense GEMMs, packed-quant GEMMs, and FlashAttention. On macOS releases or hardware without Metal 4 the runtime compile fails cleanly and everything stays on the existing kernels, so the minimum OS requirement is unchanged.
  • Concurrent decode on packed-quant checkpoints (AWQ, GPTQ, compressed-tensors) no longer degrades below single-stream throughput: batched GEMV kernels share one weight read across the batch rows, closing the same pathology fixed for GGUF in alpha.12.1.

New Features

  • Gated DeltaNet runtime (src/common/gdn.rs, src/common/block.rs, src/common/paged.rs): linear-attention layers run a chunked parallel scan for prefill and an O(1) recurrent step for decode. Per-sequence recurrent state (causal-conv window plus a 32x128x128 F32 memory) lives in the same per-(sequence, layer) slot as paged KV, inheriting the retire/preempt/abort lifecycle. The per-chunk unit-lower-triangular system is inverted blockwise (doubling product on 16x16 diagonal blocks, then pairwise block combination): the reference implementation's sequential row loop needs one kernel launch per row and is unusable on Metal, while plain doubling at chunk 64 overflowed F32 on real prompts with repeated tokens because the explicit matrix powers grow even when the true inverse is small. The math is fixture-tested against the verbatim transformers reference, plus an adversarial fully-correlated-keys regression test.
  • Gated attention and partial RoPE (src/common/attention.rs): q_proj emits per-head query and gate halves, with the sigmoid gate applied before o_proj; only the first rotary_dim head dimensions rotate (64 of 256 on Qwen3.5). Available on dense, packed-quant, and GGUF attention paths. All Qwen3.5 RMSNorms are Gemma-style zero-centered.
  • Hybrid-aware engine (src/engine.rs, src/models/traits.rs): models with recurrent state automatically disable prefix caching (a recurrent state cannot skip tokens) and refuse speculative decoding (rollback cannot rewind the state). KV budget and block allocators cover only the full-attention layers; linear layers alias the first full layer's allocator so scheduler accounting stays correct.
  • compressed-tensors INT4 loader (src/common/awq.rs::compressed_to_awq, src/common/weights.rs): detects quant_method = "compressed-tensors" with format = "pack-quantized" (int4 symmetric, group strategy) and converts weight_packed/weight_scale into AWQ qweight/qzeros/scales at load; the stored nibbles are already offset-binary and transfer verbatim, and the symmetric zero-points become constant words. Mixed-precision checkpoints work through the existing dense fallback for ignore-listed modules. Packing verified against compressed-tensors 0.17.0. The tied-embedding lm_head RTN quantization (previously AWQ-4bit only) now also covers this scheme.
  • GGUF qwen35 architecture (src/models/gguf_model.rs, src/common/gdn.rs::load_gguf): hybrid layout from full_attention_interval and the ssm.* metadata keys. Accounts for the three transforms llama.cpp's converter bakes into the file: ssm_a already stores the negated exponential of A_log, every norm except the DeltaNet gated norm has the plus-one pre-added (so the GGUF arch loads Standard norms while the HF arch loads Gemma norms), and V heads are reordered to tiled order (q/k expand by whole-block tiling instead of repeat-interleave).
  • Qwen3.5 function calling (src/server/routes/chat.rs, src/chat_template.rs): the XML-style tool-call format these templates instruct (<tool_call><function=NAME><parameter=K>V</parameter>) is parsed into proper tool_calls, with multiline values preserved, JSON-typed scalars coerced, reasoning text before the block tolerated, and parallel calls supported. Tool-call arguments are parsed from the OpenAI wire string into a mapping for the Jinja context, matching the transformers convention: Qwen3.5's template iterates them with items (previously a 500 on tool-result round trips) and Qwen3-style tojson no longer double-encodes.
  • Vision tower (model.visual.*) and MTP (mtp.*) tensors are skipped at safetensors load (about 2 GB saved on Qwen3.5-9B); DeltaNet scalar parameters keep their checkpoint F32 dtype end to end.
  • GDN debug aids: OXYDLLM_GDN_DEBUG=1 (per-stage NaN/Inf probe), OXYDLLM_GDN_CHUNK=N (chunk-size override).

Performance

  • TensorOps prefill GEMM (src/common/mpp_gemm.metal, metal_ops.rs::maybe_mpp_matmul, linear.rs): BF16 matmuls with 64 or more rows route onto Metal 4 matmul2d, which uses the M5 neural accelerator. Wired into the dense Linear path (transposed-weight kernel) and the packed-quant prefill path; decode and small batches are untouched. The library compiles at runtime once per device; OXYDLLM_DISABLE_MPP=1 force-disables the path.
  • Staged packed-quant prefill GEMM (mpp_gemm_{w4,w8,gptq4,gptq8}_staged, metal_ops.rs::maybe_mpp_quant_matmul): packed-quant prefill no longer materializes the dense weight. Each K-iteration dequantizes a 64x64 tile of B into threadgroup memory and feeds matmul2d in multiply-accumulate mode with a float cooperative-tensor accumulator. Covers the AWQ layout (arbitrary zero-points, so AWQ, compressed-tensors INT4, and W8A16) and the GPTQ layout, with dequantize-plus-GEMM as fallback. Besides the kernel time, this removes a transient dense-weight allocation per quantized linear per forward (up to about 100 MB on fused gate-up layers).
  • FlashAttention prefill on TensorOps (mpp_fa_bf16_d{64,128,256}, routed inside flash_attention_metal_prefill): the BF16 no-softcap prefill attention path runs on matmul2d with cooperative-tensor destinations. The score matrix accumulates in registers, the online softmax keeps the running row max and denominator in threadgroup scratch indexed by cooperative-tensor element coordinates, and the P-times-V product accumulates into the output cooperative tensor. Causal mask with prefix offset, GQA-native, one simdgroup per 32-row Q block. F32, F16, and softcap inputs keep the existing kernel.
  • Batched W4A16/W8A16 GEMV for concurrent decode (src/common/quant_kernels.metal, src/common/metal_ops.rs, src/common/linear.rs): packed-quant decode at 2 to 8 concurrent sequences previously fell onto the dequantize-whole-weight-plus-GEMM prefill path, making concurrency slower than serial execution. The new batch kernels unpack each weight word once and reuse it across the batch rows, the same design as the GGUF batch kernels.
  • Batched GPTQ GEMV for concurrent decode (gptq{4,8}_gemv_batch_{f16,bf16}): GPTQ checkpoints had the same concurrency pathology. The batch kernel keeps the GPTQ geometry (one thread per output column, weights packed along the input dimension) and holds the dequantized word in registers with the row loop outermost. Concurrency coherence verified: twelve distinct concurrent prompts each answer correctly and byte-identical to their single-stream runs.
  • Chunked dispatch for decode batches above 8 sequences (PackedQuantLinear::forward): instead of dequantizing the whole weight per layer per step, batches up to a measured per-layout threshold (AWQ 16, GPTQ 32) run as chunks of at most 8 rows on the batch GEMV kernels; larger batches and prefill keep the GEMM path.
  • Gated DeltaNet fast paths: input projections, gated norm, and out_proj run once over the whole token batch; the gated RMSNorm uses the fused Metal kernels; the small beta/decay projections fuse into one matmul when dense; the cross-chunk prefill loop precomputes everything state-independent in batch.
  • MoE prefill was measured and intentionally not moved to TensorOps: OLMoE expert FFNs already route through the dense Linear path and measure neutral, and gpt-oss is dominated by per-expert dispatch overhead, so faster expert GEMMs cannot change end-to-end numbers until the fused-MoE-dispatch work lands.

Tests

  • 294 unit tests green. New coverage: GDN fixtures against the transformers reference, prefill-to-decode handoff, adversarial chunked-scan stability, compressed-tensors converter ground truth, Qwen3.5 XML tool-call parsing, template-context arguments mapping, batched GEMV parity (AWQ, W8A16, GPTQ at multiple batch sizes), chunked-dispatch parity across tile boundaries, TensorOps GEMM parity (aligned, unaligned, transposed layouts), staged packed-quant parity for all four kernels on partial tiles and group sizes 32 and 64, and a FlashAttention d256 GQA-plus-prefix case matching the Qwen3.5 geometry.
  • Full scripts/stress_baseline.py regression: 25 of 25 core models load and pass coherence with throughput at or above the documented baseline.
  • New scripts/e2e_torture.py (local tooling): about 25 adversarial black-box tests per model covering long-context multi-needle recall, identical-token prompts, batched-vs-single byte determinism, mixed concurrency, stop/max_tokens/n/logprobs/penalties, json_object and strict json_schema with defs and anyOf, function calling, thinking on/off/streaming, and error paths. Qwen3-4B-Q4_K_M and Qwen3.5-4B pass everything except one documented expected failure (token-level stop-sequence matching).

Benchmarks

All numbers measured on the Apple Silicon reference machine (M5, 24 GB unified memory), median of three runs.

Time to first token, prompt of about 1300 tokens, before and after the TensorOps prefill work (GEMM, staged packed-quant, FlashAttention combined):

  • Qwen3-0.6B (BF16): 1.27 s to 0.63 s
  • Qwen3-4B-AWQ: 6.98 s to 2.72 s
  • Qwen3-1.7B-GPTQ-Int8: 2.18 s to 1.37 s
  • Qwen3.5-4B (BF16): 6.79 s to 5.06 s (24 of its 32 layers are linear attention, which the FlashAttention path does not touch)
    Decode throughput is unchanged by the prefill work.

Concurrent decode, aggregate tok/s, before and after the batched GEMV kernels:

  • Qwen3-4B-AWQ, concurrency 4: 17.6 to 47.7
  • Qwen3.5-4B-INT4, concurrency 4: 12.3 to 36.1
  • Qwen3-1.7B-GPTQ-Int8: concurrency 2 from 17.8 to 58.4, concurrency 4 from 30.3 to 72.9, concurrency 8 from 52.7 to 92.0, concurrency 16 from 55.9 to 92.9; single-stream unchanged
    Before the fix, concurrency 2 to 4 on these checkpoints ran below single-stream throughput.

Qwen3.5-4B decode, single stream: BF16 9.2 tok/s, compressed-tensors INT4 27.1 tok/s (20.4 before the tied lm_head quantization), mixed BF16/INT4 14.7 tok/s, GGUF Q4_K_M 24.1 tok/s. BF16 concurrency 4 aggregate scales 2.7x.

Installation

macOS (Apple Silicon)

curl -fsSL https://raw.githubusercontent.com/giovannifil-64/oxydllm/main/install.sh | sh

Linux x86_64 (CUDA — Ada / Hopper / Blackwell / Blackwell Ultra / Blackwell Desktop)

curl -fsSL https://raw.githubusercontent.com/giovannifil-64/oxydllm/main/install.sh | sh

Linux arm64 (CUDA — GH200 / GB300 / Jetson Thor / DGX Spark)

curl -fsSL https://raw.githubusercontent.com/giovannifil-64/oxydllm/main/install.sh | sh

Installer auto-selects the right binary from detected GPU compute capability.
Override with OXYDLLM_CUDA_TARGET=<ada|hopper|blackwell|blackwell-ultra|blackwell-desktop> (x86_64)
or OXYDLLM_CUDA_TARGET=<hopper|blackwell|blackwell-ultra|thor|blackwell-desktop> (arm64).

Linux (CPU only)

curl -fsSL https://raw.githubusercontent.com/giovannifil-64/oxydllm/main/install.sh | OXYDLLM_NO_GPU=1 sh

Docker (CUDA)

docker run --gpus all -p 11313:11313 \
  -v ~/.oxydllm/models:/root/.oxydllm/models \
  ghcr.io/giovannifil-64/oxydllm:latest start

x86_64 explicit CUDA tags: :cuda-ada, :cuda-hopper, :cuda-blackwell, :cuda-blackwell-ultra, :cuda-blackwell-desktop.
ARM64 explicit CUDA tags: :cuda-hopper-arm64, :cuda-blackwell-arm64, :cuda-blackwell-ultra-arm64, :cuda-thor-arm64, :cuda-blackwell-desktop-arm64.

Docker (CPU)

docker run -p 11313:11313 \
  -v ~/.oxydllm/models:/root/.oxydllm/models \
  ghcr.io/giovannifil-64/oxydllm:cpu start

SHA-256 checksums

See the .sha256 files attached to this release.

Full Changelog: nightly...0.0.0-alpha.12.2

Assets 30
Loading