Nightly (2026-07-11 · 337d0ab)

Nightly build

Built from main at commit 337d0ab on 2026-07-11.
Build workflow result: success.

Some targets may be missing when one or more build jobs fail.

Warning: Nightly builds are untested and may contain breaking changes.
Use a tagged release for production.

Install

macOS (Apple Silicon)

curl -fsSL https://raw.githubusercontent.com/giovannifil-64/oxydllm/main/install.sh | OXYDLLM_CHANNEL=nightly sh

Linux x86_64 (CUDA)

curl -fsSL https://raw.githubusercontent.com/giovannifil-64/oxydllm/main/install.sh | OXYDLLM_CHANNEL=nightly sh

Linux arm64 (CUDA — GH200 / GB300 / Jetson Thor / DGX Spark)

curl -fsSL https://raw.githubusercontent.com/giovannifil-64/oxydllm/main/install.sh | OXYDLLM_CHANNEL=nightly sh

Installer auto-selects the right CUDA target from detected GPU compute capability.
Override with OXYDLLM_CUDA_TARGET: x86_64 values: ada, hopper, blackwell, blackwell-ultra, blackwell-desktop; arm64 values: hopper, blackwell, blackwell-ultra, thor, blackwell-desktop.

Linux (CPU)

curl -fsSL https://raw.githubusercontent.com/giovannifil-64/oxydllm/main/install.sh | OXYDLLM_CHANNEL=nightly OXYDLLM_NO_GPU=1 sh

Docker (CUDA)

docker run --gpus all -p 11313:11313 \
  -v ~/.oxydllm/models:/root/.oxydllm/models \
  ghcr.io/giovannifil-64/oxydllm:nightly start

x86_64 per-arch nightlies: :nightly-cuda-ada, :nightly-cuda-hopper, :nightly-cuda-blackwell, :nightly-cuda-blackwell-ultra, :nightly-cuda-blackwell-desktop.
ARM64 per-arch nightlies: :nightly-cuda-hopper-arm64, :nightly-cuda-blackwell-arm64, :nightly-cuda-blackwell-ultra-arm64, :nightly-cuda-thor-arm64, :nightly-cuda-blackwell-desktop-arm64.

Docker (CPU)

docker run -p 11313:11313 \
  -v ~/.oxydllm/models:/root/.oxydllm/models \
  ghcr.io/giovannifil-64/oxydllm:nightly-cpu start

0.0.0-alpha.12.2

• Qwen3.5 hybrid linear-attention support (Qwen3_5ForConditionalGeneration, text-only) across three checkpoint formats: BF16 safetensors, compressed-tensors pack-quantized INT4 (full and mixed BF16/INT4), and GGUF (llama.cpp qwen35 arch). All four tested 4B variants pass a 13-test adversarial battery covering long-context recall, identical-token adversarial prompts, byte-identical batched-vs-single decode, mixed prefill/decode concurrency, structured output, and streaming parity.
• Prefill now runs on Metal 4 TensorOps (mpp::tensor_ops), which engages the M5 neural accelerator: dense GEMMs, packed-quant GEMMs, and FlashAttention. On macOS releases or hardware without Metal 4 the runtime compile fails cleanly and everything stays on the existing kernels, so the minimum OS requirement is unchanged.
• Concurrent decode on packed-quant checkpoints (AWQ, GPTQ, compressed-tensors) no longer degrades below single-stream throughput: batched GEMV kernels share one weight read across the batch rows, closing the same pathology fixed for GGUF in alpha.12.1.

New Features

• Gated DeltaNet runtime (src/common/gdn.rs, src/common/block.rs, src/common/paged.rs): linear-attention layers run a chunked parallel scan for prefill and an O(1) recurrent step for decode. Per-sequence recurrent state (causal-conv window plus a 32x128x128 F32 memory) lives in the same per-(sequence, layer) slot as paged KV, inheriting the retire/preempt/abort lifecycle. The per-chunk unit-lower-triangular system is inverted blockwise (doubling product on 16x16 diagonal blocks, then pairwise block combination): the reference implementation's sequential row loop needs one kernel launch per row and is unusable on Metal, while plain doubling at chunk 64 overflowed F32 on real prompts with repeated tokens because the explicit matrix powers grow even when the true inverse is small. The math is fixture-tested against the verbatim transformers reference, plus an adversarial fully-correlated-keys regression test.
• Gated attention and partial RoPE (src/common/attention.rs): q_proj emits per-head query and gate halves, with the sigmoid gate applied before o_proj; only the first rotary_dim head dimensions rotate (64 of 256 on Qwen3.5). Available on dense, packed-quant, and GGUF attention paths. All Qwen3.5 RMSNorms are Gemma-style zero-centered.
• Hybrid-aware engine (src/engine.rs, src/models/traits.rs): models with recurrent state automatically disable prefix caching (a recurrent state cannot skip tokens) and refuse speculative decoding (rollback cannot rewind the state). KV budget and block allocators cover only the full-attention layers; linear layers alias the first full layer's allocator so scheduler accounting stays correct.
• compressed-tensors INT4 loader (src/common/awq.rs::compressed_to_awq, src/common/weights.rs): detects quant_method = "compressed-tensors" with format = "pack-quantized" (int4 symmetric, group strategy) and converts weight_packed/weight_scale into AWQ qweight/qzeros/scales at load; the stored nibbles are already offset-binary and transfer verbatim, and the symmetric zero-points become constant words. Mixed-precision checkpoints work through the existing dense fallback for ignore-listed modules. Packing verified against compressed-tensors 0.17.0. The tied-embedding lm_head RTN quantization (previously AWQ-4bit only) now also covers this scheme.
• GGUF qwen35 architecture (src/models/gguf_model.rs, src/common/gdn.rs::load_gguf): hybrid layout from full_attention_interval and the ssm.* metadata keys. Accounts for the three transforms llama.cpp's converter bakes into the file: ssm_a already stores the negated exponential of A_log, every norm except the DeltaNet gated norm has the plus-one pre-added (so the GGUF arch loads Standard norms while the HF arch loads Gemma norms), and V heads are reordered to tiled order (q/k expand by whole-block tiling instead of repeat-interleave).
• Qwen3.5 function calling (src/server/routes/chat.rs, src/chat_template.rs): the XML-style tool-call format these templates instruct (<tool_call><function=NAME><parameter=K>V</parameter>) is parsed into proper tool_calls, with multiline values preserved, JSON-typed scalars coerced, reasoning text before the block tolerated, and parallel calls supported. Tool-call arguments are parsed from the OpenAI wire string into a mapping for the Jinja context, matching the transformers convention: Qwen3.5's template iterates them with items (previously a 500 on tool-result round trips) and Qwen3-style tojson no longer double-encodes.
• Vision tower (model.visual.*) and MTP (mtp.*) tensors are skipped at safetensors load (about 2 GB saved on Qwen3.5-9B); DeltaNet scalar parameters keep their checkpoint F32 dtype end to end.
• GDN debug aids: OXYDLLM_GDN_DEBUG=1 (per-stage NaN/Inf probe), OXYDLLM_GDN_CHUNK=N (chunk-size override).

Performance

• TensorOps prefill GEMM (src/common/mpp_gemm.metal, metal_ops.rs::maybe_mpp_matmul, linear.rs): BF16 matmuls with 64 or more rows route onto Metal 4 matmul2d, which uses the M5 neural accelerator. Wired into the dense Linear path (transposed-weight kernel) and the packed-quant prefill path; decode and small batches are untouched. The library compiles at runtime once per device; OXYDLLM_DISABLE_MPP=1 force-disables the path.
• Staged packed-quant prefill GEMM (mpp_gemm_{w4,w8,gptq4,gptq8}_staged, metal_ops.rs::maybe_mpp_quant_matmul): packed-quant prefill no longer materializes the dense weight. Each K-iteration dequantizes a 64x64 tile of B into threadgroup memory and feeds matmul2d in multiply-accumulate mode with a float cooperative-tensor accumulator. Covers the AWQ layout (arbitrary zero-points, so AWQ, compressed-tensors INT4, and W8A16) and the GPTQ layout, with dequantize-plus-GEMM as fallback. Besides the kernel time, this removes a transient dense-weight allocation per quantized linear per forward (up to about 100 MB on fused gate-up layers).
• FlashAttention prefill on TensorOps (mpp_fa_bf16_d{64,128,256}, routed inside flash_attention_metal_prefill): the BF16 no-softcap prefill attention path runs on matmul2d with cooperative-tensor destinations. The score matrix accumulates in registers, the online softmax keeps the running row max and denominator in threadgroup scratch indexed by cooperative-tensor element coordinates, and the P-times-V product accumulates into the output cooperative tensor. Causal mask with prefix offset, GQA-native, one simdgroup per 32-row Q block. F32, F16, and softcap inputs keep the existing kernel.
• Batched W4A16/W8A16 GEMV for concurrent decode (src/common/quant_kernels.metal, src/common/metal_ops.rs, src/common/linear.rs): packed-quant decode at 2 to 8 concurrent sequences previously fell onto the dequantize-whole-weight-plus-GEMM prefill path, making concurrency slower than serial execution. The new batch kernels unpack each weight word once and reuse it across the batch rows, the same design as the GGUF batch kernels.
• Batched GPTQ GEMV for concurrent decode (gptq{4,8}_gemv_batch_{f16,bf16}): GPTQ checkpoints had the same concurrency pathology. The batch kernel keeps the GPTQ geometry (one thread per output column, weights packed along the input dimension) and holds the dequantized word in registers with the row loop outermost. Concurrency coherence verified: twelve distinct concurrent prompts each answer correctly and byte-identical to their single-stream runs.
• Chunked dispatch for decode batches above 8 sequences (PackedQuantLinear::forward): instead of dequantizing the whole weight per layer per step, batches up to a measured per-layout threshold (AWQ 16, GPTQ 32) run as chunks of at most 8 rows on the batch GEMV kernels; larger batches and prefill keep the GEMM path.
• Gated DeltaNet fast paths: input projections, gated norm, and out_proj run once over the whole token batch; the gated RMSNorm uses the fused Metal kernels; the small beta/decay projections fuse into one matmul when dense; the cross-chunk prefill loop precomputes everything state-independent in batch.
• MoE prefill was measured and intentionally not moved to TensorOps: OLMoE expert FFNs already route through the dense Linear path and measure neutral, and gpt-oss is dominated by per-expert dispatch overhead, so faster expert GEMMs cannot change end-to-end numbers until the fused-MoE-dispatch work lands.

Tests

• 294 unit tests green. New coverage: GDN fixtures against the transformers reference, prefill-to-decode handoff, adversarial chunked-scan stability, compressed-tensors converter ground truth, Qwen3.5 XML tool-call parsing, template-context arguments mapping, batched GEMV parity (AWQ, W8A16, GPTQ at multiple batch sizes), chunked-dispatch parity across tile boundaries, TensorOps GEMM parity (aligned, unaligned, transposed layouts), staged packed-quant parity for all four kernels on partial tiles and group sizes 32 and 64, and a FlashAttention d256 GQA-plus-prefix case matching the Qwen3.5 geometry.
• Full scripts/stress_baseline.py regression: 25 of 25 core models load and pass coherence with throughput at or above the documented baseline.
• New scripts/e2e_torture.py (local tooling): about 25 adversarial black-box tests per model covering long-context multi-needle recall, identical-token prompts, batched-vs-single byte determinism, mixed concurrency, stop/max_tokens/n/logprobs/penalties, json_object and strict json_schema with defs and anyOf, function calling, thinking on/off/streaming, and error paths. Qwen3-4B-Q4_K_M and Qwen3.5-4B pass everything except one documented expected failure (token-level stop-sequence matching).

Benchmarks

All numbers measured on the Apple Silicon reference machine (M5, 24 GB unified memory), median of three runs.

Time to first token, prompt of about 1300 tokens, before and after the TensorOps prefill work (GEMM, staged packed-quant, FlashAttention comb...

0.0.0-alpha.12.1

• GPT-OSS support (GptOssForCausalLM) shipped end-to-end: MXFP4 expert weights stay packed on Metal with fused dequant kernels, attention sinks run through a dedicated decode SDPA kernel, harmony channels are parsed server-side into reasoning_content / content, and reasoning depth is controlled with the new reasoning_effort request field. Verified on gpt-oss-20b at 14.3 tok/s decode on the Apple Silicon reference machine (M5, 24 GB); gpt-oss-120b uses the same architecture but exceeds the reference machine's memory and is untested.
• Concurrent serving throughput fixed: batched decode (2 to 8 sequences per forward) previously fell onto the prefill GEMM kernel and was slower than serial execution (0.53x at concurrency 2). All ten GGUF quant types now have a batched GEMV where the M activation vectors share one weight read; aggregate throughput on Qwen3-4B-Q4_K_M goes from 14.9 to 52.1 tok/s at concurrency 2 and reaches 60.7 tok/s at concurrency 8.
• GGUF models with tied embeddings (no separate output.weight) no longer dequantize the embedding table to a dense BF16 LM head: the quantized weights are reused directly through the fast GEMV. Qwen3-4B-Q4_K_M decode improves from 29 to 36.5 tok/s (+25%) with about 0.5 GB less resident memory.
• Non-greedy sampling overhead cut from ~2.2 ms to ~0.4 ms per token: top_p no longer full-sorts the vocabulary and the log-probability pass is computed only when logprobs are requested.
• Greedy speculative decoding (phase 1) behind --draft-model on run and start: a draft model proposes 4 tokens per step, the target verifies them in one forward, and the KV cache rolls back to the accepted prefix. Output is exact with respect to the target's greedy decode; non-greedy requests are routed to the normal sampler automatically.
• oxydllm pull now survives interrupted downloads: truncated shards are detected against the safetensors index, failed downloads keep completed files and drop only the partial one, and retries resume at file granularity instead of restarting from zero.
• An environment-gated decode profiler (OXYDLLM_PROFILE_DECODE=1) breaks per-token decode time into embed / attention / FFN / LM-head phases with Metal synchronization, at zero cost when disabled.

New Features

• GPT-OSS / GptOssForCausalLM (src/common/mxfp4.rs, src/common/moe.rs, src/common/attention.rs, src/models/parsers/hf_parser.rs): architecture-level support covering both gpt-oss-20b (verified) and gpt-oss-120b (untested, does not fit the reference machine). MXFP4 (OCP microscaling FP4) expert weights are kept packed (blocks of 32 E2M1 values with one E8M0 exponent byte) and run through fused Metal kernels: a batched GEMV for decode (M = 1..8) and a tiled GEMM for prefill. Dequantizing gpt-oss-20b's ~19 B expert parameters to BF16 would need ~38 GB; packed residency is ~13 GB on a 24 GB machine. Experts use the gpt-oss convention: interleaved gate/up projections, clamped swiglu activation (limit = 7.0, alpha = 1.702, (up + 1) * glu), per-expert biases, and a router with bias. Routing reuses the existing top-k path (softmax over the selected experts is mathematically identical to norm_topk_prob = true). Alternating sliding/full attention layers and YaRN RoPE are parsed from config.json through the existing per-layer infrastructure.
• Attention sinks with a dedicated decode kernel (src/common/quant_kernels.metal::sdpa_vector_sink_bf16, src/common/attention.rs): gpt-oss attention adds a learned per-head sink logit to the softmax denominator, which the stock SDPA/FA kernels cannot express. Decode (q_len = 1) now runs a purpose-built kernel: one simdgroup per head, streaming online softmax over the KV positions, the sink folded into the denominator after the reduction, and native GQA indexing that reads the unrepeated K/V heads directly (no 8x repeat_kv materialization). Prefill falls back to the standard attention path. The kernel is parity-tested against a scalar reference and lifted gpt-oss decode from 12.3 to 14.3 tok/s (+17%). o_proj.bias is now loaded when present (gpt-oss is the first supported checkpoint that has one).
• Harmony channel parsing (src/server/routes/engine_loop.rs): a token-level state machine (markers, role, header, body) activates when the tokenizer carries the harmony marker tokens. Analysis and commentary channels stream as reasoning_content, the final channel as content; role names and protocol framing tokens are stripped. Unknown structure fails open to content. Both streaming and non-streaming responses are covered by the existing reasoning plumbing.
• reasoning_effort request field (src/server/routes/types.rs, src/chat_template.rs): OpenAI-style low / medium / high for harmony models, which cannot disable reasoning. Validated before model resolution (invalid values return 400 invalid_request_error without loading anything); omitted from the Jinja context when unset so the template's own default (medium) applies. The long apply_chat_template signature was folded into a TemplateOptions struct. pull now also downloads .jinja chat template files.
• Greedy speculative decoding, phase 1 (src/engine.rs, src/common/paged.rs, src/scheduler/): --draft-model <id> on run and start (env OXYDLLM_DRAFT_MODEL). Per step the draft generates 4 tokens autoregressively, the target verifies all of them in a single forward, the longest greedy-matching prefix is accepted plus a correction or bonus token, and both KV caches roll back via the new PagedKvCache::truncate_to (frees orphaned blocks, respects prefix-cache refcounts). Only plain-greedy sequences enter the speculative cycle (SamplingParams::is_plain_greedy); anything with temperature, penalties, logprobs, or logit bias decodes through the normal sampler. On the server the draft reserves half the KV budget up front, otherwise the target's allocation would starve the draft load and silently disable speculation. With the small drafts available locally there is no measured speedup yet; the machinery is correctness-verified and waits for a distilled draft.
• Decode profiler (src/common/decode_profile.rs): OXYDLLM_PROFILE_DECODE=1 times decode phases (embed, per-layer attention and FFN, final norm, LM head) with a Metal sync per phase and reports cumulative ms/token every 64 forwards. Zero overhead when the variable is unset. This is the tool that located the tied-LM-head regression and the MoE dispatch costs.

Performance and Efficiency

• Batched decode GEMV for all ten GGUF quant types (src/common/quant_kernels.metal, src/common/metal_ops.rs, src/common/linear.rs): the scheduler already batches concurrent decodes into one forward, but the M >= 2 path used the prefill GEMM kernel, whose fixed tile overhead made two batched tokens 3.7x more expensive than two serial ones. Each quant type now has a gguf_*_gemv_batch_bf16 kernel where the inner loop runs the M activation vectors against a single weight read; dispatch is M = 1 to the plain GEMV, 2..8 to the batched kernel, above that to the GEMM. The four legacy kernels share one macro template. Measured per-token matmul cost at M = 2 drops 3.8x (Q4_K), and end-to-end aggregate throughput on Qwen3-4B-Q4_K_M scales 1.46x / 1.62x / 1.69x at concurrency 2 / 4 / 8 instead of regressing.
• Register-resident dequant for heavy quants (Q5_K, Q6_K, Q3_K): the batched kernels initially re-dequantized the weights for every token in the batch, which kept their per-token cost flat. The dequant now runs once per (row, super-block) with the scales folded into register-resident weights, making the inner loop pure FMA. Per-token cost at M = 8: Q5_K 32 to 13.1 us, Q6_K 25 to 15.2 us, Q3_K 26 to 13.6 us. With the Q6_K tied LM head batched, Qwen3-4B-Q5_K_M reaches 79 aggregate tok/s at concurrency 8 (2.47x).
• Tied-embedding LM head stays quantized (src/models/gguf_model.rs): GGUF checkpoints without a separate output.weight previously dequantized the embedding table into a dense BF16 [vocab, hidden] LM head (778 MB on Qwen3-4B) that consumed ~19% of every decode step. The tied branch now reuses the quantized embedding tensor directly through the fast GEMV: 28.95 to 36.5 tok/s (+25%) and ~0.5 GB less resident memory on Qwen3-4B-Q4_K_M. Models with a separate output.weight are unaffected.
• Non-greedy sampling fast paths (src/sampling.rs): apply_top_p no longer sorts the full vocabulary per token. A single scan collects candidates above max_prob * 1e-4; when their mass covers top_p (measured 100% of the time on real model logits, 6-7 candidates on average) the global prefix is provably contained in them and only the candidates are sorted, with a full-sort fallback for flat distributions. The log-probability pass (a full-vocabulary ln) is computed only when top_logprobs is requested, and top_k_by_logprob selects before sorting. Sampled decode now costs ~0.4 ms/token over greedy instead of ~2.2 ms.
• MoE decode dispatch de-glued (src/common/moe.rs): the decode path read a dense per-expert mass vector back from the GPU each layer to find the active experts. It now reads the (tiny) top-k routing result once and builds the per-expert weights on the CPU, removing the dense gate tensor, the scatter_add, and the full-width readback. Throughput-neutral on its own but simpler and covered by a new naive-equals-sparse equivalence test.

Reliability and Correctness

• Download resume (src/models/hub.rs): three related defects found by two consecutive real network failures. (1) is_incomplete_download treated any directory containing at least one .safetensors file as complete, so a truncated shard passed the check; sharded checkpoints are now validated against model.safetensors.index.json (every shard present and on-disk bytes covering total_size). (2) The failure cleanup deleted the completed files and kept the truncated in-fl...

0.0.0-alpha.12.0.1

• Fixed nondeterministic garbage output on Metal under concurrent serving — including the intermittent Qwen3-4B-Instruct-2507-FP8 gibberish — by pinning candle's Metal command-buffer pool to a single buffer; throughput is unchanged.
• Fixed every Docker image build (CPU and all CUDA architectures) failing on a missing compile-time OXYDLLM_BUILD_TS.
• Silenced a dead_code warning emitted on the non-Metal CI test builds.
• Removed a full-vocabulary F32 materialization from the greedy per-token NaN guard.

Reliability and Correctness

• Metal command-buffer pool corruption under concurrency (src/main.rs): candle-metal-kernels 0.10.2 spreads GPU work across a pool of command buffers (CANDLE_METAL_COMMAND_POOL_SIZE, default 5) that is not safe for concurrent encoding. When more than one thread touches the device at once — the server's tokio workers, the model-load/warmup thread, and off-thread Tensor drops freeing MTLBuffers — operations collide and the output is silently corrupted (gibberish or NaN). This is why Qwen3-4B-Instruct-2507-FP8 decoded coherently on some loads and produced garbage on others, but it affected any model served under concurrency. main() now sets CANDLE_METAL_COMMAND_POOL_SIZE=1 at startup, before any device or thread is created, serializing onto one command buffer. GPU work is already serialized per device by the internal GPU lock, so there is no throughput cost — measured 53.32 vs 53.30 tok/s on Qwen3-0.6B (pool size 1 vs 5). Set the environment variable explicitly to override. With the fix the full stress_baseline.py run passes 25/25 models on the coherence check (the FP8 model previously failed).

Performance and Efficiency

• Greedy decode NaN guard (src/sampling.rs): the temperature == 0 fast path no longer promotes the entire logit row (~152 K values on the Qwen vocab) to F32 just to check for NaN. It reduces in the native dtype and casts only the one-element sum, so the argmax token id is the only value copied off the GPU per step.

CI / Infra

• Docker / CUDA image builds fixed (src/main.rs, build.rs): every Docker target (CPU and all CUDA architectures) failed with error: environment variable OXYDLLM_BUILD_TS not defined at compile time. The Dockerfile's dummy-build layer — build dependencies with a stub main, then rebuild against the real sources — does not reliably reapply the build script's compile env to the final compile. env!("OXYDLLM_BUILD_TS") is now option_env!(…): it is optional build metadata and the code already defaulted to 0. The dist_build cfg gating oxydllm update / uninstall was likewise replaced by an OXYDLLM_DIST_BUILD compile env (emitted by build.rs, read via option_env!), removing the custom cfg, its cargo::rustc-check-cfg declaration, and any need for an #[allow(unexpected_cfgs)].
• dead_code warning on non-Metal builds (src/common/awq.rs): QuantWeight::runtime_size_bytes and to_device are Metal-only but were gated #[cfg(any(feature = "metal", test))], so they compiled — unused — into the CUDA/CPU test binaries and emitted a dead_code warning on those CI jobs. Narrowed to #[cfg(feature = "metal")].

Tests

• Metal command-buffer race reproducer (src/common/linear.rs::metal_pool_ordering_race_repro, env-gated): several threads run the same deterministic candle-matmul + custom Metal rms_norm chain on one shared device; identical results across threads prove correctness, while divergence or NaN exposes the pool race. Documents the bug and confirms CANDLE_METAL_COMMAND_POOL_SIZE=1 fixes it.

Installation

macOS (Apple Silicon)

curl -fsSL https://raw.githubusercontent.com/giovannifil-64/oxydllm/main/install.sh | sh

Linux x86_64 (CUDA — Ada / Hopper / Blackwell / Blackwell Ultra / Blackwell Desktop)

curl -fsSL https://raw.githubusercontent.com/giovannifil-64/oxydllm/main/install.sh | sh

Linux arm64 (CUDA — GH200 / GB300 / Jetson Thor / DGX Spark)

curl -fsSL https://raw.githubusercontent.com/giovannifil-64/oxydllm/main/install.sh | sh

Installer auto-selects the right binary from detected GPU compute capability.
Override with OXYDLLM_CUDA_TARGET=<ada|hopper|blackwell|blackwell-ultra|blackwell-desktop> (x86_64)
or OXYDLLM_CUDA_TARGET=<hopper|blackwell|blackwell-ultra|thor|blackwell-desktop> (arm64).

Linux (CPU only)

curl -fsSL https://raw.githubusercontent.com/giovannifil-64/oxydllm/main/install.sh | OXYDLLM_NO_GPU=1 sh

Docker (CUDA)

docker run --gpus all -p 11313:11313 \
  -v ~/.oxydllm/models:/root/.oxydllm/models \
  ghcr.io/giovannifil-64/oxydllm:latest start

x86_64 explicit CUDA tags: :cuda-ada, :cuda-hopper, :cuda-blackwell, :cuda-blackwell-ultra, :cuda-blackwell-desktop.
ARM64 explicit CUDA tags: :cuda-hopper-arm64, :cuda-blackwell-arm64, :cuda-blackwell-ultra-arm64, :cuda-thor-arm64, :cuda-blackwell-desktop-arm64.

Docker (CPU)

docker run -p 11313:11313 \
  -v ~/.oxydllm/models:/root/.oxydllm/models \
  ghcr.io/giovannifil-64/oxydllm:cpu start

SHA-256 checksums

See the .sha256 files attached to this release.

Full Changelog: nightly...0.0.0-alpha.12.0.1

0.0.0-alpha.12

• Mixture-of-Experts support shipped end-to-end: Qwen3MoeForCausalLM and OlmoeForCausalLM are routed through a new MoeFeedForward module with top-k router, hybrid sparse/naive dispatch chosen per call from n_tokens vs top_k, and runtime-verified output coherence on allenai/OLMoE-1B-7B-0924-Instruct.
• AWQ checkpoints now stay packed in memory on Metal: a fused W4A16 GEMV kernel (PackedQuantLinear) cuts resident weight memory ~3× (Qwen3-4B-AWQ: 7.5 → 2.5 GB) while keeping decode throughput on par with the previous fp16 dequant-at-load path.
• W8A16 GEMV + dequant wrappers built off the same bit-parametric AWQ template; hf_parser now accepts AWQ with bits ∈ {4, 8} and dispatches the right kernel at runtime.
• GPTQ checkpoints (4-bit and 8-bit, desc_act=false) load and run end-to-end with a dedicated Metal resident kernel family (gptq{4,8}_gemv_* + dequantize_gptq{4,8}_*); Qwen/Qwen3-{0.6B,1.7B}-GPTQ-Int8 reach 85+ tok/s decode on Apple Silicon.
• Bf16-aware GGUF Metal fast path now covers ten quant types — Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q2_K, Q3_K, Q4_K, Q5_K, Q6_K — with a fused mul_mm for prefill that dequantizes inline and removes the previous 2× residency cost on the M=1 fast path.
• GGUF loader rewritten on top of memmap2: Qwen3-4B-Q4_K_M cold load drops from 9.4 s to 2.7 s (3.47×) with bit-identical outputs on the verified checkpoint set.
• FP8 dequant on Metal is now numerically correct on block-wise-scaled checkpoints (Qwen3-4B-Instruct-2507-FP8): the weight × scale_inv multiply is promoted to F32 to avoid the BF16 mantissa loss that produced coherent-but-wrong outputs through 36 layers of rescaling.
• HTTP API hardened with optional Bearer authentication, per-request wall-clock timeout, sampling parameter range validation, and queue-full backpressure.
• Batched scheduler now derives max_num_seqs automatically from the available KV cache and exposes a bounded request queue via --max-queued-requests.
• Prometheus metrics endpoint (/metrics) exposes TTFT, decode throughput, queue depth, prefix-cache hit ratio, model weight memory, and KV cache allocation.
• New CLI commands: oxydllm update [--pre|--nightly] for self-update via GitHub releases and oxydllm uninstall [--purge] for clean removal of binary + OS service.
• CUDA_COMPUTE_CAP is validated at compile time against a curated list of supported architectures; the loader also warns when the binary's compiled cap is below the hardware cap.
• GPU is now required by default; pass --allow-cpu (or OXYDLLM_ALLOW_CPU=1) to opt into the CPU fallback path.
• New Metal fused kernels (GatedGeLU-Tanh, GeLU-Tanh-Mul) for Gemma-family activations, plus a NaN-propagation fix in SDPA full mode for sequences past 16 tokens.
• Per-layer KV contig buffer pool (ContigBufferPool) recycles tensors across sequences, removing the per-fresh-prefill zero-fill allocation.
• A focused code review pass identified 34 findings and closed 31 of them across server, performance, robustness, model loading, and engine.

New Features

• Mixture-of-Experts (MoE) (src/common/moe.rs): MoeFeedForward { router, experts, top_k, activation, norm_topk } with top-k softmax routing (arg_sort_last_dim + gather + scatter_add). The transformer block dispatches to a dense FeedForward or MoeFeedForward based on BlockConfig.moe. Two equivalent forward paths are picked per call:
  • Naive (n_tokens ≤ top_k): dense gate via scatter_add, run each non-empty expert on the full x_flat. Decode-friendly (no per-expert index_select / index_add overhead).
  • Sparse (n_tokens > top_k): group token indices per expert on the CPU (small: n_tokens × top_k ints), then index_select + FFN + index_add per non-empty expert. Per-expert compute drops from n_tokens to ~n_tokens × top_k / num_experts — up to ~8× on OLMoE-1B-7B for long prefill.
• OLMoE / Qwen3-MoE support (src/models/arch_defaults.rs, src/models/parsers/hf_parser.rs): OlmoeForCausalLM and Qwen3MoeForCausalLM are now in arch_defaults. hf_parser reads num_experts / num_local_experts, num_experts_per_tok, and norm_topk_prob from config.json. Mixtral / DeepSeek-V2/V3 remain unsupported (different tensor naming or MLA attention).
• QkNormLayout auto-detection (src/common/attention.rs): q_norm / k_norm weight shape is inspected at load; [head_dim] selects per-head normalisation (Qwen3, Gemma3), [n_heads × head_dim] selects flat normalisation applied to [B, T, q_dim] before reshape (OLMoE). Forward branches on the detected layout; no config flag added because the two cases are mutually exclusive at the shape level.
• AWQ W4A16 resident path on Metal (src/common/quant_kernels.metal, src/common/metal_ops.rs, src/common/linear.rs): packed 4-bit weights stay resident; decode runs a fused split-K GEMV kernel (w4a16_gemv_*), prefill/M>1 uses an inline dequantize_w4_* plus matmul. Qwen3-4B-AWQ resident weight memory drops from ~7.5 GB (fp16 dequant-at-load) to ~2.5 GB.
• W8A16 (8-bit AWQ) kernel wrappers: w8a16_gemv_{f16,bf16} and dequantize_w8_{f16,bf16} are bit-parametric instantiations of the same template as the 4-bit path. PackedQuantLinear carries a bits field and dispatches to the right kernel; validate_quantization_config accepts awq with bits ∈ {4, 8}.
• GPTQ end-to-end support: validate_quantization_config now accepts gptq with bits ∈ {4, 8} and desc_act=false; weights.rs::try_get_gptq loads qweight (packed along in_features), qzeros, scales, and optional g_idx; dequantize_gptq implements the PlusOne zero-point convention. A QuantWeight struct unifies AWQ and GPTQ behind a single AnyLinear::from_quant factory with a pack_dim field that disambiguates the layouts.
• GPTQ resident Metal kernel (src/common/quant_kernels.metal): a dedicated gptq_gemv_impl<T, BITS> template handles the GPTQ [in/pack_factor, out] layout (1 thread per output column instead of AWQ's 1 thread per pack_factor columns) plus gptq_dequantize_impl<T, BITS>. Wrappers gptq{4,8}_gemv_{f16,bf16} + dequantize_gptq{4,8}_{f16,bf16} are dispatched by PackedQuantLinear based on pack_dim. Decode on Qwen3-0.6B-GPTQ-Int8 improves from ~50 tok/s (dequant-at-load) to ~89 tok/s with the resident path.
• GGUF kernel suite bf16-aware (src/common/quant_kernels.metal): ten quant types — Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q2_K, Q3_K, Q4_K, Q5_K, Q6_K — each with a gguf_*_gemv_bf16 (M=1 decode) and a fused gguf_*_mul_mm_bf16 (M>1 prefill, dequant inline). GgufFastPath in linear.rs::QLinear activates the fast path on Metal + BF16; QMatMul fallback is dropped when active so weights are not held twice (no 2× memory cost). Coverage spans ~98% of the GGUF mainstream parc.
• GGUF mmap loader (src/common/gguf_weights.rs): GgufWeights::load and load_shards open the file with memmap2::Mmap, parse the header from a Cursor, and build QTensor from slices of the mmap (zero copy in user space; one memcpy into MTLBuffer). parallelise_tensor_load materialises tensors in parallel via rayon::par_iter. End-to-end load + first inference on Qwen3-4B-Q4_K_M drops from 9.4 s to 2.7 s (3.47×).
• Stress baseline script (scripts/stress_baseline.py): per-model cold/warm load time, TTFT, median decode TPS, RSS, and deterministic coherence check ("Tokyo is the capital of" → must contain "japan"). Writes CSV + JSON per run under test-results/stress-baseline/ for regression tracking.
• HTTP API authentication (src/server/routes/mod.rs): optional OXYDLLM_API_KEY / --api-key enables a Bearer-token middleware on /v1/* and /metrics; X-API-Key is accepted as an alternative header; /health stays unauthenticated for liveness probes; mismatches return 401 invalid_api_key via a constant-time comparison.
• Per-request wall-clock timeout (--request-timeout / OXYDLLM_REQUEST_TIMEOUT, default 300 s, 0 disables): non-streaming responses return 408 Request Timeout; streaming responses emit a final request_timeout error chunk followed by [DONE] via a watchdog that races the SSE task. Dropping the inner future closes the engine channel so the underlying sequence is aborted at the next step boundary.
• Sampling parameter range validation (validate_sampling_params in chat.rs): out-of-range values for temperature, top_p, min_p, frequency_penalty, presence_penalty, top_logprobs, repetition_penalty, n, max_tokens, and max_completion_tokens return 400 invalid_request_error instead of silently degrading the sampler. Also closes the user-triggerable NaN-logits path through repetition_penalty: 0.
• Batched scheduler with dynamic concurrency: at model load, max_num_seqs is computed as total_kv_blocks / ceil(max_context_len / block_size), capped at 256, and logged. Overridable via --max-num-seqs / OXYDLLM_MAX_NUM_SEQS. Requests are held in a bounded tokio::sync::mpsc channel of capacity --max-queued-requests (default 200); once full, new arrivals receive HTTP 429 immediately.
• Prometheus metrics endpoint (/metrics, src/server/routes/metrics.rs): oxydllm_ttft_milliseconds (histogram), oxydllm_tokens_per_second (histogram), oxydllm_requests_total (counter by status), oxydllm_queue_depth (gauge), oxydllm_prefix_cache_requests_total (counter by hit/miss), oxydllm_model_weights_bytes (gauge), oxydllm_kv_cache_allocated_bytes (gauge), oxydllm_vram_used_bytes (gauge). Every request is also assigned a UUID-v4 request_id that propagates through structured logs; setting LOG_FORMAT=json switches the logger to JSON-per-line for Loki / Datadog / jq.
• oxydllm update [--pre|--nightly]: queries the GitHub releases API for the latest stable / pre-release / nightly build, compares against the version compiled int...

0.0.0-alpha.11

• Added AWQ quantization support across attention, FFN, and lm_head, with QKV and gate+up fused at load time.
• New Metal-accelerated fused kernels for gated_silu, silu_mul, and logit/attention softcap.
• Added bias loading for QKV and output projections in both safetensors and GGUF paths.
• Reworked streaming decode to always emit from a full-sequence canonical decode, eliminating BPE drift and double-spacing artifacts.
• Introduced chunked prefill for large batched prompts to reduce peak memory during the prefill phase.

New Features

• AWQ weight loading (src/common/awq.rs): AwqRawTensors, dequantize_awq, concat_awq_along_out, and AWQ_PACK_ORDER = [0, 2, 4, 6, 1, 3, 5, 7]. Auto-detected via ModelWeights::try_get_awq(prefix) for q/k/v/o, gate/up/down, and lm_head. Dequantization happens at load time, so no Metal kernel changes are required at inference time.
• AWQ attention loader: when qkv_proj is detected as AWQ, q/k/v are concatenated along the output dimension and loaded as a single fused projection (matmul count per layer = 2, identical to the fp16 path). Mixed-format layers are rejected with a descriptive error.
• AWQ FFN loader: gate_proj + up_proj AWQ tensors are fused into a single packed projection; down_proj is loaded as a separate AWQ linear.
• AnyLinear::from_awq constructor for building linear layers directly from AWQ raw tensors with optional bias.
• Quantization-config validation in the HF parser (hf_parser.rs) ensures only AWQ configurations supported by the runtime are accepted.
• Bias support for attention projections: q_proj.bias, k_proj.bias, v_proj.bias, and o_proj.bias are now loaded in both the safetensors and GGUF (attn_q.bias, attn_k.bias, attn_v.bias, attn_qkv.bias, attn_output.bias) paths. Fused QKV biases are concatenated to match the fused weight layout.
• Chunked prefill (pick_prefill_chunk_split in engine.rs): batches with two or more prefill sequences and a combined uncached length ≥ 1024 tokens are split into two halves and forwarded sequentially, sharing the same KV cache slices. Single-sequence prefills and small batches keep the original single-pass path.
• flush_caches extracted as a reusable helper in block.rs so the engine can flush pending KV writes after each prefill chunk.

Performance and Efficiency

• Metal gated_silu_fused: single-kernel silu(x[..H]) * x[H..] for packed gate+up activations (Phi-3 / Phi-3.5 and AWQ-fused FFN paths), avoiding two separate kernel launches and intermediate buffers.
• Metal silu_mul_fused: single-kernel silu(gate) * up for the standard split gate/up FFN layout.
• Metal softcap_fused: single-kernel tanh(x / cap) * cap used by Gemma2/3 attention softcap and final logit softcap; replaces the three-op (x / cap).tanh() * cap chain.
• Attention scratch buffer: added out_buf: RefCell<Option<Tensor>> to Attention for reusing the output projection tensor across decode steps.
• KV-quant flush dtype reorder: to_device(Cpu) is now applied before to_dtype(F32) when reading quantized K/V back to host memory, so the (potentially expensive) dtype conversion runs on CPU instead of issuing an extra GPU cast.
• AWQ runtime size estimation in estimate.rs excludes pre-dequantization scratch tensors so the reported in-memory footprint matches the actual fp16/bf16 weight size after dequantization.

Reliability and Correctness

• Streaming decode rewrite (src/server/routes/engine_loop.rs, src/main.rs): removed the per-token fast path that re-inserted a leading space when the vocab piece started with ▁ or Ġ. The heuristic could double-space ("assistente" + " " + "virtuale" → "assistente virtuale") and produce character drift that later swallowed user-visible characters. Streaming now always decodes the full accumulated id list and emits only the suffix beyond decoded_len, with trailing U+FFFD held back until the continuation token arrives. The piece field on the decode cache is no longer needed and was removed.
• Output of fused QKV biases requires either all of q/k/v biases or none; mismatched bias presence falls back to a separate (non-fused) projection layout to avoid silent shape errors.
• AWQ projections validate that out-features (scales.dim(1)) match n_heads * head_dim (q) and n_kv_heads * head_dim (k/v) before fusing.

Refactors and Maintainability

• Engine prefill body extracted into a small closure so the chunked and single-pass paths share cache-slice handling and flush_caches is called uniformly at the end.
• engine_loop.rs split the streaming decode into a pure emit_suffix(full, decoded_len) helper that is independently unit-tested.

Tests

• Streaming decode unit tests (streaming_decode_tests in engine_loop.rs): cumulative-emission matches canonical decode, no double-spacing across Ġ-prefixed tokens, no character loss across partial-UTF-8 token boundaries, idempotency when decoded_len == full.len(), defensive clamping past end of full, and char-boundary clamping inside multibyte sequences.
• AWQ unit tests covering load round-trips and runtime-size accounting.
• Engine chunk-split tests (pick_prefill_chunk_split): small-batch skip, balanced two-sequence split, dominant-first-sequence skip, and three-sequence balancing.
• Architecture regression coverage extended with additional run_prefill slice setups; existing slice initialization simplified.

Documentation

• README updated to clarify model-compatibility notes.

CI / Infra

• Release notes now include the CHANGELOG.md section for the tag: previously the release workflow set body: to the Installation block alone, fully overwriting the release body and leaving the changelog content visible only in the repository file. A new Extract changelog section for this tag step in release.yml slices the relevant block out of CHANGELOG.md (from ## <version> to the next ## heading) and prepends it to the body, with the Installation/checksums block following and the auto-generated **Full Changelog** link appended by softprops/action-gh-release at the end.
• Heading match is literal, not regex: the awk extractor matches the version heading via index() rather than interpolating the tag into a regex. This makes characters like ., + (semver build metadata), and - behave as plain text and removes the need to escape future tag formats.
• Prefix-collision guard: the heading match also requires either an exact length match or a trailing whitespace character, so a tag 1.2 cannot accidentally match a heading ## 1.2.3.
• Inline **Full Changelog** and --- separators stripped from extracted sections: the manual **Full Changelog** line at the bottom of each changelog section is dropped during extraction so it doesn't duplicate the auto-generated comparison link, and standalone --- separators between historical sections are removed for the same reason.
• Missing-entry fallback: if a tag is pushed without a matching ## <version> in CHANGELOG.md, the release proceeds with a placeholder body and a GitHub Actions ::warning:: instead of failing, so a forgotten changelog entry never blocks a hotfix.

Full Changelog: 0.0.0-alpha.10.0.1...0.0.0-alpha.11

0.0.0-alpha.10.0.1

• Fixed release workflow stalling indefinitely on tag push due to missing repository context and insufficient permissions.
• Replaced deprecated rustsec/audit-check with taiki-e/install-action + cargo audit.
• Migrated all CI and build workflow cache jobs from actions/cache to Swatinem/rust-cache.
• Restricted CI cache writes to main branch only, reducing total cache storage from ~9.2 GB to ~2.6 GB.
• Added path filters to CI so it only triggers on code changes, not on docs or workflow-only commits.

CI / Infra

• Release workflow — repository context: verify-ci was calling gh run list without --repo, causing it to fail silently in the ephemeral runner environment (no git checkout in that job). Added --repo ${{ github.repository }}.
• Release workflow — permissions: The workflow-level permissions block (contents: write, packages: write) implicitly set actions: read to none. gh run list was failing with HTTP 403, silently caught by 2>/dev/null || echo "[]", making every poll return status=<none>. Fixed by adding permissions: actions: read at the verify-ci job level.
• Release workflow — polling timeout: Reduced polling attempts from 40 to 20 (10 minutes). CI completes well within this window under normal conditions.
• Security audit: Replaced deprecated rustsec/audit-check@v2 (required checks: write and is no longer maintained) with taiki-e/install-action@v2 + cargo audit. Installs a prebuilt binary in ~5 s with no extra permissions needed.
• CI cache: Replaced all actions/cache@v5 blocks in ci.yml with Swatinem/rust-cache@v2, which automatically prunes incremental/ and .fingerprint/ directories before saving. CUDA cache size reduced from ~490 MB to ~265 MB per architecture. Added save-if: ${{ github.ref == 'refs/heads/main' }} — PR runs restore from existing caches but do not write new entries. Total cache storage reduced from ~9.2 GB to ~2.6 GB.
• Build cache: Replaced all actions/cache@v5 blocks in build.yml with Swatinem/rust-cache@v2. Each native binary build job uses a distinct prefix-key to prevent cache collisions with CI jobs. No save-if restriction since build jobs are only triggered by nightly cron and release tags, never by PRs.
• CI path filters: Added paths filters to both push and pull_request triggers. CI now only runs when src/**, Cargo.toml, Cargo.lock, rust-toolchain.toml, install.sh, or .github/workflows/ci.yml change. Commits that only touch docs, images, or other workflow files no longer trigger a full CI run.

Full Changelog: 0.0.0-alpha.10...0.0.0-alpha.10.0.1

Installation

macOS (Apple Silicon)

curl -fsSL https://raw.githubusercontent.com/giovannifil-64/oxydllm/main/install.sh | sh

Linux x86_64 (CUDA — Ada / Hopper / Blackwell / Blackwell Ultra / Blackwell consumer)

curl -fsSL https://raw.githubusercontent.com/giovannifil-64/oxydllm/main/install.sh | sh

Linux arm64 (CUDA — GH200 / DGX Spark / GB300 / Jetson Thor)

curl -fsSL https://raw.githubusercontent.com/giovannifil-64/oxydllm/main/install.sh | sh

Installer auto-selects the right binary from detected GPU compute capability.
Override with OXYDLLM_CUDA_TARGET=<ada|hopper|blackwell|blackwell-ultra|blackwell-consumer> (x86_64)
or OXYDLLM_CUDA_TARGET=<hopper|blackwell|blackwell-ultra|thor> (arm64).

Linux (CPU only)

curl -fsSL https://raw.githubusercontent.com/giovannifil-64/oxydllm/main/install.sh | OXYDLLM_NO_GPU=1 sh

Docker (CUDA)

docker run --gpus all -p 11313:11313 \
  -v ~/.oxydllm/models:/root/.oxydllm/models \
  ghcr.io/giovannifil-64/oxydllm:latest start

x86_64 explicit CUDA tags: :cuda-ada, :cuda-hopper, :cuda-blackwell, :cuda-blackwell-ultra, :cuda-blackwell-consumer.
ARM64 explicit CUDA tags: :cuda-hopper-arm64, :cuda-blackwell-arm64, :cuda-blackwell-ultra-arm64, :cuda-thor-arm64.

Docker (CPU)

docker run -p 11313:11313 \
  -v ~/.oxydllm/models:/root/.oxydllm/models \
  ghcr.io/giovannifil-64/oxydllm:cpu start

SHA-256 checksums

See the .sha256 files attached to this release.

Full Changelog: 0.0.0-alpha.10...0.0.0-alpha.10.0.1

0.0.0-alpha.10

• Project renamed from rllm to oxydllm — binary, data directory, registry file, and env vars all updated.
• Added Phi-3 / Phi-3.5 support (safetensors + GGUF), including fused weight projection and LongRoPE scaling.
• Added Mistral3ForConditionalGeneration architecture support.
• Introduced OpenAI-compatible function calling and structured output (JSON Schema validation).
• Added FP8 weight loading with runtime dequantization and per-tensor scale handling.
• Built out a full CI/CD pipeline: CUDA multi-arch builds, ARM64, multi-platform Docker images, and architecture regression tests.
• New CLI commands: oxydllm list and oxydllm version.
• Integrated tracing for structured logging across the codebase.

New Features

• Project-wide rename: binary oxydllm, data dir ~/.oxydllm/, registry .oxydllm_registry.json, env var OXYDLLM_DEVICES.
• Phi-3 / Phi-3.5 model support: fused qkv_proj / gate_up_proj weight handling in both safetensors and GGUF; build_llama_tokenizer for GGUF tokenizer type; LongRoPE (RopeScaling::LongRoPE) for Phi-3.5.
• Mistral3ForConditionalGeneration added to the supported architecture list; text_config nesting in HF parser handled for multimodal configs.
• OpenAI-compatible function calling: tool definitions, tool-call detection, ToolCallDelta streaming, finish_reason: "tool_calls".
• Structured output with JSON Schema validation (type, required, additionalProperties, properties, items, enum); invalid output yields finish_reason: "content_filter".
• FP8 weight loading with per-tensor _scale_inv / scale dequantization at load time.
• Hadamard transform in KV quantization with fallback for non-power-of-two head_dim.
• Alternating sliding-window attention support at the layer level.
• Per-device GPU locking mechanism replacing the global lock for finer-grained concurrency control.
• oxydllm list command: shows locally available models with size, architecture, and last-used date.
• oxydllm version command.
• Graceful server shutdown with configurable timeout and OS signal handling.
• Abort-sequence API: abort_sequence in engine, scheduler, and routes for cancelling in-flight requests.

Performance and Efficiency

• FP8 runtime dequantization avoids storing full-precision weight copies, reducing peak memory at load time.
• apply_qk_with_positions in RoPE for optimised per-position tensor handling.
• KV quantization Hadamard transform improves quantization quality for large head dims.
• GateUpProjection::Packed variant for pre-fused gate+up tensors (Phi-3 / Phi-3.5) avoids splitting overhead.

Reliability and Correctness

• Fixed Gemma2 arch defaults: FFN pre/post norms were inadvertently disabled.
• Fixed RoPE dims4 handling on the Metal feature flag path.
• Added require-gpu guard and hardened attention/causal-mask paths against out-of-bounds indexing.
• Fixed case-insensitive model path resolution and improved FP8 tensor key matching.
• Fixed leading-whitespace stripping in token decoding for both server streaming and interactive mode.
• Fixed Cargo release profile settings that were causing oversized debug builds.
• Causal mask functions now accept an explicit DType to avoid implicit promotions.

Refactors and Maintainability

• routes.rs split into focused sub-modules for improved readability.
• Device key representation migrated to DeviceLocation enum.
• QkvProjection now uses AnyLinear internally; GateUpProjection extended with Packed and Simple variants.
• Registry management switched from HashMap to BTreeMap for deterministic ordering.
• Removed unused user field from ChatCompletionRequest.
• tracing integrated for structured, levelled logging across engine, scheduler, and server.

Tests

• Unit tests for Attention and RotaryEmbedding.
• Unit tests for ModelManager and tokenizer (error handling, encoding round-trips).
• Architecture regression tests for StandardTransformer (CPU, run in pre-push hook and CI).

CI / Infra

• Full GitHub Actions pipeline: CPU, CUDA (multi-arch: Ampere, Ada, Hopper, Blackwell Ultra), ARM64, macOS.
• Multi-platform Docker images for CPU with manifest-list support.
• Nightly build workflow with GHCR image cleanup for untagged images.
• Architecture regression test step gated before release publishing.
• Docker fallback: rebuild without cache on image pull failure.
• Binary stripping for macOS and Linux release artifacts.
• Rust toolchain version pinned in rust-toolchain.toml and read by all workflows.

Dependencies

• candle-core / candle-metal-kernels updated to 0.10.2.
• candle / cudarc updated to 0.10.1 / 0.19.4.
• Added fastrand and tempfile.
• action-gh-release upgraded to v3 in nightly and release workflows.
• General dependency updates (Cargo.lock bump).

Full Changelog: 0.0.0-alpha.9...0.0.0-alpha.10

0.0.0-alpha.9

• Added Gemma4 support with stronger per-layer transformer configuration.
• Introduced KV cache quantization, plus QJL quantization for key residuals.
• Expanded OpenAI API compatibility with missing endpoints, fields, response objects, and error formats.
• Added system fingerprint generation for chat completion model identification.
• Improved sampling controls with logprobs, top_logprobs, logit_bias, and repetition window support.

New Features

• Gemma4 architecture support and related model-loading/config upgrades.
• KV cache quantization path for reduced memory usage.
• QJL key-residual quantization support in the KV pool (--qjl-quantization).
• Repetition-window control for improved anti-repetition behavior.
• Extended sampling outputs to return token logprobs and top-logprobs.
• System fingerprint in chat completion responses.
• Broader OpenAI-compatible API surface and schema-aligned responses.

Performance and Efficiency

• Quantized KV pool handling for lower memory footprint.
• Deferred-write and allocator updates for quantized cache paths.
• Separate key/value quantization size handling for tighter memory control.
• End-to-end propagation of quantization settings through loader/manager/scheduler flow.

Reliability and Correctness

• Improved OpenAI-style error response formatting and route behavior.
• Better tokenizer handling for special tokens and chat templates.
• Stronger parser/config handling for advanced per-layer model settings.

Refactors and Maintainability

• Removed unused bytes_per_head from KvQuantizer.
• Internal cleanup around sampling output structures and KV quantization flow.

Dependencies

• Updated windows-sys dependency.

Full Changelog: 0.0.0-alpha.8...0.0.0-alpha.9

0.0.0-alpha.8

• Added sliding-window support and improved normalization handling for model execution.
• Introduced Metal-accelerated ops for RMSNorm, Softmax, and RoPE, with SDPA logic refactoring.
• Expanded RopeScaling support (including additional YaRN parameters) and updated parsing.
• Added abort capabilities in engine/scheduler flows for running sequence control.
• Improved model lifecycle management with model removal and registry handling improvements.

New Features

• Sliding-window attention and related cache/model-path improvements.
• Abort functionality in engine and scheduler paths.
• Completion token tracking in engine events.
• Optional bias support in attention-related linear projection.
• Support for known unsupported architectures in defaults/parsing, with better surfacing.
• Better message truncation behavior in interactive mode.
• Additional file-type support in model size estimation.

Performance and Efficiency

• Metal kernel usage for key transformer primitives (RMSNorm, Softmax, RoPE).
• Attention and paged KV cache optimizations for tensor handling and memory efficiency.
• Ensured tensor contiguity before critical ops in attention/cache paths.
• Simplified attention path by removing unnecessary padding logic.
• Feed-forward path optimized via GateUpProjection enum restructuring.

Reliability and Correctness

• Improved error handling across model loading, chat template application, engine loop, and registry save flow.
• Added abort mechanism for consecutive engine errors.
• Enforced max_tokens limit in chat completions.
• Corrected architecture display for Qwen2 and Qwen3 in GGUF discovery.
• Improved transformer layer validation logic.

Refactors and Maintainability

• Core module maintainability refactors across attention/block/mask/prefix-cache/sampling/routes.
• Simplified token decoding logic in interactive and request enqueue flows.

Dependencies

• Removed unused rayon dependency.
• Updated Candle package source/version in Cargo.toml and Cargo.lock.