Skip to content

0.0.0-alpha.13

Pre-release
Pre-release

Choose a tag to compare

View all tags
@giovannifil-64 giovannifil-64 released this 14 Jul 19:01
0.0.0-alpha.13
bc120a0
  • IBM Granite 3.x dense support (GraniteForCausalLM), safetensors and GGUF, verified E2E on granite-3.3-2b-instruct across chat, streaming, concurrency, multi-turn prefix reuse, tool calling, and structured output.
  • A correctness fix that affected every Llama-family GGUF file (arch llama and granite): the converter's per-head q/k row interleave was never undone for our NeoX-style RoPE, so these models produced fluent but positionally corrupted output. Llama 3.x GGUFs were broken this way since GGUF support landed.
  • candle 0.11.0 and tokenizers 0.23.1 upgrade. Four Metal-backend issues in candle 0.11 were identified with dedicated repros and worked around; model loading is now two-phase (parallel CPU decode, sequential drained device transfers), which also cut FP8 warmup from 326 s to 0.5 s.
  • Quantized GEMV kernels are now bitwise deterministic: the AWQ/GPTQ split-K epilogue accumulated partial sums with float atomic_fetch_add, whose scheduling-dependent order flipped borderline tokens at temperature 0 roughly one run in five on GPTQ-Int8 checkpoints.
  • OpenTelemetry integration: OTLP trace export (request-lifecycle span with a decode child span whose offset visualises TTFT) and expanded Prometheus metrics with per-model labels and memory gauges.

New Features

  • Granite architecture (src/models/parsers/hf_parser.rs, src/models/arch_defaults.rs, src/common/block.rs, src/common/config.rs): the four Granite scalar multipliers map onto the blueprint: attention_multiplier is the softmax scale itself (not an inverted divisor), embedding_multiplier scales embeddings, the new residual_multiplier scales every sub-layer output before its residual add, and the new logits_scaling divides the final logits. GGUF metadata keys (granite.attention.scale, granite.embedding_scale, granite.residual_scale, granite.logit_scale) map to the same fields, treating absent and non-positive values as unpublished. GraniteMoeForCausalLM and GraniteMoeHybridForCausalLM are rejected with explicit reasons (fused 3D expert tensors / Mamba2 layers).
  • OpenTelemetry (src/telemetry.rs, src/main.rs, src/server/routes/engine_loop.rs): --otel-endpoint (server only) installs an OTLP layer at subscriber init; each request carries a lifecycle span with model, token counts, and finish reason, plus a decode child span from first token to completion. Prometheus metrics gained per-model labels, TTFT histograms, and memory gauges driven by the engine loop.

Reliability and Correctness

  • Llama-family GGUF q/k de-interleave (src/common/gguf_weights.rs::depermute_qk_rows, ArchDefaults.gguf_qk_permuted): convert_hf_to_gguf.py reorders each head's q/k projection rows from the HF [first_half | second_half] layout to the interleaved layout llama.cpp's paired-rotation RoPE expects; our RoPE is NeoX/HF split-half. The de-interleave is a pure row-wise byte shuffle, valid for every GGML dtype because quantization blocks run along the input dimension. Verified: bartowski/Llama-3.2-1B-Instruct-Q4_K_M now answers byte-identically to its meta-llama safetensors twin; before the fix it misread digits and garbled dates while remaining fluent.
  • GGUF pre-tokenizer repair (src/tokenizer.rs): candle maps unknown tokenizer.ggml.pre kinds onto ByteLevel::default(), whose add_prefix_space = true injects a spurious space at the start of every segment after a special token (<|start_of_role|>user encoded the role as user). refact, starcoder, and gpt-2 now get ByteLevel without the prefix space; llama-bpe (what llama.cpp actually writes for Llama 3, which candle does not match) gets the Llama 3 split regex with digit runs capped at three. Encoding verified identical to the reference tokenizer.json and to llama-tokenize.
  • Multi-variant GGUF directories (src/models/loader.rs::find_gguf_files): previously only the first .gguf in a directory was returned, so a Q4_K_M + F16 pair silently resolved every request to the first file (or fell through to a sibling safetensors model). All variants are now listed and resolved by stem; ambiguous requests fail with the available variants named.
  • Deterministic split-K GEMV (src/common/quant_kernels.metal, src/common/metal_ops.rs): every k-split of an output column now lands in one threadgroup and reduces in threadgroup memory in a fixed order with a single writer per output. Geometry is selected per shape class (AWQ M=1 adapts to packed_out; batch kernels reduce one round per activation row). The retired *_atomic kernels remain solely for the paired micro-benchmark (quant_gemv_bench), which interleaves det/atomic samples so GPU timing drift cancels per pair. Measured cost: GPTQ at parity (74.6 to 74.8 tok/s), AWQ minus 3.7% (36.8 to 35.5). A 48-run bitwise determinism test fails on the atomic epilogue and passes on the deterministic one; E2E reruns are hash-identical (10/10 GPTQ, 8/8 AWQ).
  • Tool calls (src/server/routes/chat.rs::parse_tool_call_tag_blocks): <tool_call> blocks carrying JSON payloads now parse into proper tool_calls, covering both the canonical single {"name", "arguments"} object the Qwen2.5/Qwen3 templates instruct and the {"tool_calls": [...]} shape models emit when echoing the injected system-prompt instruction. Parallel blocks, unknown-function filtering, and reasoning text before the block are handled; verified E2E on Qwen2.5-1.5B emission and streaming.
  • KV budget diagnostics (src/models/manager.rs): without --memory-budget the budget derives from the memory free at startup, which under system pressure can start near zero and fail every load with an unexplained "KV cache budget exhausted"; the failure now names the derived budget and the remedy.

Dependencies

  • candle-core 0.10.2 to 0.11.0, tokenizers 0.22.2 to 0.23.1. The four candle 0.11 Metal issues and their mitigations (src/common/weights.rs): (1) MTLResidencySet is mutated without a lock, so concurrent tensor creation from rayon loader threads leaves buffers non-resident and the GPU reads zeros: all device-touching loader calls now serialize through metal_alloc_lock. (2) F8E4M3 ships with no Metal cast kernels ("not implemented" on contiguous, garbage on strided): FP8 checkpoints dequantize entirely on CPU at load, and a dedicated cast_f8e4m3_f32 kernel (bitwise-verified against the CPU cast over all 256 encodings) covers runtime widening. (3) Host-to-device copies queue as GPU commands and staging buffers are reclaimed only at sync points, so loading a multi-GB checkpoint without syncing exceeds the Metal working-set limit and failed command buffers silently zero the weights: ModelWeights::load is now two-phase, with parallel CPU-only reads/casts/scale-folds followed by sequential device transfers drained every 16 tensors. (4) Synchronizing from one thread while another encodes corrupts state: the loader never does. The CANDLE_METAL_COMMAND_POOL_SIZE=1 workaround from alpha.12 is removed; candle 0.11's per-encoder hazard tracking replaces it (FP8 output re-verified byte-identical). The GGUF tokenizer builder is vendored into src/tokenizer.rs (candle 0.11 pins tokenizers 0.22, this project ships 0.23), folding the pre-tokenizer repairs into its table.
  • Security and unsoundness bumps: crossbeam-epoch 0.9.20 (RUSTSEC-2026-0204, also failing CI's cargo audit), quinn-proto (RUSTSEC-2026-0185), and anyhow/memmap2/rand advisories.

Documentation

  • Rustdoc pass across modules; comment cleanup pass (stale historical notes removed, the repeated Metal staging-buffer rationale consolidated into the two-phase note in ModelWeights::load).

Tests

  • 312 unit tests green (294 in alpha.12.2). New coverage: Granite arch regression (including logits_scaling division semantics), de-interleave inversion against the converter's permute, GGUF pre-tokenizer no-prefix-space matrix, K-quant GGUF mul_mm prefill parity and odd-N M=1 GEMV parity (previously untested kernels), F8E4M3 spec-decode and Metal/CPU cast equivalence, 48-run GEMV bitwise determinism, and JSON-in-tags tool-call parsing.
  • A/B regression campaigns against baseline binaries at each step: 33 local models, two prompts each, after the Granite work (57/66 byte-identical, 8 expected differences: Granite newly supported, Llama GGUF previously corrupted); light sweep after the candle upgrade (46/48 byte-identical, 2 = a stable borderline-token flip on Llama-3.2-1B GGUF, output correct); heavy sweep one model at a time under memory barriers (OLMoE, Phi-3.5, Ministral, Qwen2.5-3B, Qwen3.5-4B, gpt-oss-20b: all byte-identical except one benign paraphrase flip on OLMoE, verified with an extended battery). FP8 probes byte-identical with 10/10 run stability.

Installation

macOS (Apple Silicon)

curl -fsSL https://raw.githubusercontent.com/giovannifil-64/oxydllm/main/install.sh | sh

Linux x86_64 (CUDA — Ada / Hopper / Blackwell / Blackwell Ultra / Blackwell Desktop)

curl -fsSL https://raw.githubusercontent.com/giovannifil-64/oxydllm/main/install.sh | sh

Linux arm64 (CUDA — GH200 / GB300 / Jetson Thor / DGX Spark)

curl -fsSL https://raw.githubusercontent.com/giovannifil-64/oxydllm/main/install.sh | sh

Installer auto-selects the right binary from detected GPU compute capability.
Override with OXYDLLM_CUDA_TARGET=<ada|hopper|blackwell|blackwell-ultra|blackwell-desktop> (x86_64)
or OXYDLLM_CUDA_TARGET=<hopper|blackwell|blackwell-ultra|thor|blackwell-desktop> (arm64).

Linux (CPU only)

curl -fsSL https://raw.githubusercontent.com/giovannifil-64/oxydllm/main/install.sh | OXYDLLM_NO_GPU=1 sh

Docker (CUDA)

docker run --gpus all -p 11313:11313 \
  -v ~/.oxydllm/models:/root/.oxydllm/models \
  ghcr.io/giovannifil-64/oxydllm:latest start

x86_64 explicit CUDA tags: :cuda-ada, :cuda-hopper, :cuda-blackwell, :cuda-blackwell-ultra, :cuda-blackwell-desktop.
ARM64 explicit CUDA tags: :cuda-hopper-arm64, :cuda-blackwell-arm64, :cuda-blackwell-ultra-arm64, :cuda-thor-arm64, :cuda-blackwell-desktop-arm64.

Docker (CPU)

docker run -p 11313:11313 \
  -v ~/.oxydllm/models:/root/.oxydllm/models \
  ghcr.io/giovannifil-64/oxydllm:cpu start

SHA-256 checksums

See the .sha256 files attached to this release.

Full Changelog: nightly...0.0.0-alpha.13

Assets 30
Loading