0.2.18

Highlights

• Emulated number formats on every backend. CompileOptions::number_format runs a graph as f64, double-single f64x2, or any binary float format from f16 down to 4-bit minifloats (bf16, f8e4m3, f6e2m3, f4e2m1, …, optionally with a per-tensor or per-block power-of-two scale), emulated on top of f32. The new Op::RoundToFormat is native on CPU, Metal, MLX, CUDA, ROCm, wgpu and Vulkan (Core ML runs it on the host), and every backend is tested bit-identical to the CPU kernel (rlx-runtime/tests/number_format.rs).
• Quantized LLM decode within 2–7 % of llama.cpp on CPU, without a BLAS. New default-off rlx-cpu features int8-gemv (AVX2 int8 GEMVs for Q4_K, Q5_K, Q6_K, Q8_0, Q4_0 and Q4_1, after llama.cpp's kernels) and portable-gemm, plus a resident KV cache on the CPU backend and a faster one-query decode attention. Qwen3-0.6B on three AVX2 cores: Q4_K_M 11.0 → 80.5 tok/s (llama.cpp 82.5), Q4_0 82.5 (88), Q5_K_M 73.5 (75), Q6_K 70.6 (73), Q8_0 61.6 (63). This is the build behind the new rlx-ps5 target bring-up (not published).
• Load checkpoints straight into a flow. rlx-flow's new formats feature reads safetensors (memory-mapped, sharded) and GGUF (dequantized or packed for DequantMatMul) as WeightSources, and WeightSourceExt adds prefixing, renaming, tied-weight aliases, fallbacks and a report of missing / unused keys. A missing key's error names the closest ones.
• MLX's CUDA backend builds and runs on Linux. The build script picks a CUDA toolkit by itself when several are installed, skipping versions MLX refuses (13.1), handles Ubuntu's split nvidia-cuda-toolkit layout, finds cuDNN and NCCL, and no longer needs CUDA_HOME at run time. MLX's C++/CUDA sources now build through sccache (else ccache) on Linux too, so a fresh target dir no longer recompiles the CUDA kernels from scratch. rlx-mlx 176/176 on an RTX 3080 Ti; selection, compile and link checked against NVIDIA's CUDA 12.8, 13.0, 13.1 and 13.2 images (rlx-mlx-sys/docker/cuda_matrix.py).
• Ternary (BitNet b1.58) weights, from training to FPGA bitstream. Quantization-aware training on any backend (rlx_compile::inject_bitnet_qat; in the bundled example, ternary weights alone drop test accuracy from 81.8 % to 55.9 % and QAT brings it back to 82.2 %), a multiplier-free ternary FPGA export, and Xilinx 7-series bitstreams through the open openXC7 flow with a UART host link (Arty A7, Basys 3, Nexys A7). The emitted RTL is now simulated against the reference in the test suite (Icarus Verilog / Yosys, when installed), which found and fixed wrong 2- and 4-bit exports, an unclamped fused ReLU and RTL that strict tools rejected.
• MLX v0.32.4, dependencies updated within the 1.89 MSRV, and a new just check-rustdoc gate that builds every crate's docs with its docs.rs features.

---

Added

• Emulated number formats (CompileOptions::number_format). A graph can
  run as f64, double-single f64x2, or any binary float format from f16
  down to 4-bit minifloats (bf16, f8e4m3, f6e2m3, f4e2m1, custom
  fNeXmY), parsed from text such as "f4e2m1@b16". For the narrow formats,
  rlx_compile::emulate_float_format inserts the new Op::RoundToFormat after
  the weights and after every op that produces a float tensor, with no scale,
  one per tensor, or one per block (FormatScaling; MX 32-element blocks by
  default at 8 bits and below), so every value a consumer reads is
  representable in the format while arithmetic inside an op still accumulates
  in f32. FloatFormat::round (ties to even, saturating, power-of-two
  scales) is the one definition: the op is native on CPU, Metal, MLX, CUDA,
  ROCm, wgpu and Vulkan (Core ML runs it on the host), and
  rlx-runtime/tests/number_format.rs checks every backend bit-identical to
  the CPU kernel.

• sccache for the MLX build (rlx-mlx-sys). MLX's C++ and CUDA sources
  now compile through sccache when it is installed, else ccache, on macOS,
  iOS-family targets and Linux; RLX_MLX_CCACHE=sccache|ccache|<path>|off
  chooses. Linux builds used no compiler cache at all before, although the
  README said MLX picked up ccache by itself — so every new OUT_DIR
  recompiled the CUDA kernels from scratch (~1 h).

• Checkpoint files and adapters for flow weights (rlx-flow). With the new
  default-off formats feature, formats::SafetensorsSource (one file or a
  directory of shards, memory-mapped; F16/BF16/F64/integer read as f32) and
  formats::GgufSource (dequantized, or .packed(true) for DequantMatMul)
  are WeightSources, and formats::open(path) picks one by path. Without the
  feature, WeightSourceExt (in the prelude) adds CloningSource (a
  non-consuming source over any TensorLookup), with_prefix / renamed,
  alias (tied weights), map_tensors, or (fallback source) and tracked
  (served / missing / unused keys after a build), all passing packed weights
  through. WeightSource::keys lists a source's tensors, and a missing key's
  error names the closest ones (missing_weight). tests/weight_formats.rs
  builds one projection from a GGUF file as F32 and packed and checks both
  against a reference.

• rlx_gguf::quant_scheme (feature ir) — the GGML type → QuantScheme
  table, now in one place; rlx_cpu::quant_scheme_for_ggml forwards to it.
  Without ir, rlx-gguf still has no rlx dependencies.

• rlx-cpu feature int8-gemv (default off): int8 decode GEMVs (m == 1)
  for Q4_K, Q5_K, Q6_K, Q8_0, Q4_0 and Q4_1 on x86-64 AVX2, after llama.cpp's
  kernels; the activation is quantized to int8 (Q8_K or 32-blocks), so results
  differ from the f32 path by that rounding. RLX_Q4K_EXACT_GEMV keeps Q4_K on
  the exact f32-activation arm.

• rlx-cpu feature portable-gemm (default off): without a BLAS, m ≥ 8
  GEMMs go to the gemm crate's blocked kernels instead of the loop nest.

• Resident KV cache on CPU. rlx-runtime's CPU backend implements the
  resident decode handles (bind_gpu_handle, register_kv_row_feed,
  feed_kv_row, read_output_row); a bound KV handle stays in place in its
  arena slot and only the new row is written per token. Guard:
  rlx-runtime/tests/cpu_resident_kv.rs.

• Opt-in runtime flags: RLX_PACKED_LM_HEAD (the flow's LM head reads the
  packed output.weight, or the packed embedding table when tied, instead of an
  F32 copy) and RLX_CPU_EXEC_IN_POOL (run each CPU schedule on a thread-pool
  worker, so parallel kernels join by spinning instead of parking the caller).

• rlx-ps5 (standalone, not published): jailbroken-PS5 target bring-up —
  just ps5 setup|build|send|run, a probe payload, a distributed-node payload,
  and a platform seam for model payloads.

• Ternary (BitNet b1.58) weights, including quantization-aware training.
  rlx_ir::ternary is the one definition training and export share: absmean
  weights (γ = mean|W|, codes clamp(round(W/γ), −1, 1), per tensor by
  default, TernaryScale::PerChannel optional) and int8 per-token absmax
  activations. rlx_compile::inject_bitnet_qat rewrites every MatMul / Conv
  whose weight traces to a leaf into w + stop_gradient(ternary(w) − w) (and
  the activation likewise) — the BitNet straight-through estimator, built from
  existing primitives, so it trains on any backend; keep_full_precision
  leaves embeddings / LM heads in f32. rlx-runtime/tests/bitnet_qat.rs checks
  the CPU forward loss and dW against the host definition. The
  bitnet_ternary_fpga example shows why QAT matters: ternary PTQ alone drops
  test accuracy from 81.8 % (f32) to 55.9 %, QAT recovers 82.2 %.

• ExportQuantMode::Ternary for FPGA export (quant="ternary" / "bitnet").
  f32 Conv / MatMul weights are ternarized with the same TernaryConfig
  (LegalizeOptions::ternary) and packed 2 bits per weight, and the export
  always takes the multiplier-free add/sub/skip MAC — on every tune preset, since
  it is exact. Walkthrough: docs/bitnet-fpga.md.

• openXC7 boards: Xilinx 7-series bitstreams and a UART host link.
  HwTarget::xc7_board / --hw xc7:<board> (Arty A7-35/100, Basys 3,
  Nexys A7-100) emits a board_top with power-on reset and a 115200-baud host
  link (0xA5 + input bytes in, pred and optionally every output byte back),
  the board's real pin constraints, a synth.sh that runs the full open flow —
  synth_xilinx → nextpnr-xilinx → fasm2frames → xc7frames2bit → out.bit,
  with the part's chipdb built once and cached — plus program.sh
  (openFPGALoader) and run_uart.py. The script finds the toolchain in the
  snap, a source build, or the regymm/openxc7 container. Built in that
  container for Arty A7-35: the example MLP routes at 164.9 MHz and a padded
  ternary CNN at 110.4 MHz against the 100 MHz board clock
  (tests/openxc7_docker.rs, --ignored). Not yet run on a board.

• Zero padding in FPGA conv layers. Both conv kernels (scalar and
  output-channel-parallel) accept any pad_h / pad_w and stride; out-of-bounds
  taps skip the accumulate, as the reference does. Previously any padding
  panicked in codegen, so most real CNNs could not export.

• The emitted RTL is now simulated, not just inspected.
  rlx-fpga/tests/sim_parity.rs runs the SystemVerilog under Icarus Verilog and
  compares every logit and the argmax with rlx_fpga::reference: dense and
  padded/strided conv models × 2/4/8-bit weights × every tune preset, plus
  TinyConv-MNIST. tests/xc7_uart.rs drives the UART link bit by bit, on the RTL
  and on the synth_xilinx netlist against Yosys's Xilinx cell models. Both
  skip when the tools are absent. Every RTL bug under Fixed was found by
  these on their first run.

• just check-rustdoc — a rustdoc gate, wired into just ci. rustdoc was in
  no repo gate, and cargo doc --workspace is the wrong instrument for one: it
  resolves the whole workspace under a single unified (default) feature set, so
  an intra-doc link into a feature-gated item reads as "unresolved" even though
  docs.rs resolves it perfectly. 26 of 28 workspace warnings were phantoms of
  exactly that kind, and the two real ones were buried among them.
  scripts/check-rustdoc.py builds one crate at a time with that crate's own
  [package.metadata.docs.rs] features — what docs.rs actually does — and skips
  publish = false crates, which docs.rs never builds.

Changed

• MLX submodule bumped to v0.32.4 (from v0.32.2): upstream Metal fixes
  that reach rlx-mlx — a fence deadlock, split-K quantized matmul (row count,
  float32 partials), conv bounds and integer overflow, SDPA threadgroup size on
  M1/M2, negative-stride scatter/SDPA-mask indexing. Both tracked patches in
  rlx-mlx-sys/patches/ apply unchanged; the metallib still carries the
  target platform (iOS 82, tvOS 83 at byte 0x0B). rlx-mlx 196/196,
  rlx-runtime with mlx 988 passed.

• Dependencies updated to the latest versions compatible with the 1.89
  MSRV (CARGO_RESOLVER_INCOMPATIBLE_RUST_VERSIONS=fallback cargo update; the
  workspace resolver is 2, which is not MSRV-aware). cargo +1.89.0 check --workspace --all-targets passes.

• Chat templates print booleans as True / False (rlx-text, minijinja
  ≥ 2.24, now required), as Python's Jinja2 — and so transformers — renders
  them; a template that prints a bool used to emit true / false.

• x86-64 one-query decode attention runs a task per head, skips masked keys
  (dot, softmax chunk and V sum), and computes eight keys' dots per AVX register;
  bit-identical to the scalar loop it replaces
  (decode_attention_lane_dots_are_bit_identical,
  rlx-cpu/tests/decode_attention_m1.rs).

• MapWeights::has reports the keys it holds (it returned the trait
  default, false).

• One decode KV-append decision for every model family.
  rlx_flow::blocks::kv_cache holds it: append_kv_row emits the O(1)
  in-place Op::KvAppend when the flow bound its inputs with
  bind_decode_inputs_with_capacity and the cache layout can alias (nothing but
  size-1 dims before the sequence axis), and concat otherwise;
  kv_spare_rows shapes the cache from the same predicate so the declared shape
  and the emitted op cannot disagree. The Llama and Gemma decode layers can now
  take the in-place path too (previously only Qwen3) — once their flow binds
  with capacity; no in-tree flow does yet, so they still concat. The opt-out is
  RLX_NO_INPLACE_KV for all families; the old RLX_QWEN3_NO_INPLACE_KV is no
  longer read. A Qwen3 cache without a spare row now falls back to concat
  instead of returning an error.

• FPGA export API additions that break exhaustive matches / struct
  literals: HwTarget::Xilinx7 gained board: Option<Xc7Board>,
  ExportQuantMode gained Ternary, and LegalizeOptions gained
  ternary: TernaryConfig (construct with ..Default::default()). The
  xilinx7:<part> string form still parses, with no board.

• rlx-js takes quickrs-core from crates.io instead of a sibling checkout.
  The engine is published as of 0.16.2, so building this workspace no longer
  needs a ../quickjs-rs directory beside it. That requirement reached further
  than one crate: a path dep makes the whole workspace unresolvable without the
  sibling, which is why just test-wgpu-linux and the emulated ISA runs both
  bind-mounted it into their containers. Both mounts are gone. To iterate on the
  engine and the binding together, add a [patch.crates-io] entry rather than
  reinstating the path dep.

• just test-js covers the whole JS surface by default. Its default features
  were cpu,gguf, but tests/js_api.rs carried no feature gates, so 26 tests
  failed with undefined is not a constructor for rlx.Trainer,
  rlx.Optimizer and renderChat — the crate's canonical command was red. Each
  test is now gated to the features it exercises, so a narrower set is green
  instead of broken, and the recipe's default is the crate's own default set, so
  the bare command runs all 91 rather than silently skipping 26. The sandbox test
  behind the README's "no I/O a script can reach" claim is the exception: its
  probe list follows the enabled features instead of being gated away wholesale,
  because a claim tested too narrowly is worse than no claim.

Fixed

• MLX's CUDA backend now builds and runs on Linux (rlx-mlx-sys; checked
  on an RTX 3080 Ti, rlx-mlx 176/176 with RLX_MLX_DEVICE=gpu):

  • Asking for CUDA (RLX_MLX_CUDA=1 / feature cuda) without cuDNN or the
    toolkit silently built the CPU-only MLX, whose tests then passed for a GPU
    run that never happened; the build now names what is missing.
  • Every cargo:warning of the build script went to stderr, where cargo
    does not read them, so none was ever shown.
  • nvcc is pinned to the probed toolkit (CMAKE_CUDA_COMPILER); a distro
    /usr/bin/nvcc beside /usr/local/cuda was picked first.
  • Several CUDA toolkits on one machine: the build picks one itself —
    RLX_MLX_CUDA_ROOT, CUDA_HOME, CUDA_PATH, /usr/local/cuda* (newest
    first), /opt/cuda, Ubuntu's split nvidia-cuda-toolkit (via a shim
    root in OUT_DIR) — skipping versions MLX's CMakeLists.txt refuses
    (13.1), and names its choice and what it skipped. Host compiler: the
    toolkit's bin/g++ wrapper, else the newest g++-N within the
    toolkit's GCC limit, unless CUDAHOSTCXX is set. cuDNN from
    CUDNN_ROOT / CUDNN_PATH, the toolkit, or the system dirs. MLX ≥
    0.32.3's cusolver is linked.
  • With NCCL installed (as in NVIDIA's CUDA images), MLX builds its NCCL
    distributed backend and libmlx.a failed to link (undefined reference to ncclAllReduce); the build now links the NCCL that MLX's CMake found.
  • Patch 0003: MLX's NVRTC JIT defaults to the build's toolkit headers and
    the CCCL copy MLX installs, instead of /usr/local/cuda — no
    CUDA_HOME needed at run time.
  • rlx-mlx-sys/docker/cuda_matrix.py checks toolkit selection and the full
    MLX CUDA compile + link on NVIDIA's CUDA 12.8 / 13.0 / 13.1 / 13.2 images
    (no GPU needed).

• Grouped conv weight gradients with stride > 1 were wrong on MLX-CUDA
  (max |Δ| ≈ 1 on conv2d_backward_weight_groups_3_stride2; MLX's CUDA
  backend mishandles a grouped conv with a dilated kernel). With the cuda
  feature, rlx-mlx computes that case per group with ungrouped convs; CPU and
  Metal builds are unchanged.

• Packed weights were dropped through &mut weight sources. impl WeightSource for &mut T forwarded only take and has, so through an
  adapter over a &mut dyn WeightSource take_packed answered None and a
  quantized model silently built as F32.

• No-BLAS sgemm/sgemv/dgemm with beta == 0 leaked NaN from the
  output buffer (0 · NaN). Guard: rlx-cpu/tests/sgemm_general_strided.rs.

• 2- and 4-bit FPGA exports computed wrong results. The weight-byte lane
  select was combinational while the weight ROM reads a cycle late, so each MAC
  took the crumb/nibble for the next index out of the previous byte. INT8
  ignores the lane, which is why nothing caught it; every INT4 and ternary export
  was affected.

• Fused conv+ReLU never clamped in hardware. The clamp compared
  q_raw < OUT_ZP[7:0]; a part-select is unsigned, so negative values passed
  through. The checked-in TinyConv-MNIST demo RTL (examples/mnist_sv/) had it
  and is regenerated.

• Emitted SystemVerilog that strict tools reject. Signals were used before
  their declaration (dense and conv acc / counters), the FP4 decode table wrote
  32'sd-1, block_rom declared parameter string (Yosys has no string
  parameter type, so no Yosys flow could read the RTL), and argmax read
  best_idx[7:0] past its MSB (X in simulation). State transitions written as
  enum ? : are now if/else, which every simulator accepts.

• nextpnr-xilinx could not place exported designs. Yosys mapped small
  buffers to distributed RAM (RAM64X1S), which nextpnr-xilinx cannot pack; the
  openXC7 script synthesizes with -nolutram.

• CPU Round rounded halves away from zero on two of its three paths. The IR
  defines half-to-even (ONNX Round, rlxsl, the third CPU path); x.round()
  is now round_ties_even() everywhere, so Round-based quantizers give the
  same codes on every path.

• A second in-flight WebGPU run panicked instead of erroring.
  WebModel::run_async and ClusterWorker::grads hold the executor borrow
  across an await; calling either again before the first promise settled hit a
  BorrowMutError panic. Both take the borrow with try_borrow_mut, so JS gets a
  catchable "already in flight" error.

• -D warnings failures on wasm and in rlx-web tests. q4k_exact_gemv is
  only called from a cfg(not(wasm32)) path and warned as dead code on the wasm
  build; it is gated the same way now. webgpu_trainable_models.rs source-includes
  the vision module and now allows the dead code it does not call. In
  autodiff_training_backward_gpu.rs the packed-table RoPE fixtures are gated to
  the GPU features of the tests that use them, so cargo clippy --all-targets
  without a GPU feature no longer fails on them.

• The env-var registry had drifted by 15 reads. RLX_BLAS_ALIAS_CHECK, seven
  Metal settings (small-m MPS routing and its trace, the share-driven weight split
  and its byte threshold, the three PTQ1_0 levers), five wgpu ones (the
  cooperative-matrix probe and its verbose mode, the feed trace, the two
  wide-GEMM pins) and RLX_XDNA_AIECC_GET were read by the code but absent from
  env_registry_data.inc.rs — so none of them appeared in
  docs/rlx-env-vars.md or just env-catalog, and a user had no way to learn
  they existed.

  just check-rlx-env-vars returns at its first failing stage, so this failure
  had been masking the next two: the shim bypasses below, and a generated doc one
  entry out of date.

• Two RLX_* reads bypassed the rlx_ir::env shim, so nothing in-process
  could set them. RLX_MODELS_DIR (rlx-js tests) and RLX_SHOW_VALS
  (rlx-metal tests) went through std::env, which RlxEnv and any harness that
  A/Bs two settings inside one process cannot reach. Both read through the shim
  now.

• Two intra-doc links pointed at the wrong item or at no item.
  StepOptions::broadcast_token linked Self::generate, but generate is on
  PipelineCoordinator; and rlx-torch-ckpt's public pickle module docs
  linked the private crate::torch. Both rendered as dead links on docs.rs, and
  both are what the new just check-rustdoc found once the phantom warnings were
  out of the way.

Performance

• Qwen3-0.6B decode on three AVX2 cores, no BLAS (the PS5 build: rlx-cpu
  portable-gemm + int8-gemv and the opt-in flags above), average tok/s
  before → after, with a CPU-only llama.cpp on the same cores for reference:
  Q4_K_M 11.0 → 80.5 (82.5), Q4_0 → 82.5 (88), Q5_K_M → 73.5 (75), Q6_K →
  70.6 (73), Q8_0 → 61.6 (63). The int8 kernels alone run 1.5–4.4× the
  previous ones per core; decode attention ~2×.