blazr 0.4.0

Serves System One decision models at POST /v1/systemone, with the Kev and Clef families, LoRA variants and a prefix-state cache.

Added

• loader::load_safetensors_with_report — the regular SafeTensors load, plus a TensorLoadReport: the tensors uploaded, the tensors left on disk, and the compute dtype. loader::model_torch_dtype gives the dtype a model's config.json names.

• tests/decision_loader_real_file.rs — a Qwen3.5 decision model (Jebadiah, the Mapika decider) through blazr's loader, then LabelTokenHead against reference fixtures, from SafeTensors or GGUF, on CPU or CUDA. It skips unless BLAZR_DECISION_MODEL_DIR or BLAZR_DECISION_GGUF, and BLAZR_DECISION_FIXTURE_DIR, are set.

• POST /v1/systemone — typed decisions from a System One model (Jebadiah, the Mapika decider), each answer in the family's own wire shape. The family comes from --decision-family, a sidecar beside the weights, or the GGUF header. Each model gets one decision worker thread: requests with a short shared prefix merge into one run, longer ones run once over their prefix. Limits answer 413 and a full queue 503. Keys: inference.decision.* and the --decision-* flags. GET /v1/models adds a decision object read without loading weights.

• blazr_decision_* metrics — requests, rows, prefill tokens, latency and queued rows, by model.

• blazr decision calibrate — fits one temperature per answer type (choice, noul, score) of a decision model on a JSON-lines label file, with boostr's fitters. A line is a labelled request (state, questions, labels), rendered by the model's renderer and run at temperature 1 on its decision worker, or a precomputed decider/calibrate.py record. A records-only file loads no weights. A type with fewer than --min-rows answers keeps the vendor temperature. The fit goes to blazr_calibration.json in the sidecar directory, or --out. --dry-run writes nothing. blazr serve and /v1/models take each type that file sets over the vendor's temperature_by_type or temperatures.json. calibration.source is then blazr_calibration.json, and /v1/models adds fitted_at and per-type sources. An invalid file stops the model from loading.

• boostr::inference::decision::fit_temperature_mixed — one temperature over listed and isolated Score answers together, the Score fit of blazr decision calibrate.

• LoRA variants — a PEFT adapter applies to a loaded qwen35 model as the variant <model>:<name>, built with boostr's LoadedModel::derive_with_adapter: shared base weights, the adapter unmerged, its own executor and KV cache. serve --lora <dir>[=<name>] (repeatable) and inference.lora register variants of the start model; POST /v1/lora registers one at run time. A request selects a variant by model: "<model>:<name>" or lora_adapter; an unknown adapter is a 404. run and generate take one --lora. /v1/models and GET /v1/lora list each variant's base, path, rank and adapted module count. Eviction drops a variant before its base; a registered variant builds again on its next request. /v1/systemone serves variants. Data-parallel start models refuse adapters.

• tests/lora_variant_real_file.rs — a Qwen3.5 base and a PEFT adapter through the scheduler's variant path, checked against transformers + peft logits. It skips unless BLAZR_LORA_BASE_DIR, BLAZR_LORA_ADAPTER_DIR, BLAZR_LORA_REF and BLAZR_LORA_BASE_REF are set.

• Kev on POST /v1/systemone — a Kev directory (a LoRA adapter and a pointer head in head.pt, no base weights) in the model directory serves by its name. blazr finds the base with --decision-base (inference.decision.base), else by the adapter's base_model_name_or_path in the model directory or a Hugging Face hub cache, and never downloads it. The base loads once under its own name. The Kev model is the LoRA variant of it, and boostr's PointerHead reads the answers at the fitted temperature head.pt holds. Prompts and answers follow Kev's own code: every question on its own row, at most 255 options, states over 65,536 tokens refused (--decision-truncate-states cuts them instead), 4-digit answers, and Kev's usage. /v1/models reports family: "kev", readout: "pointer", the base, the adapter rank and the temperature.

• tests/decision_kev_fixtures.rs and tests/decision_kev_route.rs — Kev's rows, positions and answers against fixtures from Kev's own code, without weights; then the route end to end on the CPU in F32. They skip unless BLAZR_DECISION_MODEL_DIR and BLAZR_DECISION_FIXTURE_DIR are set, and the route test also BLAZR_DECISION_BASE.

• Clef on POST /v1/systemone — a Clef or Clef-Flash directory serves by its name: a full fine-tuned Qwen3.5 checkpoint with joint_head_config.json and joint_head.safetensors beside it, or any checkpoint named Clef with --decision-family clef. The backbone loads through blazr's own loader; boostr's JointSchemaHead reads every hidden row of the record and scores all questions together, in F32, with no temperature. Prompts, option order, limits, error words and answers follow the release's joint_schema_model.py: one record of at most 16,384 tokens with the state cut to fit, choice probabilities in the order the request wrote the keys, confidence the chosen option's probability, and a one-option field accepted. Control-token text in a request stays text, unlike the release. Requests with images or video are refused. /v1/models reports family: "clef", readout: "joint_schema" and the head's width and dtype. blazr decision calibrate refuses Clef. Clef-Flash's text backbone is 16.7 GiB in BF16: it serves on the CPU with that much free RAM, or on one GPU that holds it.

• tests/decision_clef_fixtures.rs and tests/decision_clef_route.rs — Clef's records and spans against the release's fixtures, and the head through blazr's readout over the fixtures' saved hidden rows, without the backbone; then the route end to end on the CPU. They skip unless BLAZR_DECISION_MODEL_DIR and BLAZR_DECISION_FIXTURE_DIR are set, and the route test also BLAZR_CLEF_FULL=1.

• Decision prefix-state cache — each decision model's worker keeps the model's state after a request's state prefix on the device. A later /v1/systemone request about the same state reads it back and runs only its questions. The key is the loaded weights (each LoRA or Kev variant has its own), the family and the prefix ids. With the cache on, a request's prefix ends at the family's state boundary, for one question too. Kept states count against lane_budget_mb: before a run the worker drops the least recently used states so that they, the run's state and its largest pass fit. A model that unloads drops its states. Clef keeps none, since its head reads the whole record. Size: inference.decision.prefix_cache_mb and --decision-prefix-cache-mb, default 2048 MiB, 0 off. Responses fill usage.cached_tokens and the new usage.cached_input_tokens with the prefix tokens a hit saved. Kev's usage is unchanged. /v1/models adds decision.prefix_cache. New metrics: blazr_decision_prefix_cache_{hits,misses,evictions}_total and blazr_decision_prefix_cache_bytes.

• tests/decision_prefix_cache_real_file.rs — the route with the cache on: each fixture twice, then the same state with other questions twice. The first send matches the reference, and each hit matches the first send to one rounding unit. It skips unless BLAZR_DECISION_MODEL_DIR and BLAZR_DECISION_FIXTURE_DIR are set.

• docs/decision-models.md — the System One guide: the families, how to run each, calibration, the prefix-state cache, limits and the tested matrix. The README, API reference and configuration reference link to it.

Changed

• engine::decision::prepare takes a PrefixRule (min_tokens, at_state) in place of the shared-prefix minimum. Pending::wait_answered returns the answers with the prompt tokens the prefix-state cache saved.
• RenderedDecision gains state_end, the row position where the state ends, and RenderedDecision::new.
• Executor::get_embeddings is a plain function. /v1/embeddings and /rerank run it on a blocking thread, never inside an async task.
• POST /v1/lora reads adapters through boostr's PeftAdapter. A load now fails when adapter_config.json is missing or lacks r or lora_alpha. It also fails on an unpaired lora_A/lora_B, a rank mismatch, .bin weights, DoRA, fan_in_fan_out, lora_bias and modules_to_save. use_rslora, rank_pattern and alpha_pattern now set each module's scaling.
• Model lookup errors — an unknown model or LoRA adapter returns 404 model_not_found. A load that fails returns 500 model_load_failed. A load the device cannot hold returns 503 model_load_out_of_memory. Before, a failed load returned 404.
• Model loads — base-model and LoRA-variant loads run on a blocking thread. A load gate makes concurrent requests for one model wait for a single load.
• Load path — the server, run, chat and bench share one set of load steps. run, chat and bench report a failed CUDA client as Failed to create CUDA client for preload: .... The text was Failed to create CUDA client: ....
• Chat template file — a model directory's chat_template.jinja is read before tokenizer_config.json. bos_token and eos_token come from the model's config. tojson honours separators, and floats render as Python's repr.
• Dependencies — boostr 0.9, voxr 0.7, and voxr-narrate 0.7.
• Tests — .config/nextest.toml runs arch_reference and batch_decode_per_row in a heavy-model group of two threads.

Removed

• engine::load_lora_adapter, engine::LoraAdapter, engine::LoraAdapterRegistry, and Executor::{load_lora, unload_lora, list_loras, lora_registry}. Scheduler::{register_variant, remove_variant, variants, select_model} replace them.

Fixed

• Command stack on Windows — the binary runs its command on a thread with a 16 MiB stack. The command's frames overflowed the 1 MiB main-thread stack of a Windows process, and sound-review and speak crashed at start. Linux and macOS have 8 MiB and did not crash.
• Sharded SafeTensors directories — a directory with model.safetensors.index.json loads only the shards the index names (boostr's SafeTensorsLoader). Every .safetensors file of the directory loaded before, so a head or adapter file beside the shards was read as part of the model.
• lora_adapter on generation requests — it now runs the request on the adapter's variant. It was logged and ignored before: the request generated with the base weights, and an unknown name fell back to the base.
• Qwen3.5 SafeTensors configs — a qwen3_5 (nested text_config) or qwen3_5_text config.json goes to boostr's qwen35_config_from_hf before any other parse. It fell through to tensor-name detection or a partial parse before.
• Config dtype — the compute dtype reads torch_dtype, then dtype, then both under text_config. Only the top-level torch_dtype was read before, so transformers v5 and Qwen3_5ForConditionalGeneration configs fell back to the backend default.
• Qwen3.5 MTP and vision tensors — they stay on disk in every SafeTensors path. They were uploaded to the device and then dropped before.
• Qwen3.5 edit precision — 1 + w and -exp(A_log) round once, to the compute dtype. Each 1-D tensor waits in F32 when the compute dtype cannot hold its file value. A BF16 model rounded these edits twice before, and the F32 A_log was rounded to BF16 first.
• Sharded checkpoints named by an index — a directory with model.safetensors.index.json is SafeTensors whatever its shard names, such as model.safetensors-00001-of-00002.safetensors. It failed with No supported model files found before.
• Offloading load — load_safetensors_with_offloading calls restructure_hf_checkpoint, as the other SafeTensors paths do. It takes the layer count from the config and the layer prefix from the tensor names. Non-Llama layouts failed to load through it before.
• Qwen3.5 GGUF configs — the GGUF config builder passes the open GGUF file to boostr's qwen35_config_from_gguf. It passed a bare metadata map before. Qwen3.5 GGUF models, decision models included, load through this path.

Archives

Archive	Platform
blazr-linux-x64.tar.gz	Linux x86_64, CPU
blazr-linux-x64-gpu.tar.gz	Linux x86_64, CPU and Vulkan
blazr-linux-arm64.tar.gz	Linux aarch64, CPU
blazr-macos-arm64.tar.gz	macOS on Apple silicon, Metal
blazr-windows-x64.zip	Windows x86_64, CPU
blazr-windows-x64-gpu.zip	Windows x86_64, CPU and DX12
blazr-android-arm64.tar.gz	Android arm64, static library
blazr-ios-arm64.tar.gz	iOS arm64, static library

Full diff: v0.3.0...v0.4.0