Skip to content

v0.8.6

Latest

Choose a tag to compare

@dahai80 dahai80 released this 07 Aug 01:59
e957560

In-place LoRA swap (#389)

Keep a single base engine resident and swap the LoRA adapter onto it in place via mlx_lm's LoRALinear machinery (load_adapters / remove_lora_layers) instead of reloading the full base model per adapter.

  • Correct for 4-bit/8-bit quantized bases (low-rank arrays added beside the quantized linear, never fused into packed weights); allocates no second base copy.
  • Per-base asyncio.Lock serializes the apply → infer → restore window; bare-base requests wait for any in-flight swap to restore.
  • Opt-in via FUSION_LORA_INPLACE_SWAP=1 (default OFF preserves existing per-adapter derived-engine behavior). Requires FUSION_LORA_ALLOWED_DIRS allow-list.
  • Switch latency ~7 ms apply / ~1 ms restore on Qwen3-0.6B-4bit.

See docs/lora-inplace-swap.md.

Full Changelog: v0.8.5...v0.8.6