Skip to content

v0.1.1

@pjordanandrsn pjordanandrsn tagged this 01 Jul 16:37
Generalize the loader beyond the OLMoE/Qwen3 pattern:
- SUPPORTED_ARCHITECTURES registry maps model_type -> experts submodule path
  (mlp.experts for OLMoE/Qwen3; experts for gemma4/gemma4_text, beside a parallel
  dense MLP + custom router that load as normal non-expert weights).
- Handle experts stored FUSED on disk (experts.gate_up_proj — Gemma-4) in addition
  to per-expert (OLMoE, Qwen3).
- Activation from config.hidden_activation|hidden_act (gelu for Gemma).
- Re-instantiate every rotary module generically (Gemma uses more than one).
ExpertsLoRA drops in unchanged — Gemma4TextExperts.forward has the same
(hidden, top_k_index, top_k_weights) signature.

tests/test_loader_architectures.py now covers OLMoE + Qwen3-MoE + Gemma-4 (4 passed).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Assets 2
Loading