Skip to content

v1.7.0 — Operator-dependent quantization (MoE routing + SSM decay)

Choose a tag to compare

@ahb-sjsu ahb-sjsu released this 16 Jul 14:13
db30064

Operator-dependent quantization for hybrid / state-space architectures: the (A2) sensitivity analysis for the two regimes beyond attention — MoE routing and SSM state decay (operator_sensitivity).

  • Routing (gates): selection is carried by the margin/relative order of the gate logits, not absolute magnitude — common-mode shifts are free, but per-expert scale/offset error moves margins and flips the expert choice. Measured: at 4-bit gate quantization, low-margin tokens flip ~88× more than high-margin ones.
  • Recurrences (SSM decay): the slow (long-memory) channels are fragile and error compounds over the sequence; quantize_decay(basis="log_tau") cuts state drift 5–6× vs linear at matched bits.

Every boundary is measured on synthetic operators before it is acted on; real-model MoE/SSM closure is the next-cycle validation task.

Full details: CHANGELOG.md.