Skip to content

feat(minimax-h3): portable LoRA keys, pruned and fp8 checkpoint - #41

Merged
imprsnst merged 4 commits into
mainfrom
portable-lora-quantised-model-support
Aug 11, 2026
Merged

feat(minimax-h3): portable LoRA keys, pruned and fp8 checkpoint#41
imprsnst merged 4 commits into
mainfrom
portable-lora-quantised-model-support

Conversation

@imprsnst

Copy link
Copy Markdown
Collaborator

MiniMax H3: portable LoRAs, smaller checkpoints, and a fuse that stops eating host RAM

Three users reported that a LoRA trained here has no effect, both in Inline and in other tools.
This fixes that, plus three further defects found while investigating it.

The actual bug

H3's checkpoint stores attention as one fused blocks.N.attn.qkv_proj. Our loader splits it into
to_q / to_k / to_v while streaming, so the adapter PEFT produces is keyed to names no other
tool has ever seen. Other tools matched nothing and skipped every layer silently, which looks
exactly like a LoRA that loaded and did nothing.

What changed

LoRA interop, both directions (models/minimaxh3/lora_keys.py)

One bidirectional map built on the existing keys.py plan, so it cannot drift from the checkpoint
loader. Training now writes the published key names, and third-party H3 LoRAs load here.

Two things make it exact rather than a rename table:

  • Fusing q, k and v cannot keep rank r, because each has its own A. Stacked A plus a
    block-diagonal B gives an identical delta at rank 3r, and the alpha triples with it so
    alpha / rank is unchanged. Missing that last part divides the adapter by three, silently.
  • The gated FFN's halves are exchanged, because diffusers' SwiGLU reads [value; gate] and the
    reference stores the other order.

The fused QKV row order differs by publisher and cannot be measured from an adapter, which has
no base weights to compare against. Import defaults to the Comfy-Org layout and says so in the log
rather than guessing.

.alpha is now written (training/trainer.py)

PEFT trains with a scale of alpha / rank and saves the factors raw, so an adapter without an alpha
fused at 1.0. Correct only while alpha == rank, which is the default and is why it went unnoticed.
Affects every architecture.

Pruned and fp8 checkpoints load (minimaxh3/adaln.py, keys.py, load.py, requirements.py)

66.3GB down to 21.0GB for the same model. The pruned builds ship no timestep path at all, only
adaln_t_table [1025, 8], so the branch cannot be rebuilt as a basis applied to a silu(temb) we
compute. adaln.tabulate rebuilds the modules around the table instead, interpolating off-grid
timesteps. fp8 is a scalar weight_scale per weight, dequantised while streaming.

int8_convrot is still refused: its weights are stored rotated and that is a transform we cannot
invert. An unrecognised quantisation is refused rather than guessed at.

The fuse no longer materialises the whole delta (models/lora.py)

Other fixes

  • A pre-reduced checkpoint is no longer re-reduced. The pipeline would have multiplied a
    [96768, 8] projection by a full-width basis, the structural sibling of never re-quantising a
    prequantized file.
  • Checkpoints are sized by what they become rather than what they weigh. A pruned file has already
    lost its AdaLN branch and an fp8 file stores half the bytes it will occupy, so scaling the on-disk
    number under-sized both by up to 3x, and under-sizing kills the host instead of raising.
  • The trainer refuses a pruned build by name, since it has no timestep path to derive the basis
    from and saves nothing in VRAM anyway.
  • fp8 offered as an optional download, labelled generation only.
  • The fuse log reports what the adapter did, and warns only when quantisation will discard it.
  • A launcher test only passed on machines with no GPU; the sandbox now shadows the driver.

Verification

Check Result
Real trained LoRA round trip bit-exact, 0.000e+00
Exported names present in the real 66.3GB checkpoint 209 of 209
Reference-keyed LoRA resolved against real module names 313 of 313
Real fp8 file: key plan coverage all 1082 tensors mapped
Real fp8 file: sizing 40.2GB resident from a 21.0GB file
Pruned table vs full bf16 modulation 1.6e-04 relative, against 1.3e-03 for one bf16 ulp
fp8 end-to-end render 124 frames, 23.49GB peak, valid output

@imprsnst
imprsnst merged commit 3af1449 into main Aug 11, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant