RulzTrainer Troubleshooting

Troubleshooting

Out of GPU memory

In this order: use int8_convrot (or nvfp4 on RTX 50-series) for the diffusion model; keep gradient checkpointing on and batch size 1; lower the training resolution; set Block swap to stream part of the model from system RAM; on two GPUs use model split. The trainer already retries a step that runs out of memory with fewer blocks kept uncheckpointed. See Models and memory.

A run stopped with an error

The run saves its LoRA and a training state from its last finished step first, and the log names the folder to resume from: put it in Resume from training state folder. A race saves the configurations that were training, without a state, so it cannot be resumed. A hard kill or a power cut skips this save, so resume from the last regular state.

A failed sample, probe or self-training round does not stop the run: the log shows why, and training goes on without it.

"Step left out: its gradients are not finite"

That step held broken numbers, so it was skipped and the LoRA is unchanged. An odd one is harmless. 25 in a row stop the run, because it is diverging: try a lower learning rate or another quantized backward. An undecided race goes on, because its probes drop that configuration; a race's winner training on alone stops like any run.

Samples look like noise or ignore the prompt

A MOSS-Image run from before v1.2.0 may have read prompts with the wrong tokenizer. Update, and the affected text caches are re-encoded automatically.
With a turbo LoRA the sampler must use its own settings (about 8 steps, guidance 1); a plain model needs its full steps and guidance. The log warns when the two are mismatched.
Previews on another model, or MOSS-Image without its turbo LoRA after training Krea 2 first: before 2.27.0 the hidden "Preview with Krea 2 Turbo" sampling model stayed set after a model switch. Update; Start now names any other model's file still set (clear it, or set it to auto). A turbo or sampling LoRA that fits none of the model's layers now stops the run with an error.
Check that the sample VAE matches the model (MOSS-Image uses the Flux2 "small" VAE).
With ai-toolkit's Automagic 4 the learning rate climbs 1.6% a step on a noisy loss (doubling every 44 to 100 steps) and the LoRA turns to noise after a few hundred steps (ai-toolkit issues #1084 and #1091). RulzTrainer's automagic4 (v2.15.0) has a fixed controller; if you train MiniMax H3 or FLUX Klein in ai-toolkit, copy rulztrainer/optimizers/automagic4.py over its toolkit/optimizers/automagic4.py (see the README). Capping ai-toolkit's max_lr at the start rate stops the noise but under-trains (issue #1084).
Look at the Monitor's Gradient norm chart (since 2.26.0). A norm that climbs steadily, sits above the dashed clip line on most steps, or shows "not finite" means the learning rate is too high for this run; lower it or keep Max grad norm on.

"No face" in the probes or the dataset scan

The detector can miss a face that fills the whole frame; the trainer retries it on a larger canvas. If the face is turned away or cropped, add front-facing references. A body score needs shoulders and hips in the frame.

SAM 3 does not load

facebook/sam3 is gated on Hugging Face. Accept its license on the model page and log in with hf auth login (or put HF_TOKEN in the environment). Without it body proportions use MediaPipe alone. The default appearance body score does not use SAM 3.

Crash in the body scan on some images

MediaPipe 1.0.0 and 1.0.1 abort the whole program when reading a person mask of some image widths. RulzTrainer works around it on those versions; upgrading MediaPipe to 1.1 or newer removes the need.