RulzTrainer Character LoRAs

Character LoRAs

Training a person (or a character) is where RulzTrainer does the most. It can measure how much the samples look like your references, and use that number to keep the best checkpoint, steer the learning rate and compare setups.

0. How much of the person? (Wizard)

On the Wizard's first page (Start), the focus cards ask what the likeness should cover. Each choice sets the loss weights (the face region and the whole-person region), the share of the face and body scores in the overall score, the crops, the probes and the references:

Focus	Loss face / person	Score face / body	Crops	Probes
Face only	3 / 1	100% / 0	face	face only, no body prompt or references
Face focus (the recipe)	3 / 1.5	70% / 30%	face + bust	both
Face and body	1.5 / 1.5	50% / 50%	face + bust	both
Body focus	1.5 / 3	30% / 70%	face + bust	both
Body only	1 / 3	0 / 100%	none	body only, no face prompt or references

The face/body curriculum is on for the three in the middle. Switching to Face only or Body only empties the references and prompt it does not use, and switching back restores yours. A loaded config opens on the focus its weights describe. Face only is also the quickest: each probe renders one picture instead of two.

1. Pick references

On the Dataset tab tick Face ref on pictures with a clear face, and Body ref on pictures that show the body from the neck to mid-thigh or lower (three-quarter and full-length shots; vary the outfits). Folders work too (face_refs / body_refs).

2. Two scores

• Face: the largest face in the image, embedded with InsightFace (ArcFace), compared with every reference face; the best match counts. A same-person match is typically 50% or more.
• Body (default appearance): the body from the neck to mid-thigh, turned upright and cut out from the background (no head), embedded with DINOv2 and compared with the references that show the same part of the body. Hair, build, skin and clothing count. A portrait with no hips in the frame gets no body score. The proportions option compares skeleton ratios from MediaPipe, optionally with SAM 3 silhouettes (SAM 3 is gated on Hugging Face: accept its license and log in, or it falls back to MediaPipe).
• Body, pose-independent (skeleton3d, smplx): build rather than look, so clothing, skin and nudity don't count. proportions 3D takes bone-length ratios from MediaPipe's 3D landmarks (a bent leg is still the same leg) and silhouette widths only where the person stands upright facing the camera. SMPL-X body shape fits the SMPL-X body model with NLF and measures it in its T-pose: bust, waist, hip and thigh girths and limb lengths over the height, and waist / hip. It needs pictures down to the knees and NLF's model, downloaded by hand because of its noncommercial license: put nlf_l_multi_0.3.2.patch6.torchscript (550 MB, from the v0.3.2 release) in models/nlf/. Both compare against the references' own spread: a measurement that varies between your photos counts for little. Tested on one woman's photos and 32 runs: the 2D proportions scored a stranger above her own photos, the 3D skeleton put a stranger at her level (adults' bone ratios differ less than the estimates' noise), and SMPL-X separated them (a stranger 32, her photos 68) and rose about 16 points with training, mostly from the waist, bust, hip and thigh girths. For build, use SMPL-X with a few full-length photos among the body references.
• Body, anchored (anchored, since v2.35.0): the appearance body window with its photographic look taken out (each colour channel evened out, then shrunk and enlarged so grain and sharpness drop out), embedded with SigLIP 2 and DINOv2. Instead of matching your photos, it measures how far the picture has moved from the base model's people toward them: 0 = the base model (the step-0 body probe without your LoRA, rendered with 8 seeds for this; the probe runs even with probe at start off) and 100 = your body references' average, on a straight scale. Tested on five runs of one woman: every other body score rated her own photos at or below the base model's renders, because her photos differ more in pose, outfit and light than the base model's woman differs from her. anchored told the trained samples from the base model's in 96% of pairs (appearance 80%, the proportion scores 34-59%). A probe in one fixed outfit and scene stays well below 100 (her own photo in the probe scene scored about 40), so compare runs and checkpoints with each other. It downloads about 1.9 GB of models once and uses about 0.4 GB of GPU memory. Tested on one character so far, so appearance stays the default.

Raw, the two live on different scales, so the likeness scale is normalized: 0 = what a stranger scores (the step-0 probe of the base model without your LoRA) and 100 = what a typical reference scores against the others, capped at 100. Everything the controller does, and everything the Monitor shows, uses that scale.

The face scale is stretched. A face score saturates: a decent likeness already matches your references, so on the plain scale the face hit 100 early while the pictures still improved, and read far above the body for pictures that look equally close. The face's 100 therefore sits 1.35 times as far above a stranger as your references do (face_scale_span), and the face score is raised to the power 1.5 (face_scale_curve): a face that used to read 90 now reads about 54, your references' own face level reads about 64, and the score keeps climbing past it.

So is the body scale (since v2.2.0; body_scale_span 1.35, body_scale_curve 1.5). With only the face stretched the body hit 100 while the face read 50-60: a body score saturates too, because a probe that copies your photos' outfit and setting matches them better than your references, in their different outfits, match each other. Now your references' own level reads about 64 on both, so a body at 100 and a face at 55 no longer happen for pictures that look equally close. Set both numbers of a kind to 1 for its old scale. Runs saved before these changes used the plain scales. The anchored body score is on the likeness scale already and is used as it is: the body span and curve don't apply.

3. Likeness probes

Set Probe every N steps (it is off by default). Each probe renders a fixed-seed headshot and full-body image with the turbo LoRA (cheap), scores them, smooths the scores and records them. The last step is always probed too, so the final LoRA has its own score. The controller can, each on its own switch:

• keep <job>_best.safetensors at the best smoothed score;
• keep <job>_best_face.safetensors and/or <job>_best_body.safetensors at the best smoothed face or body score (two more switches, off by default, independent of the overall best);
• New best on any rise (off by default): normally a best file is replaced only when the smoothed score beats it by more than the noise band, so the log can show a score slightly above "best" without a new save. Switched on, any higher smoothed score replaces it. Early stop still counts only a rise beyond the band as progress;
• stop after N probes without improvement (off by default);
• nudge the learning rate up or down with the trend (with any optimizer, Automagic included);
• when the face lags the body (or the other way round), sample close-ups more often and shift the noise levels.

The scores never enter the loss. Use the same references and prompts for a whole run so the numbers compare.

Gold face scorer. The probes judge the face with ArcFace. The reward stage (3b), the identity loss on its arcface backend and identity alignment train against that same ArcFace, so they can raise its score without the face looking more like the person. The Gold face scorer (face_gold, auto by default) then adds FaceNet, a second recognizer that nothing trains on (about 110 MB, downloaded once; about 0.2 s per probe face). It scores the same faces against your face references on its own scale, and with Best checkpoint, early stop and race follow it (on) its score takes ArcFace's place in those decisions; the lr control and the curriculum keep ArcFace. The Monitor draws it as a dashed face (FaceNet) line next to ArcFace's face line: ArcFace pulling away from it is the warning sign. auto also turns it on for the probe noise check (4c); FaceNet, always turns it on for any run.

Export spreadsheet (Monitor, Likeness probes & controller, during or after a run) downloads the run's probe scores as an Excel file: every probe's smoothed and per-probe face, body and combined score, the best one, the raw similarities, and the run's scale on its own sheet. The scores are formulas, so changing a span, curve or weight there rescores the run. In a race each configuration gets its own sheets.

Every run's scores are kept after it ends: overlay any number of runs on the Previous Runs tab (see Previous runs).

3a. Self-training on its best pictures

Self-train on its best pictures (selftrain; Likeness probes section, Likeness-filtered self-training; or the Wizard's experimental extras) runs once halfway through the steps (selftrain_at, 0.5; 0.5, 0.75 for two rounds):

1. the LoRA draws 32 pictures of your person over varied close-up, upper-body and full-body prompts;
2. pictures with several faces, no face, an unsure or cut-off face, or blurrier than your photos are left out, and so are pictures that copy one of your photos (the copy check, below);
3. the rest are scored like the probes; only those above both the floor (60) and the probes' current score are kept, best first, no near-copies, at most a fifth of your photos;
4. they train beside your photos at half weight, with their prompt as the caption, until the end or the next round, which replaces them.

It costs a few minutes (one round of renders). It shows the LoRA its own best pictures of the person in scenes your photos never had, but it can only teach what the LoRA already draws on a good seed, and training on its own output can copy its flaws, so the share stays small and the rounds few. Look at the kept pictures in <job>/selftrain/ (selftrain.json says why each other one was left out) and race it against a run without it.

Copy check (copy_check, auto: on with self-training; Copy check (SSCD) in the same section). A LoRA trained on a few photos can learn to redraw one of them, with the same pose, outfit and background, and self-training would teach that copy again. SSCD, a copy detector (about 99 MB, downloaded once; about 40 ms a picture on the GPU), compares every self-training picture with your photos and leaves out one above the Copy limit (copy_max, 0.5; raised to your photos' own typical match among themselves + 0.1 when they already look alike, as in one shoot in one room) as "copies a photo". For scale: a flipped copy of a photo scores about 0.94, a crop of one about 0.55, an unrelated picture about 0. The probes log their closest photo too ("copy" on the Monitor's probe line); that decides nothing. on turns it on for any run with probes.

3b. Second stage: likeness reward fine-tuning

reward_steps (Likeness probes section, Second stage; or the Wizard's experimental extras) adds a stage after the regular steps that trains on likeness itself, after Flow-GRPO (NeurIPS 2025):

1. the LoRA renders a group of pictures (4) of a new prompt from different seeds, with a stochastic sampler;
2. the probes' scorers rate each one's likeness (face and body, on the normalized scale);
3. the pictures are compared within the group, and the moves that led to the better ones are made more likely, the worse ones less, with a KL penalty that keeps the LoRA near the first stage's.

It aims straight at what you want, over prompts the photos never had, but it is the most expensive stage (every step renders a group of pictures) and it can learn what the scorer likes more than what a person sees. Keep the probes on (they keep the best checkpoint and can stop it) and look at the results. The gold face scorer (FaceNet, see 3) is on for this stage by default: it, not the ArcFace the stage trains on, picks the best checkpoint, and the log warns when ArcFace runs more than 10 points further ahead of it than when the stage began. The first stage's LoRA stays <job>.safetensors; the stage saves <job>_reward_<step>, <job>_reward.safetensors and, when a probe beats the best so far, <job>_reward_best.safetensors. A closing probe scores the stage's final LoRA when its last step wasn't already probed, so the chart always shows where the stage ended. On Krea 2 with a 16 GB card a reward step took about 77 s. With steps 0 and a LoRA in Resume from, it runs alone, for example on a race winner's file.

4. Race setups against each other

Not sure about rank, learning rate or LoRA vs LoKr? Configuration race trains several rows in one run; each trains until its next probe, then the next one does, and after every round the lowest-scoring one is dropped. The winner trains on to the end with the run's own file names. On two GPUs, One configuration per GPU at the same time runs two contenders side by side. On the Monitor, the race table shows each contender's scores and the step they were probed at, and a Drop button takes one out by hand; the likeness chart draws each contender's face, body and combined lines in its own colour, named after it. The gallery puts the same generation of every contender side by side.

The race wizard. On the Wizard's last page answer Yes to experimental settings and turn on Configuration race. The usual pages set up the base racer; below the switch you then set:

1. Racers: how many, which settings may differ between them (anything a race row may change, including self-training and the seed), and a card per racer with a field for each, filled in with the base value. Change only what that racer should do differently.
2. Race rules: drop the lowest after each round (and after how many warm-up rounds, and whether only when the probe seeds agree), and drop a racer when it stops improving (and after how many probes), with the cost in training steps.

The review offers Start the race, or Adjust in all settings to continue on the Train tab, whose Configuration race section has the same rows.

Some settings learn slowly but end higher, and dropping the lowest each round cuts them early. Two switches in the race section change that:

• Drop each one when it stops improving (race_patience_drop, with Probes without improvement, 3): a contender leaves once its own probes have not improved for that many in a row, however it compares with the others, and the rest race on. Keep one whose likeness is rising again (race_patience_rising) gives it a chance after a dip: out of patience, it stays as long as each probe's smoothed likeness beats its previous probe's, and leaves at the first one that does not.

• Drop the lowest after each round (race_drop_lowest) off: nobody is cut for being behind; every contender trains all the steps unless the patience drop ends it (up to one full run each; Steps must be a multiple of the probe interval).

• Drop only when the probe seeds agree (race_drop_confidence, 0.9 is a good start): every racer is probed on the same seeds, so each seed compares two of them like for like. The lowest leaves only when its scores, seed by seed, sit below the round leader's with that confidence; a round too close to call drops nobody and the next one adds its seeds to the evidence. After 3 such rounds in a row (race_max_close_rounds), or once it has trained all the steps, it leaves anyway. The log says how sure each drop was. Use 2 or more probe seeds with it.

With the patience drop or without dropping the lowest, the winner is the contender with the highest score at any point, decided at the end: its best checkpoint becomes <job>_best.safetensors and its settings <job>_winner.yaml. The last one racing still trains to the end as <job>.safetensors; the log says when that is not the winner. Its best checkpoints stay in its own <job>_<name>_best.safetensors (since v2.35.1), so every configuration keeps a complete best file.

A race row may also set its own seed (seed=7): that contender starts from its own LoRA weights and sees its own image order and noise. Rows that differ only in seed are seed twins, the ingredients of a seed soup. Self-training can differ per racer too (selftrain=true, its round at that racer's own steps), but not with one racer per GPU.

The quantized layers' backward is a race key too (nvfp4_backward=fp4, int8_backward=int8): one row with the faster FP4 backward beside the exact bf16 run, plus a bf16 row with its own seed for the seed spread, shows whether FP4 keeps the likeness on your photos (it is about 9% faster per step on Krea 2 nvfp4 on an RTX 50 card).

The optimizer and every Keel setting are race keys too (optimizer=keel, optimizer=keel_fb, keel_band_weights=identity, keel_consensus=0, keel_consensus_mode=magnitude, keel_lookahead=true, keel_face_gain=1, keel_body_weight=1): Keel against Automagic 4 and AdamW 8-bit, then its switches one at a time, then keel against keel_fb, shows which of them help the likeness on your photos; docs/keel-optimizer.md lists those races.

The relay: keep the best seed at every probe

Instead of training whole runs on different seeds and comparing them afterwards, a relay (race_follow) keeps the best seed as it goes. Turn on Relay: everyone continues from the round's winner (or press Add seed racers with seeds such as 1-4: one racer per seed, relay on). Then:

1. Every racer trains until the next probe with its own seed (its own image order and noise) and is probed.
2. The racer with the highest smoothed score wins the round.
3. Every other racer takes over the winner's LoRA: its weights, optimizer state, weight average and likeness history. It keeps its own settings and seed, so the next round tries a few different paths from the best LoRA so far.
4. After the last round, that round's winner is the run's LoRA (<job>.safetensors). The others are kept as <job>_<name>_final.safetensors, and the best checkpoint any of them reached is <job>_best.safetensors.

Nobody is dropped, so every racer trains all the steps: 4 seeds cost 4 runs' steps. Racers can also differ in other settings (learning rate, LoRA+, timesteps...), but not in the LoRA's shape (rank, alpha, layers, adapter type), and the relay does not run with one racer per GPU. Probe scores are a little noisy, so part of each round's pick is luck; more seeds per probe make it steadier. The Monitor's race table shows which racer led the last round.

4b. Seed soup

Two or three trainings of the same person each learn the likeness plus some quirks of their own. The Seed soup section (Train tab) averages what they change in the model into one LoRA, so what they agree on stays and the quirks fade: the "model soup" idea (Wortsman et al., 2022), untested here so far.

• List the LoRA files (LoRA or LoKr of the same model; not OFT or DoRA), one per line: different seeds, or a race's _best files.
• Average now makes the plain average at once (<job>_soup_plain.safetensors).
• Pick the soup with the probes when I press Start: Start trains nothing; it scores each file with the likeness probes, then adds them best-first, keeping each only when the soup scores at least as well, at strengths 1.0, 1.15 and 1.3 (an average of runs that disagree is a little weaker). The result is <job>_soup.safetensors, the scores are in soup/soup.json.
• Rank 32 keeps what the files share; 0 keeps the average exactly (LoRA files only).

4c. How noisy are the probes?

A probe renders a few seeds and averages their scores, so part of every score is the seeds' luck. Before trusting a race or a best checkpoint that won by a few points, measure that luck once per model:

• In the Probe noise check section (Train tab), list two finished LoRAs of one run, a middle checkpoint and the best one, switch on Run the noise check when I press Start, and press Start. Nothing trains.
• It renders the base model, then each file on 16 seeds (the same seeds for both), and writes noise/noise.json (the log has the summary): how far one seed strays from a file's average, how alike the two files score on the same seed, how many probe seeds it takes to tell a 3, 5 or 8 point gap apart, and a noise band that fits your probe seeds (suggested_noise_band, for Noise band in the Likeness probes section).
• Every probe now also keeps each seed's own score in likeness.jsonl; python -m rulztrainer.likeness_stats output/<job> gives a rough version of the same numbers from any run probed with 2 or more seeds.
• The check also scores every face with the gold scorer (FaceNet, see 3) and reports its noise next to ArcFace's, with a noise band for runs where the FaceNet face decides (suggested_noise_band_gold).

5. Make the identity stick

• Region-weighted loss puts more weight on the face, then the person, than the background.
• Automatic face and bust crops add close-ups of every image with a face.
• Weight average (lora_ema) makes checkpoints and probes use a smoothed copy of the LoRA.
• Keel optimizer (optimizer: keel, v2.22.0) is Automagic 4 with identity mechanisms built in: a LoRA gauge preconditioner, a cautious mask, noise bands, a cross-photo consensus gate, a probe-gated lookahead and a face-token gain; see Training options. Not yet measured for likeness: race it.
• Keel (face + body) (optimizer: keel_fb, v2.23.0) is the same optimizer with the body treated as the face's equal: the photos' consensus vote counts each gradient by its size, so a full-body photo outweighs a face close-up on the body's weights, and the token gain adds a body share next to the face's (it needs person masks). With the gain off, the default, it differs from Keel by the vote alone. Unmeasured: race it against keel.
• OFT and DoRA are adapters to race against LoKr 8; see Training options.
• Trigger attention on the subject (attn_localize) reads where the trigger word's attention lands inside the model and pulls it onto the person's silhouette and face box, binding the identity to the trigger rather than to the background or other people. In LoRA Isolation's side-by-side pictures it is pulled onto your subject's panel and off the stranger beside them.
• Identity alignment (id_align) teaches the face's tokens after one block to carry that photo's identity (its ArcFace embedding), through a small helper network that is thrown away. Feedback at every noise level.
• Body alignment (body_align) does the same for the build: the person's tokens outside the face are mapped onto the photo's DINOv2 body-window embedding (what the body probe scores). Only photos showing the neck to mid-thigh count.
• Both are training-only (the LoRA file is the same) and not yet measured for likeness: race them.
• LoKr instead of a LoRA is Fizgig's best measured likeness setting (factor 8). The Wizard recommends a LoRA for every model: in RulzTrainer's tests so far a LoRA reached likeness much sooner, though LoKr showed some positives. A FIBO 2.0 LoKr loads only back into RulzTrainer.
• Look warm-up eases the dataset's look outliers in at a lower loss weight: see Dataset tools.
• LoRA Isolation stops the look leaking onto other people: see Dataset tools.

The README sections Identity similarity, Likeness probes and controller and Configuration race hold every option and the measurements.