Skip to content

v2.7.0 — The run that looks after itself

Choose a tag to compare

@shootthesound shootthesound released this 11 Jul 08:29

The Krea 2 trainer becomes a live dataset curator — four Training-tab toggles no other trainer has:

  • Detect problem images — every image's loss is tracked across epochs (normalized for the random noise level each step draws). Images that stay hard without improving get flagged live in the Problem Images window — thumbnails, verdicts, per-image trends, and mid-run caption editing. In real runs the top flags were all caption/image mismatches, found from the loss trajectory alone.
  • Per-image adaptive LR — stuck images are throttled (×0.7 suspects, confirmed-stuck ×0.5 escalating toward ×0.1) so one bad caption can't yank the weights all run; consistently-healthy learned images get a gentle ×1.1 boost. Matched-epoch A/Bs: faster likeness and a higher ceiling, with real skin texture where untreated runs went plastic.
  • Auto-recaption stuck images — the same Qwen3-VL that conditions training looks at each confirmed-stuck image between epochs and rewrites its caption from what's actually visible. Two strikes → the image is excluded for the rest of the run, and the exclusion is remembered per-dataset (travels with your images). Fix the caption and it's re-admitted automatically.
  • Warm up look outliers — real-but-unusual images (tight angles, profiles) ease in at ×0.4 LR over the first epochs instead of fighting the forming identity, releasing to full LR the moment they start improving.

"When am I done?" — answered from two independent directions

Until now the answer was "train too long and eyeball the checkpoints". v2.7.0 measures it, from both the loss side and the image side — on both model families:

  • The loss watch calls the plateau. When nothing is still improving, you get a banner with a best-checkpoint estimate (from each image's own finish epoch) — and it's honest about certainty, distinguishing a provisional plateau (images still being adjudicated that may give the run a second wind) from a confirmed one.
  • The sample gallery scores likeness live. Pick the 3 dataset photos that best nail the look and every training sample gets a colour-coded likeness badge as it renders (CPU ArcFace averaged across all three baselines — zero impact on training speed). A per-epoch trend chart plots the current run's likeness curve with the best epoch highlighted — you literally watch identity converge, and where the curve flattens is your answer from the image side. When both instruments point at the same neighbourhood, you can trust it.
  • Scrub it to confirm. The new Training Run Visualiser plays the run epoch by epoch in the browser, Royale-style, with the likeness score inline on every frame — ping-pong playback, and share-ready export (WebM clip with epoch ticker + Fizgig tag, or full-res PNG frames). Then take the suggested epoch window into LoRA Royale for the final pick by eye — likeness measures identity, but overbake and skin texture are still yours to judge.
  • Pause or restart, lose nothing. A resumed run replays its own loss log to restore every verdict, trend, and exclusion exactly — the plateau clock never resets.

Curate the dataset before it ever trains

  • Look Consistency Filter (Image Prep, the final prep stage) — pick the 3 images that best nail the look; every image's face is scored against all three, averaged (close-up faces included — detection pads-and-retries). Built for synthetic-heavy datasets: the subtle off-look near-misses a loss curve can never see are exactly what embedding distance sees. Mark drifters by click or let Auto-Suggest flag the statistical outliers, then move them out in one go — nothing deleted, moving them back re-admits them. The scores save with your dataset and drive the trainer's look-outlier warm-up.

Also in this release

  • Per-dataset caches, cross-checked — each dataset gets its own cache folder and every cached item is verified against the images actually present, so a deleted image or a previous dataset can never silently train into a run. (Existing caches re-cache once into the new layout on the next run.)
  • Adaptive LR: the Min LR floor is now authoritative over the starting LR at launch; new 3e-4 — low-rank only floor option.
  • Problem Images window: 33% wider, persistent rows (no more rebuild lag), background thumbnail loading, scroll position preserved, status text wraps to window width.
  • Gallery: full-width likeness chart, opt-in current-run filter, baseline picker shows which folder it lists, and a samples-dir ownership guard so two Fizgig instances can never fight over the sidecar files.