Character LoRA training — follow-ups (resume, auto-slicing, e2e training validation)

[atomantic]

Follow-ups deferred from the initial Character LoRA Training feature (see docs/plans/2026-06-12-character-lora-training.md):

• End-to-end smoke training on both runtimes. The trainers are unit/CLI-verified but no real run has executed yet. mflux: install the venv (scripts/setup-image-video.sh), run a 20-step training on a 5-image dataset, confirm the wrapper's entrypoint probe + output regexes match the installed mflux version (scripts/train_mflux_lora.py STEP_RES / checkpoint zip layout / adapter discovery in find_adapter), pin mflux in server/lib/pythonSetup.js PIP_NAMES if its training surface churns. flux2: 20-step run on black-forest-labs/FLUX.2-klein-4B, verify save_lora_weights round-trips through scripts/flux2_macos.py --lora-paths on the SDNQ-quantized inference pipeline (watch for lora_utils.py silent-skip warnings), and validate the render_sample 8-step Euler preview decodes correctly.
• Full optimizer-state resume. --resume-from in scripts/train_flux2_lora.py reloads adapter weights only — optimizer state is not persisted, so resume restarts AdamW moments. The run-delete route and boot reconcile already keep checkpoints; add a 'retry from checkpoint' param on POST /api/lora-training/runs that threads resumeFrom/resumeCheckpoint (builders already accept it: server/services/loraTraining/runtimes.js).
• Vision-LLM crop proposals for reference-sheet slicing. sliceReferenceSheet (server/services/loraDatasetGenerate.js) uses a fixed grid because turnaround layouts are model-generated/non-deterministic; the user prunes bad crops. A vision-LLM pass could propose bounding boxes per panel instead. — Shipped in Character LoRA follow-ups: vision-LLM auto-slicing, CLI-provider captioning, training queue UX #1408.
• Caption via CLI providers. loraDatasetCaption.js reuses describeImageDataUrl which only supports API-type providers; CLI providers (claude/codex with vision) could caption too. — Shipped in Character LoRA follow-ups: vision-LLM auto-slicing, CLI-provider captioning, training queue UX #1408.
• Training queue UX. The Render Queue UI shows training rows via the PARAM_ALLOWLIST label fields but has no training-specific row treatment (sample thumbnails, loss sparkline). — Shipped in Character LoRA follow-ups: vision-LLM auto-slicing, CLI-provider captioning, training queue UX #1408.

[atomantic]

Update after live verification on a 48 GB M-series machine (mflux 0.17.5 installed at /opt/miniconda3):

Now verified / reworked:

• mflux 0.17.5 removed FLUX.1 training — the 'mflux' runtime now trains FLUX.2 Klein natively in MLX (mflux-train --config), and FLUX.1-dev was dropped from the trainable model list. Both engines (mflux MLX + torch/diffusers) now produce FLUX.2 adapters; engine pick is automatic (mflux preferred when mflux-train ships next to the configured python).
• Config builder rewritten to the real ≥0.17 schema (kohya-style NNNN.png+NNNN.txt data staging, preview*.txt samples, lora_layers.targets with per-variant block counts: 4B=5+20, 9B=8+24) — validated via mflux-train --dry-run.
• Real micro-training executed end-to-end through the wrapper (z-image-turbo, fully cached): tqdm \r-stream parsing → STEP lines, artifact watcher → CHECKPOINT, checkpoint-zip adapter extraction (*_adapter.safetensors), RESULT JSON, SIGTERM cancel path. mflux's timestamp-suffix output-dir behavior handled.
• OOM hardening: first attempt (bf16, no guards) hard-rebooted the 48 GB machine. deriveMfluxMemoryConfig now derives quantize/low_ram from total RAM (<64 GB → QLoRA-style quantize-4 + disk-backed cache; verified the same training then fits comfortably).

Still open:

• First real flux2-klein-base-4b training — needs the ~16 GB bf16 download; this machine had only 20 GB free disk, too tight to run responsibly. Config passes --dry-run; mechanics proven via the z-image run.
• Verify the flux2-trained adapter (diffusers transformer. key prefix per mflux source) round-trips through scripts/flux2_macos.py --lora-paths on the SDNQ inference pipeline.
• torch fallback trainer (train_flux2_lora.py) smoke run on a non-mflux machine.
• Other items from the original list unchanged (optimizer-state resume, vision crop proposals, CLI-provider captions, queue UX).

[atomantic]

Split during a future-tag triage (2026-06-18). The three no-bench-needed follow-ups — vision-LLM auto-slicing, CLI-provider captioning, training queue UX — are carved into #1408 (un-futured, claimable now). This issue stays future for the parts that need a real training run: e2e smoke training on both runtimes + optimizer-state resume validation.

[atomantic]

Triage 2026-06-23 — keep future; blocker confirmed (needs a real training run)

Reviewed in the future-tag sweep. The two remaining items both require a real LoRA training run on the GPU box to validate, which can't be done autonomously:

• End-to-end smoke training on both runtimes — must actually execute a 20-step run (mflux MLX + torch/diffusers) and confirm the wrapper's entrypoint probe / output regexes / checkpoint-zip layout / adapter discovery match the installed versions, then verify the trained adapter round-trips through scripts/flux2_macos.py --lora-paths on the SDNQ inference pipeline. The remaining flux2 base run still needs the ~16 GB bf16 download + free disk (per the 2026-06-12 live-verification comment).
• Optimizer-state resume — the code (persist AdamW moments in scripts/train_flux2_lora.py, thread resumeFrom/resumeCheckpoint on POST /api/lora-training/runs; builders already accept it) is writable, but proving resume actually continues (vs restarts) the optimizer is only meaningful against a real multi-checkpoint run — so it stays coupled to the bench session rather than shipping blind.

Blocker: human GPU bench session (real training run on the M5/48GB box). Stays future, no milestone (condition/availability-gated, not calendar-gated). The three no-bench follow-ups already carved out to #1408.

[atomantic]

Automated triage (swarm /do:next, 2026-07-07): The optimizer-state resume slice called out here (persist AdamW moments in scripts/train_flux2_lora.py, thread resumeFrom/resumeCheckpoint through the run builders) is already shipped — merged in PR #1253 (issue #1252, 2026-06-13) and documented in the v2.21.0 changelog.

Verified in current main:

• scripts/train_flux2_lora.py save_checkpoint() writes optimizer.pt (AdamW state + PEFT adapter state + step counter); --resume-from restores all three and continues range(start_step + 1, steps + 1) (no over-training / step-collision), erroring on pre-optimizer checkpoints.
• End-to-end threading is complete: resumeTrainingRun → resumeCheckpoint → runTraining → buildFlux2TrainArgs({ resumeFrom }) → --resume-from, covering both mflux and torch runtimes.
• server/services/loraTraining/resume.test.js has a flux2 resume round-trip test asserting no RESUME_UNSUPPORTED_RUNTIME.

No code change to ship. The remaining item — end-to-end smoke training on both runtimes (real 20-step GPU/venv runs to validate entrypoint probe / output regexes / checkpoint-zip layout / adapter round-trip through scripts/flux2_macos.py --lora-paths) — genuinely needs a human GPU bench session and cannot be done autonomously. Leaving this issue open for that.