Skip to content

v0.0.4 — LoRA training support (ai-toolkit)

Choose a tag to compare

@an80sPWNstar an80sPWNstar released this 13 Aug 03:18
· 31 commits to main since this release

comfyuiWATCHER now watches LoRA training, not just image generation. Add an
ai-toolkit host and its runs appear in the same rack as
your ComfyUI instances — step X/Y, s/it, ETA, loss, and what the run is doing right now.

Training hosts

Add a host, pick Training instead of Generation, and point it at ai-toolkit's UI server
(default http://host:8675). The top bar gains a Show filter — All / Generation / Training —
so a big rack can be narrowed to one kind. Existing hosts.json files keep working untouched; a
host with no kind is treated as ComfyUI.

A training module reads:

  • STEPS LEFT and a progress bar, from the run's own config rather than the database column
  • TRAINING RATE on a dial scaled for trainers: 60 … 1 s/it
  • BASE MODEL / RESOLUTION / RANK, read from the job config — metadata only, never dataset
    contents or captions
  • LOSS, and ELAPSED / ETA / RATE
  • a phase readout in the top right — Model Loaded, Loading dataset, Training. A run is
    silent for minutes while it loads a model and caches a dataset, and without this the card is
    indistinguishable from a stall.

A dial that fits training speeds

Training gets its own face, 60 → 1 s/it. On the generation face (0.01–100 it/s) every real
training run piles up within a few degrees of centre, because LoRA training is far slower than
sampling — a 2 s/it run and a 30 s/it run were nearly indistinguishable. Both faces are graduated
in it/s underneath, so needle-right means faster on every module in the rack, whichever kind it
is.

The dial is also much larger — 176×81 to 371×171 pixels at the default window size.

Fixes

  • ETA and rate were badly wrong on trainers, in three separate ways, all found by watching a
    real run: a model-load stall poisoned the average (45 s/it on a job 25 minutes from done), a 1s
    poll quantised the estimate, and a cached API response raced the live one to produce 0.005 s/it
    with a 3-second ETA. The estimator now takes a median of per-step intervals, discards pauses,
    and samples only the uncached read.
  • A lone card no longer stretches across the window. With one host visible, the grid collapsed
    its empty tracks and smeared a single card to 863px.
  • HTTP requests now time out instead of hanging a card on stale data forever.

Honesty rules, unchanged

Unknown is still shown as unknown. A run already in progress when the widget starts shows -- for
elapsed, because ai-toolkit records no training start time and time-since-noticed is not elapsed.
A stopped run clears the card rather than turning it red — stopping is a decision, not a fault.

Downloads

  • WindowscomfyuiWATCHER Setup 0.0.4.exe
  • LinuxcomfyuiWATCHER-0.0.4.AppImage or comfyuiwatcher_0.0.4_amd64.deb