v0.0.4 — LoRA training support (ai-toolkit)
comfyuiWATCHER now watches LoRA training, not just image generation. Add an
ai-toolkit host and its runs appear in the same rack as
your ComfyUI instances — step X/Y, s/it, ETA, loss, and what the run is doing right now.
Training hosts
Add a host, pick Training instead of Generation, and point it at ai-toolkit's UI server
(default http://host:8675). The top bar gains a Show filter — All / Generation / Training —
so a big rack can be narrowed to one kind. Existing hosts.json files keep working untouched; a
host with no kind is treated as ComfyUI.
A training module reads:
- STEPS LEFT and a progress bar, from the run's own config rather than the database column
- TRAINING RATE on a dial scaled for trainers: 60 … 1 s/it
- BASE MODEL / RESOLUTION / RANK, read from the job config — metadata only, never dataset
contents or captions - LOSS, and ELAPSED / ETA / RATE
- a phase readout in the top right —
Model Loaded,Loading dataset,Training. A run is
silent for minutes while it loads a model and caches a dataset, and without this the card is
indistinguishable from a stall.
A dial that fits training speeds
Training gets its own face, 60 → 1 s/it. On the generation face (0.01–100 it/s) every real
training run piles up within a few degrees of centre, because LoRA training is far slower than
sampling — a 2 s/it run and a 30 s/it run were nearly indistinguishable. Both faces are graduated
in it/s underneath, so needle-right means faster on every module in the rack, whichever kind it
is.
The dial is also much larger — 176×81 to 371×171 pixels at the default window size.
Fixes
- ETA and rate were badly wrong on trainers, in three separate ways, all found by watching a
real run: a model-load stall poisoned the average (45 s/it on a job 25 minutes from done), a 1s
poll quantised the estimate, and a cached API response raced the live one to produce 0.005 s/it
with a 3-second ETA. The estimator now takes a median of per-step intervals, discards pauses,
and samples only the uncached read. - A lone card no longer stretches across the window. With one host visible, the grid collapsed
its empty tracks and smeared a single card to 863px. - HTTP requests now time out instead of hanging a card on stale data forever.
Honesty rules, unchanged
Unknown is still shown as unknown. A run already in progress when the widget starts shows -- for
elapsed, because ai-toolkit records no training start time and time-since-noticed is not elapsed.
A stopped run clears the card rather than turning it red — stopping is a decision, not a fault.
Downloads
- Windows —
comfyuiWATCHER Setup 0.0.4.exe - Linux —
comfyuiWATCHER-0.0.4.AppImageorcomfyuiwatcher_0.0.4_amd64.deb